Yeah, but I mostly go in for the ecology anyways.
Maybe the chance of me dying from the fish is like 0.0001%, but for the fish, they're 100% dead, each time. So I like to make sure they've still got enough family in the water, so we aren't the last generation to know cod or orange roughy.
Also, that the fish aren't harvested in a destructive and wasteful manner (by-catch), even if the stocks aren't being over-depleted. Some fishing methods scrape up the ocean floor, or kill off non-game fish, in an effort to harvest more of the target fish at defrayed costs.