Excuse what may be a long answer but it's an gross oversimplification to say that seed X is a source of phytoestrogens and you will grow tits. The ligands in flax actually block the binding of oestrogen to the receptor thus blocking its action. People have even looked at using it in oestrogen dependent breast tumours with some good results. Alot are lost in the processing of the ground seed to oil as well.
Phytoestrogens are a large group of chemicals that are composed of compounds with quite different properties that are exposed to all sorts of reactions/processing either in the bacteria of the gut or in the body itself.
Soy is a good example, a few different types of isofalvones with multiple sites of metabolic processing in the gut and body some more estrogenic in behaviour than others (the flavanoid daidzein has been shown to have significant anti-estrogen effects). Another example is the flavnoid chrysin (from passiflora plants) that is a phyotestrogen and sold to bodybuilders as an anti-aromatase. The issue isn't a simple one.
Conclusion: eat yer flax but also eat yer fish oils and walnuts as well. Never rely on one source for any nutrient.