Interesting question you raised, so I decided to spend a few minutes geeking out on this.
According to the USDA nutrient database, olives have about the same amount of iron as beef (3mg per 100g). I certainly eat a lot more beef than olives, and I don't worry about how much iron I'm getting while chowing down a steak.
Another thing to consider is that heme iron (found in meat, poultry, fish) is much better absorbed than the non-heme iron found in plant foods (such as olives). It's a big difference: about 23% absorption for heme iron vs less than 8% for non-heme.
Prepared foods tend to contain vastly more iron than whole foods, and most of that comes from synthetic sources that are added during manufacturing, rather than occurring naturally. For example, Cheerios have ten times more iron than olives (34mg per 100g).
Now, apparently California canned black olives (the kind most of us in the States probably eat) are treated with ferrous gluconate to preserve their color, so their iron content might be considerably higher than fresh black olives or canned green olives. Unfortunately, I can't find any numbers. Again, this is non-heme iron we're talking about so it's not well absorbed.
Personally, if I was concerned about getting too much iron, I'd cut out the baked goods and processed foods before ditching the olives.
Or donate blood a few times a year.