Well, I used to swim in high school, with times very close to yours (51 for 100 free, 55 or 56 for 100 fly, 23 for a 50 free from a flat, 22 for a 50 free on a relay).
Work on doing pull-ups, dips, and rows. I can not stress enough how much weighted pull-ups and dips were able to transfer DIRECTLY over to the pool. If anything though, just eat your face off, especially in the off-season. Don't expect to look like Ronnie Coleman or anything. I mean, come December there's a good chance your strength may suck while your doing all that yardage (New Year's eve 100x100's? FUCK ME!).