You can start with week 4 no problem. Keep the rest intervals as short as you can without hindering performance of course.
You will be swimming in boot camp. Not to the extent of what is in this program, but you should be comfortable in the water and be able to swim confidently. Depending on your MOS, you will be required to pass a swim qualification at different levels. If you are infantry (03), you will have to pass SwimQual 2 minimum, if you interest Recon, SwimQual1 (unless they have changed this since the early 2000's.).
Boot camp really is not a big deal, it is more mental than anything else and just being there gets really old and irritating because you feel like you're in a POW camp. Don't sweat it, its only 13 weeks.