I would bring them if you got 'em. Seriously, you just can't account for the weather up here, and especially downeast coastal weather near Bar Harbor. You may need shorts, or you may need 3 layers at night. Just can't tell.
I would at least be prepared to sleep in 30 degree weather (nights) especially if you're tenting. Can't go wrong with sub-zero mummy bags if ya got em.
Start following the weather for wherever you're camping a couple weeks ahead of your arrival to get the pattern for that time. Don't be afraid to call the campground and ask about the weather/temp forecasts for when you're going to be there.
If all goes bad, I think there's a Hampton Inn in Ellsworth