The shaky hands is going to be a problem likely both for archery and pistol shooting (at least at moderate ranges). It wouldn't matter as much if you're planning on shooting rifles/shotguns, as you can brace them against yourself and have the option of shooting from prone.
Shooting guns is a lot more expensive (after the initial purchase), that's true.
I have done archery and shot around 20 different guns, but only have a beginner/moderate level of skill with each. I don't see why you couldn't get good at both if you practiced enough though. It seems there are a fair amount of hunters who go out for both gun and bow hunting seasons, there have to be some out there who are good at each.
You could always consider meeting halfway and getting a crossbow. Offers stability from bracing but allows you to retrieve your arrows/bolts (less expensive).