It's possible, but you're going to have to up your food intake to make it work. I'm 5'11" - 165 and at a little over 3 years of training have no trouble with big guys (250 plus). The more mat time you get, the more you'll learn how to handle stronger guys, using technique, speed, and flexibility.
Not to mention, you'll learn how to make yourself feel much heavier from the top. My entire game is focused on gaining top position early and grinding them out, floating from one top position to the next until a sub opens up. If you've got a strong, deep grapevine and a stiff cross face, holding down even the biggest guy is not too difficult.
With all that said, it's still recommended that you try to work on your strength. I rotate between 5x5 and 5/3/1 programs every 3 months. I lift 3 days a week with a push / pull / legs split. Each day I have one major lift that correlates into a grappling motion (horizontal push, horizontal pull, hip dominant leg movement) that is strength focused (heavy weight-low reps). Then, I round my workout out with accessory lifts for that particular day with medium-heavy weight and higher reps / sets for endurance / hypertrophy.
Since we fight in set weight classes, my goal is to be as strong as possible for my weight class (167 - 154). I want to walk around no heavier than 175, then cut down to 167 to fight.
Also, while there's nothing wrong with training 5 days a week, you can make solid steady progress with as little as 2 - 3 days on the mat. Currently, I teach 2 days a week and train 3 others, and I've been making the most steady progress with this schedule over the past 6 months.
Another thing you can do away from the gym is spend some time each day drilling the 3 basic movements. Every movement in grappling (regardless of the art) is based on one or more of these 3 movements: bridge, shrimp, and thread the needle. Even the sit-out is just an inverted thread the needle. A basic armbar is essentially a shrimp followed by a bridge. What I tell my students is to take a minute whenever a commercial comes on TV to knock out 10 or so reps of one or more of these movements. This will help you to develop muscles / movements specific to grappling in addition to your weight lifting strength training.
edit I just realized that I missed the part where you mentioned wanting to gain for the mirror and not the mat. Sorry, I wasn't quite awake when I posted my reply. In that case, the short answer then would be my very first sentence above about bumping your calories significantly.