Agreed. If I am understanding the problem correctly, you can move your hand closer to the front of the dumbbell(the way you are facing) to solve this. Generally, heavier weights tend to sag to one side due to the mechanics of your wrist/hand.
And on a side note, I think Dumbbells are great for farmers walks. I have neither a trap bar or farmer bars but dumbbells have worked fine.