Eric Helms wrote in his Pyramid of Muscle and Strength: Nutrition, paraphrasing:
There aren’t foods that are actively harmful. What’s harmful about a food is if consuming it leads to not consuming other foods in a way that leads to deficiency of essential nutrients.
What I gathered from that book, which I would recommend any day, is that, as long as your diet has your bases covered (which definitely seems to be your case, even in light of the various steps you said you’ve been taking to check your health), there aren’t any foods that will single handedly cause adverse effects (that’s barring medical conditions, of course).
The takeaway is that your food selection should be based off of an inclusive paradigm, rather than exclusive. That is, the question is not what should you be eliminating, but what should you be eating which you’re not.
I realize it might look like I’m not answering your question, but here’s the thing: eating a single source of carbs, or mostly a single source, isn’t something I’d recommend under normal circumstances but the reason is you risk depriving yourself of nutrients found in other sources. However, you aren’t eating a lot of carbs to begin with, so you’re likely getting the nutrients you need from other foods.
Therefore, I don’t see anything wrong with what you’re suggesting.
Also, to give my n=1 experience, I used isomaltulose for a while a couple years ago. Didn’t observe any noteworthy changes or effects, although I was having around 30 g per workout, so much less than what you’re discussing here.