This could very well be the case, as I have a personal example. A while back (couple years, maybe?) I had my T-levels checked purely out of curiosity (wasn't on any T-boosting supplements or anything). They did the test in the afternoon. My total T levels were 250, which shocked me a bit (I believe I was 29 years old at the time), so the doctor suggested getting them re-checked, only drawing the blood in the morning this time. I did it a week later in the morning, and my total T was 950, or thereabouts.
Sleep (or lack thereof), stress, alcohol consumption and other factors can negatively affect T levels, but I really think the only difference in these two tests, taken a week apart, was the time of day.
For what it's worth . . .