Having a file on him isn't the same as active, scrutinous surveillance and harassment.
Remember, Oswald was a former member of the Armed Forces with experience/knowledge of U.S. radar technology, something he openly acknowledged the Soviets would be interested in upon his defection. He openly renounced his U.S. citizenship and renounced democracy in general when he defected. Now, he never officially renounced his citizenship, but he made clear all of these intentions when he reported to the U.S. embassy in Moscow that he planned to defect. Keep in mind that while Oswald was in Russia Gary Powers was shot down in his U-2 spy plane.
I just have a hard time believing that, IF Oswald was a legitimate defector and given that the U.S. embassy in Moscow was at least aware of his INTENTION to formally renounce his U.S. citizenship, the U.S. intelligence community would have let him back into the country with relatively little scrutiny.
The reason he was under surveillance regarding the whole Fair Play for Cuba thing is mostly because of where he was making noise about it: in the middle of the New Orleans intelligence community, as pointed by Oliver Stone in that fucking fantasy movie JFK. That part was one of the few accurate parts of the whole movie. He also attracted attention simply due to his brazenness in promoting FPC.
Beyond that, the attempt to gain a visa to Cuba through the Russian AND Cuban consulates in Mexico City should have been MAJOR red flags. Yes, this activity did garner some attention by the CIA. However, IF he was under heavy surveillance at the time (which he SHOULD have been given his activities over the last several years), the CIA and the FBI definitely would have been aware of where he was at all times and he would have been a person of major interest when Kennedy came to Dallas. In fact, he would have been under heavy surveillance on the 22nd. He probably would have had an FBI agent assigned specifically to him, ESPECIALLY if they knew he worked in a building along the motorcade route.
Of course, this could all just as likely be a case of the CIA fucking up, as they were wont to do back then. But this, along with many other factors (including his known association with David Ferrie, Sergio Arcacha-Smith, Clay Shaw and also Shaw's known connections with the CIA) make me think that this was not a simple case of the intelligence community dropping the ball.