Everyone at sometime in their life as had that feeling of being set up, perhaps it was just for a practical joke if your lucky but we know that feeling and don't forget it. Oswald was no chump, what would he have done if he had that feeling on this day? Go along with it and then when he had the chance at the last minute, give 'em the slip? Why not?
The only way to avoid this is to physically stop him, what happens then? Well think about it and explain how that is even an option?
No chump, no way, never pictured him like that, not once.