Much later.
True, but a lot of the ingredients of the second floor lunchroom encounter are present in Baker's affidavit on the day of the assassination:
Baker (not Truly) sees a man walking away from the stairs
He calls out to the man who turns around and walks back towards him
Truly vouches for the man
Baker let's the man loose and hits the stairs
I suppose the point I'm making is that Baker could've made a mistake about which floor it happened on and when looking closer at the third and fourth floors as options it seems this might be the case. If, however, Baker did mean the second floor it is notable he makes no mention of seeing his man through a door. The impression I get from Baker's affidavit is that the encounter happens in an open space.
One thing has always bothered me about the idea the encounter with Baker, Truly and Oswald in the second floor lunchroom is a hoax, with Baker and Truly culpable in it - why not have it on the fifth or sixth floor? Why leave so much room for doubt?