Frazier is addressing the whole room, not just one person.
Like myself, Brian has noticed that Frazier's eyes, as well as his head, move to the right in synchronisation with the "We looked at one another" part of his statement.
The video quality of this section is better than the quality of this gif.

Duncan,
The man who asked the question that Frazier was addressing was sitting on the right side of the room, from Frazier's POV.
If you actually watch the part of the
48-minute and 53-second youtube video in which that man is asking that question, and how Frazier looks at him a few times in exactly the same direction
while answering his question, you'll realize that Frazier doesn't "motion" with his head or eyes as you and Brian claim.
-- MWT
