Is it possible that it was easier to know if someone was CIA back in the early 1960s than when you were working in army intel? Alot of people in ww2, such as being in the military and OSS, went on to join the CIA. Therefore, because of their shared history during ww2, people might be able to track one another's career path thereafter and deduce someone was now working for the CIA.
I guess the other possibility is that Nagell was a CIA officer working under military cover. A bit like David Morales. Had Nagell publicly stated that, maybe he would lose his pension. This might explain how he knew so many CIA officers.