He wasn't seen with a rifle at both events.
Good point.
Do you have proof that Oswald took his rifle home from the Walker residence?
Marina who is the only witness we have, gave the superfluous information that Oswald did leave his rifle near to the Walker residence.
The DeMorenschildts gave testimony about the rifle after the Walker incident that contradicted Marina's story. They claimed they saw Oswald's rifle in his home at the time when Marina said he hadn't gone back to Walker's neighborhood to retrieve it yet.
So someone lied or misremembered some details. It's impossible to know who forgot or lied but Marina got a number of details wrong in her testimonies on other issues so...
Amongst Oswald's possessions were multiple photos taken of Walkers house which were taken a month before the assassination attempt and were taken with the same camera that took Oswald's Neely street backyard photos.
Yeah, there's enough evidence to conclude that LHO had an interest in Walker. But that alone wouldn't have been enough to convict him for the crime.
The "Walker note" gave details to Marina on what to do after Oswald committed a deed that could lead to his imprisonment or death.
Mostly useless as evidence in the Walker incident for reasons that have already been addressed.