You haven't defined this property you call "agency". How then can you definitely determine whether you posses it or that someone else doesn't? The only thing I can be assured of is my own existence.
Well agency is the feeling I have of being able to impact the world. I cannot know if you have it but I have extrapolated that based on my impression of you. For all I know, I'm the only one to exist.