Powershell Slower based on Runtime account

So I have a powershell script that goes out and grabs some database properties on approximately 30 servers. It returns data on a couple hundred databases in under 1minute, however when it gathers data on the instance the job is running from it takes almost 30minutes to return the info on the final 10 databases. When I run it using the SQLAgent Service account it returns almost instantly however when I use the proxy is when it takes 30minutes.I'm a little stumped how the account it runs under would be slower just for that one server.