If I'm understanding your post correctly,
a) You get lag spurts playing on a remote server.
b) Remote players get lag spurts playing on the server you're running off of your own connection.
What do these two things have in common? Your Internet connection. What type of connection do you have? Is it shared? Are you running bittorrent or something like that? Are you on a wireless network?