Happens to me all the time. The only thing I can think of is that it's the routing. I think my horrible lag stopped a while ago, but I did a tracert and there was one point that was being REALLY laggy.
So yeah, you'll have to wait it out. Or get a new ISP.