Having lag spikes (internet) only in the afternoon?

Hey there.

So, the stuff is, that I'm having lag spikes, or how should I call them, only in the afternoon, but the thing is, that I've got a pretty good internet connection. Never had any issue on other servers, just wynncraft.
To describe what lag spikes I mean. Those, when you can walk and stuff, but can't cast spells and the mobs doesn't move around, then just in a flash, they teleport you and such. But I only have this in the afternoon, like I said. So if anybody have an idea, about how I could fix this and such, then I would really appreciate it, because I can't enjoy playing with friends like this. (btw, the other guy in our group has like 10Mb/s download and like 30-50 ping normally and doesn't have this issue. I have 90-100Mb/s connection and normally a stable 5-8 ping on the server and usually everywhere, but I'm getting this.)