Trying to diagnose a laggy server

[sk89q]

I've had this problem with my server for a while now. It's impossible to run a game server because there is a lot of packet loss. However, that was back then, when we were on the "cheap bandwidth," but now we're redundant via seven ISPs on a 100 mbit pipe, and we're still having the problem. Can anyone care to tell me what I can do to diagnose it? I suspect it's a problem with the data center, and I *really* wouldn't be surprised if it was, but if I want to ask them about it, I would need something to show.

[KnacK]

Describe the server: OS, hardware (mem, cpu, RAID ?)

Have you determined if its network lag or actually caused by the hardware?

[sk89q]

FreeBSD 6.2-RELEASE
CPU: Dual Core AMD Opteron(tm) Processor 275 (2192.82-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f12  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x3<LAHF,CMP>
  Cores per package: 2
CPU: Dual Core AMD Opteron(tm) Processor 275 (2192.82-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20f12  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x1<SSE3>
  AMD Features=0xe2500800<SYSCALL,NX,MMX+,FFXSR,LM,3DNow+,3DNow>
  AMD Features2=0x3<LAHF,CMP>
  Cores per package: 2
real memory  = 4160684032 (3967 MB)
avail memory = 4016058368 (3830 MB)
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9>
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9>
ad4: 305245MB <Seagate ST3320620AS 3.AAK> at ata2-master SATA150
da0: <AMCC 9550SX-4LP DISK 3.08> Fixed Direct Access SCSI-5 device

I haven't been able to determine anything.

[KnacK]

you could try running ping from teh command prompt for a boatload of packets and dump that to a text file and see if there is any variance over time.

[sk89q]

I haven't really been able to reproduce actual packet loss by pinging repeatedly. There's flux in the response time, but that could also be due to my connection.

[jitspoe]

At what rate are you pinging?  Keep in mind the clients typically send 40-80packets/second.

[sk89q]

The packets sent by ping -f are getting dropped by a firewall or something, at least on my end. If I wait a bit, the continuous flood works for a few seconds, but then all the packets start getting dropped.