Faster Downloading Implementation Discussion

[IronFist]

Community funded central server, unlimited badwidth, cURL, proxies go kill urslef kthnx.

You make a compelling argument.

Either way, speeding up the UDP download is going to be done. HTTP would be a very nice addition, but it is one of those fringe features that isn't necessary right now.

[Smokey]

Community funded central server, unlimited badwidth, cURL, proxies go kill urslef kthnx.

You make a compelling argument.

Either way, speeding up the UDP download is going to be done. HTTP would be a very nice addition, but it is one of those fringe features that isn't necessary right now.

Speeding up UDP is blah, just go HTTP.

[jitspoe]

Why are you so anti-UDP?

[sk89q]

Why not do something like this? Get the speeds and still get the compatibility.

Code: [Select]

+ Does the game server specify a HTTP server?
  - Yes, it does
    + Is this HTTP server authorized? (i.e. robots.txt kind of thing)
      - Yes, it is
        + Does the file exist?
          - File found
            - Download through HTTP
          - File missing on HTTP server
            - Ask the game server for a download through UDP
      - No, it isn't
        - Ask the game server for a download through UDP
  - No, it doesn't
    - Ask the game server for a download through UDP
 

[Smokey]

Why are you so anti-UDP?

Because I know how simple HTTP would be.

[y00tz]

What are the cons to using libcurl?

[jitspoe]

- Adds bloat.
- Doesn't get around secure proxies and some of the other issues people had downloading the server list.
- Won't speed up LAN server downloads (unless somebody sets up an HTTP server on the LAN).

[sort]

maybe thttpd can help

http://www.acme.com/software/thttpd/

[Eiii]

Paintball2 can double as a webserver, as well!
I don't think that's entirely a good idea. >_> Just one more port for people to open, or hell might happen if someone connected to a paintball2 server through a web browser.

Also, the name of that library is fun to pronounce.

[jitspoe]

I'd prefer to steer clear of HTTP.  Adding an HTTP server to paintball2 would potentially open a whole new can of security worms, and as Eiii said, it's just one more port to deal with.  Plus many networks block HTTP traffic.  UDP is guaranteed to work if you can connect to the server (and, if you can't, it doesn't matter if files transfer or not).  Theoretically, it might even be possible to transfer at a faster rate, too, since there is less header overhead.

[Apocalypse]

I think UDP is what you should use it seems to be a better choice.

[DrRickDaglessMD]

Jits, please forgive my inexperienced ramblings, I've never programmed anything like a network game engine before, but after reading your initial statement something occurs to me:

your proposed idea seems to have similar logic to TCP's error recovery - why not use TCP-based file transfer anyway?, then all error recovery is catered for in-protocol. This way is also less affected by latency times as your TCP ack's not only take care of themselves but probably also are also more reliable than your UDP client requests. For instance, if I have a high ping (200+), and receive a segment from the server ok, yet my request for the next block is dropped on the way to the server due to some congestion or CPU bottleneck (or in some cases traffic shaping if your ISP is as rubbish as mine) I'll have to wait for the client timeout value to count off before I even have a chance of receiving the next desired segment. Whereas a TCP request would mean the server and client wouldn't really need to worry about timeouts and missing acks and things as it would all be taken care of for it by the TCP stack + traffic shapers and line congestion would be less likely to drop the ack/request. On reading that back, it strikes me that this may not speed up file transfers in general, just for those who are losing out due to the nature of UDP's unreliable nature mixed with unfortunate next hops.

Why does the server run at 10hz? I imagine theres a good reason for this, but like I say I have no idea of the ins and outs of netcode.

- Rick

[jitspoe]

Well, what I plan to do would essentially behave like TCP, except it would be done at the application level.  It's more or lest a "best of both worlds" approach.  You get good transfer rates regardless of latency, but without the hassle of adding TCP ports into the mix.

As for the server running at 10Hz, it was just a matter of using minimal resources, I guess.  It doesn't exactly run at 10Hz, but it sends packets out at that rate and does game/physics calculations at that rate as well.  It can receive packets at a higher rate, and with some slight adjustments that I've already done, it will be able to send them faster as well (but only for downloads).

[loial21]

- Rick

You!

Question....from Loial Loialess: How will this effect the servers, specifically in regards to cpu and bandwidth usage? Just curious?

[y00tz]

I noticed some of your code updates in the CVS for build 20/faster downloading, but when I updated my repository the new code didn't compile because of "e" not being defined in this code:

Code: [Select]

if (e.x)
e.x(Cmd_Argv(1), atoi(Cmd_Argv(2)));
Bah, hohum oh well.

[DrRickDaglessMD]

Ahh ok, so really the current implementation is very similar to TFTP, but limited to 10 req's a second.

Loial, I wouildn't have thought running the file transfers over TCP would impact CPU usage much at all, and while TCP headers are larger than UDP's (about 4 times the size) I doubt the few extra blocks would take a noticable amount of time more than with a UDP datastream.

It looks like you want to implement something similar to TCP's flow control, but don't forget that TCP ack's the next seq number in the stream from the last one it received, not each segment just later on. Are you planning a sliding window? I don't know what the realities of efficiency would be like over UDP if you did once the windows start getting rather large, especially over poor quality connections, but then again I doubt this would harm the performance of the server if its only serving 10 requests per client per second.

- Dr Rick Dagless M.D

[jitspoe]

Loial: I imagine the end result will be this: Players will spend less time downloading and more time playing, causing the game to be more enjoyable and increasing the likelihood that they will continue to play.  New players won't have to wait 10 minutes to download a map and play, only to have the map change and wait another 10 minutes for the next map and be turned off, never to play again, but will instead increase the player base and draw in more players.  The servers will have more people connected, and naturally use more CPU and bandwidth.

Of course, that's in an ideal world, but I'm sure it will help some.  The results of the file transfers themselves won't be any different than people downloading files off of the site or accessing the forums and other activities.

y00tz: Code on the CVS probably isn't in a compilable state.  That's more or less my backup so I don't lose the code I've been working on.  The fast downloading system isn't complete yet, anyway.

Rick: I will have a sliding window.  It'll probably start out ~10k/s.  I'm not sure how fast I should make it "slide", though.  If the window becomes large, it's not going to hurt the server performance any.  It'll just send packets out at a slower rate.  I'm not quite sure I follow why TCP ack's the next sequence... why would it ack something it hasn't received yet?

[DrRickDaglessMD]

IIRC its called forward acknowledgement or something, the basic jist of it is that the ack's keep track of how much has been sent - the seq numbers aren't arbitrary, they are the number of the first byte in the block. If a block goes missing, the server will receive a lower ack seq number than it was expecting and know to send a block again starting from that number. That way you only have to ack at the end of your receive window or on a block timeout, instead of after every block and the server will still know which block has gone missing.

I'm not sure what you mean about how fast you'd make it slide - if for every consecutive error-less window you increase the window size, then it will slide up as high as it can, then when it starts to see retransmission requests, stop or make it slide down a little, that way your speed will ramp up quite quickly to its optimum level.

- Dr Rick

[loial21]

Dr. Jitless Dagpoe (MD),

If you were the host what would you do when yearly net income started to drop because of a software upgrade your client just uploaded? I would be a bit pissed and looking for payback literally.

So parcentage wise what are we looking at? Dagletus said it was about 4 times the size.

I saw this upgrade needed back in 2000...so I am not backing out or questioning the logic of thy upgrades. Infact I fully support this. *shame on Jitler trying to show me the already known logic. I considered that a slap in the face. * i sorry nathan, craig miss read.

Here is your turn to slap me back. Just help me forcast additional expense, if any.

Live long and prosper DP.

(I say all this owning Jitspoe and Snipen money, i digress and poop backwards)

[loial21]

You make a compelling argument.

Not just compelling but is it more cost efficient, Ironphistah?


-1 post me...