Faster Downloading Implementation Discussion

[jitspoe]

I'm not sure what you mean about how fast you'd make it slide - if for every consecutive error-less window you increase the window size, then it will slide up as high as it can, then when it starts to see retransmission requests, stop or make it slide down a little, that way your speed will ramp up quite quickly to its optimum level.

What I mean by that is how much should it "slide up" (or down)?

The approach I have is actually a bit different than TCP.  It may be a bad idea, but I think it will be fairly effective:

The files will be broken up into 1KB chunks.  The size can be changed, but I'm using 1K for simplicity.  When the download starts, the server creates an array for the status of each chunk: 0 means it hasn't been sent, -1 means it has been sent and acknowledged, and any other value is the timestamp for when the chunk was sent.  The chunks start sending at a rate of 10k/s.  Each time a chunk is sent, the rate slows down a bit.  When a client receives a chunk, it ack's it.  When the server receives the ack, it sets the chunk status to -1 and bumps the rate back up.  After all the 0 status chunks have been sent, the server goes back and finds the oldest chunk and re-sends it and continues until all chunks have been ack'd.  This could be tweaked so that chunks past a certain age get sent before chunks that haven't been sent at all, but it doesn't really matter.  In the end, the client will get all of the chunks, and the order they're in doesn't matter.  Each chunk is ID'd and put in the right order on the client side.

See any major flaws with that logic?  It's about half implemented now, so hopefully I'll be able to do some testing soon.

loial: In the end, the bandwidth used would be the same.  The overhead of the headers isn't going to make much of a difference.  Essentially, what would make the difference is how many people are on the server.  Maps would download faster, so people would be more likely to stick around and play.

[loial21]

what would make the difference is how many people are on the server.

So if more people are on the server......which is our goal.....then a differnece would be seen?

Duh

I am asking will how much larger of usage would there be if the same amount of people connected. Regarding the current version vs the newer one?

[DrRickDaglessMD]

Hmm if i understand this right, you probably don't need a window at all.

If the current downloads are slow because of the inefficient error correction algorithm, then if you just send all the blocks out as fast as you can (without impairing the servers bandwidth required for players! I imagine something like specifying your servers bandwidth in the config file when you are setting up your server, and then taking the number of connected players, multiplying that by a safe average bandwidth requirement per client, subtracting that from the specified bandwidth in the cfg, then further dividing that by the number of clients in the downloading state) and quench slightly if you get excessive numbers of missing ACKs (e.g. if you have 1kB blocks, if you receive < (transmit rate in kBps)/2) a second), and the multiple-passes will take care of anything else. This way, you don't need to make any decisions about sliding the window up and such.

Thinking about it, this way is quite simple really and probably pretty effective, as long as you start high and go down as the level of lost acks becomes known, instead of needing logic to increment speeds after a number of sucessful runs (which might need tolerance levels and stuff for normal lost packets (acks) to function most efficiently). Having said that, I don't know what the bandwidth situation is like when running popular servers, it might be you need a highly scalable method dependant on how much free bandwidth you can possibly use if the servers are at high capacity.

Nice ideas anyway, can't wait to see it implemented - If only so I don't have to shed a tear everytime someone complains about having to wait ages to download KungFu...

- Dag

[Eiii]

Downloads are slow now because the server runs fairly slowly and sends out small packets. >_>

[DrRickDaglessMD]

yeah good point, I had forgotten about the 10hz thing. So how fast will the server be able to send out packets now?

- Rick

[jitspoe]

yeah good point, I had forgotten about the 10hz thing. So how fast will the server be able to send out packets now?

- Rick

1000hz.  I figure that's plenty, as people won't likely be getting over 1000k/s.

While your idea to just send data out as fast as the server can would technically work, saturated bandwidth tends to go a lot slower (acks might get dropped, and packets would have to be re-sent multiple times).  Plus it would just mean lots of wasted server bandwidth.  Even if it takes a while to get up to speed, the transfer rate will be remembered by the server, so transferring lots of small files should be more efficient than with TCP.

[KnacK]

1000hz.  I figure that's plenty, as people won't likely be getting over 1000k/s.

is that 1000Hz or 1000kbs?

I don't thinkg my dialup could handle 1000kbs

[jitspoe]

Both.  1000Hz transferring 1KB blocks.

[Apocalypse]

Once you decide which form you will use which appears to be almost done how difficult do you think it would be to implement? Will you just put it in for build 20?

[jitspoe]

Like I said, it's already about halfway implemented.  I will probably put it in build 20, but have it as a separate command for testing purposes in case there are any bugs or other issues that need to be worked out, then it will be the default method of transfer in a future version.

[Apocalypse]

Hmmm... that sounds good to me, I wish you good luck as their is not much I can help you with in this.

[jitspoe]

lol, whoops.

[y00tz]

Whoa.  160% of Carpathian is like 12 maps put together, sweet.

[Apocalypse]

That looks really really messed up lol.

[Zorchenhimer]

[jitspoe]

I'm making progress.  The performance isn't quite as good as I want it to be yet, though.  Seems the server tends to max out at 250-500 hz (2-4 ms delay between packets).  That's about 300k/s.  Much better than it used to be, but still slow for LAN connections.  I think I can get around it by sending multiple packets per frame.  I'd consider making larger packets, too, but I think packet fragmentation can be a problem.  I'm not sure how that works with UDP.

[jitspoe]

Calrathan whipped out his networking book and found the TCP specifications.  I've basically got TCP over UDP right now.  It works well enough.  I was able to transfer at 35-42k/s from cable connection to cable connection.  On LAN connections, it's a bit slower than it should be, but naturally a lot faster than the old system.  While the file transferring itself is functional, I still have a lot of cleanup, error checking, etc. to work on, so I'm not sure when the next release will be.