Author Topic: Faster Downloading Implementation Discussion  (Read 11481 times)

jitspoe

  • Administrator
  • Autococker
  • Posts: 18802
Faster Downloading Implementation Discussion
« on: June 29, 2007, 01:29:14 PM »
Since this is by far the most requested feature, I might as well kick up some discussion on plans for the implementation.

For starters, and this isn't really related to the faster part, I think there needs to be a progress bar outside of the console, so people don't think the game just locked up at the "loading" screen.

Now, I know a lot of people have suggested HTTP downloads, but I don't think that will work very well.  Here's why:
- As I discovered from the server list and later the login system, a number of people have weird proxy configurations that don't allow a basic HTTP connection.
- LAN servers may not have Internet access, and those are the types of servers that people should be downloading from the fastest.
- Who is going to foot the bill for the bandwidth?
- What if the map repository doesn't contain the map a server is running?
- What if the version of the map is different?

I think the most effective method is for the server hosting the game to transfer the files, too, using the same method of communication as game traffic (UDP).  If you can connect to the server, you can download maps.  This is the way map downloads are currently handled.  It just needs to be beefed up a bit.

Currently the way file transfers work is this:
Client requests a download ->
<- Server checks if the file exists and responds accordingly
Client requests first 1k block of file ->
<- Server sends first 1k block of file
Client requests second 1k block of file ->
<- Server sends second 1k block of file
...
until the transfer is finished.

Two key problems with that.  #1, the server runs at 10hz, so even under optimal conditions, the maximum file transfer rate is 10k/s.  #2, the server waits for the client to request the next block to send it, and the client waits until it receives a block to request the next one.  If the latency is high, it drastically impacts the overall throughput.   In short, your download speed is determined by your ping + it's rounded up to the nearest 10th of a second, so if you have a ping of over 100ms, you'd probably be getting transfer rates of 5k/s.

Now, the new file transfer system I think should work something like this:

Client requests a download ->
<- Server checks if the file exists, and returns the filename of an appropriate file if it exists (this could be a pak file, if the file is part of a package, or perhaps an alternate image format, like jpg instead of wal)
Client OK's or denies the download of the file (won't overwrite existing files or files outside of the paintball2 directory) ->
<- Server sends first block of file
<- Server sends second block of file
<- Server sends third block of file
Client acknowledges first block of file ->
<- Server sends fourth block of file
Client acknowledges second block of file ->
...

If a client fails to acknowledge a block after a set period of time, the server will re-send that block and slow down the rate at which it sends data.  If the client acknowledges all the blocks, the server will increase the rate at which it sends data.

KiLo

  • Autococker
  • Posts: 2086
Re: Faster Downloading Implementation Discussion
« Reply #1 on: June 29, 2007, 02:29:31 PM »
You can already chat like an obs when you are loading but you can't see the chat of other people.

ViciouZ

  • Map Committee
  • Autococker
  • Posts: 2227
Re: Faster Downloading Implementation Discussion
« Reply #2 on: June 29, 2007, 02:50:55 PM »
I have a cunning plan...
Spread the server loads. E.G. system as it is now, but other servers on the serverlist server multiple parts, like a multi source p2p download. More traffic is taken from the less empty full servers.

jitspoe

  • Administrator
  • Autococker
  • Posts: 18802
Re: Faster Downloading Implementation Discussion
« Reply #3 on: June 29, 2007, 02:56:36 PM »
I think that would make thigs overly complex, and issues with missing/different file versions and LAN servers wouldn't be resolved that way, either.

lekky

  • Autococker
  • Posts: 2449
Re: Faster Downloading Implementation Discussion
« Reply #4 on: June 29, 2007, 07:04:36 PM »
how about the client only acknowledging a failed block? Would the server still to send blocks if the client had a problem and was unable to send anything back to the server :/

jitspoe

  • Administrator
  • Autococker
  • Posts: 18802
Re: Faster Downloading Implementation Discussion
« Reply #5 on: June 29, 2007, 07:08:01 PM »
How would the client know if a block failed?  And if the client didn't send an ack, the server would keep sending that block.  If the client had a problem sending anything back to the server, it wouldn't be able to play anyway.

Smokey

  • Autococker
  • Posts: 1172
Re: Faster Downloading Implementation Discussion
« Reply #6 on: June 29, 2007, 11:45:00 PM »
All fast ingame download systems I know of use HTTP.
Just do it.

loial21

  • Autococker
  • Posts: 2807
Re: Faster Downloading Implementation Discussion
« Reply #7 on: June 30, 2007, 01:52:19 AM »
Um.
User friendly.

Period.


Eiii

  • Autococker
  • Posts: 4595
Re: Faster Downloading Implementation Discussion
« Reply #8 on: June 30, 2007, 06:19:59 PM »
All fast ingame download systems I know of use HTTP.
Just do it.
I agree with Smokey here. I'm kinda worried about lagging the server, if the chunks it sends are that much bigger.

Smokey

  • Autococker
  • Posts: 1172
Re: Faster Downloading Implementation Discussion
« Reply #9 on: July 01, 2007, 09:51:03 AM »
Tremulous and most ET mods use curl, maybe you should check it out?
http://curl.haxx.se/

IronFist

  • Autococker
  • Posts: 1304
Re: Faster Downloading Implementation Discussion
« Reply #10 on: July 01, 2007, 06:59:07 PM »
Steam Source-based games (and I think HL-based) allow a server operator to configure an HTTP download location for maps -- typically, it is some server they control. Even if the server is on the same box as the game server, it is still faster than the speed limits imposed by the game's protocol.

I don't recall if any anti-leeching stuff is provided. However, you could do the following to discourage it:
1) Use a standard file (ie: "paintball2-map-allow.txt") in the root of the HTTP maps directory that determines if that directory is okay for paintball2 in-game map leeching. If it does not exist, it is assumed that it isn't okay.
2) Use a header (ie: "X-paintball2-server-referrer") that includes the ip:port of the server we are leeching a map to play on with. This header would be sent when retrieving "paintball2-map-allow.txt" and map files, so that the server could straight out deny or generate a dynamic "allow" file based on the referring pball server (you could dynamically deny/allow access and block map downloads directly using this header).
3) Have the server respect "paintball2-map-allow.txt" and refuse to give the download HTTP location to clients if the HTTP server does not allow map downloads.
4) The client may need to check "paintball2-map-allow.txt" before a download too, since a game server may lie in order to leech bandwidth (of course, since the client sends a referring header, the HTTP server operator can always manually block requests).

If a map desired is not on the given HTTP server, the client might ask the game server for an alternate location that might possibly have it, then eventually default to the normal map download process if it is unable to complete an HTTP download.

Like HL, you could probably expand this to include all media types.

You might allow for rotation of the HTTP download servers in case for some reason or another there are different sets of maps at different locations, or the server operator feels they need to load balance a bit to allow optimum HTTP download speeds.

jitspoe

  • Administrator
  • Autococker
  • Posts: 18802
Re: Faster Downloading Implementation Discussion
« Reply #11 on: July 02, 2007, 01:44:25 AM »
I really don't like the idea of making server setups even more complicated.   Having to set up an HTTPD server alongside of the paintball server is going to cause a skyrocket in the already high "Why won't my server work?" posts.  cURL doesn't resolve the issue of secured proxies (already tested this a while back for the server browser).  Map downloads via UDP aren't going to lag the server any more than HTTP.  In fact, they will probably have even lower overhead since the UDP headers are smaller.

Fred187

  • PGP
  • Posts: 48
Re: Faster Downloading Implementation Discussion
« Reply #12 on: July 02, 2007, 04:07:48 AM »
The downloads aren't that slow. No slower than any other game.

lekky

  • Autococker
  • Posts: 2449
Re: Faster Downloading Implementation Discussion
« Reply #13 on: July 02, 2007, 06:45:14 AM »
They are, you probably dont realise as these maps are smaller in size to most other games (i expect).

Smokey

  • Autococker
  • Posts: 1172
Re: Faster Downloading Implementation Discussion
« Reply #14 on: July 02, 2007, 09:25:17 AM »
I really don't like the idea of making server setups even more complicated.   Having to set up an HTTPD server alongside of the paintball server is going to cause a skyrocket in the already high "Why won't my server work?" posts.  cURL doesn't resolve the issue of secured proxies (already tested this a while back for the server browser).  Map downloads via UDP aren't going to lag the server any more than HTTP.  In fact, they will probably have even lower overhead since the UDP headers are smaller.
If they have proxy issues, then I'm sure they can deal with slow downloads. :)

Zorchenhimer

  • Autococker
  • Posts: 2614
Re: Faster Downloading Implementation Discussion
« Reply #15 on: July 02, 2007, 10:08:39 AM »
They are, you probably dont realise as these maps are smaller in size to most other games (i expect).

No, I think Fred187 is right. I mean, RtCW has a average download rate of about 10 Kb/s. But then again, the server admin could also set the max download rate.

y00tz

  • Autococker
  • Posts: 2742
Re: Faster Downloading Implementation Discussion
« Reply #16 on: July 02, 2007, 11:15:43 AM »
No, I think Fred187 is right. I mean, RtCW has a average download rate of about 10 Kb/s. But then again, the server admin could also set the max download rate.

I've never gotten below 150kb

I think these maps, for being <1meg to >3mb should download much faster than 4-8 minutes on my connection.

KiLo

  • Autococker
  • Posts: 2086
Re: Faster Downloading Implementation Discussion
« Reply #17 on: July 02, 2007, 12:02:10 PM »
I think these maps, for being <1meg to >3mb should download much faster than 4-8 minutes on my connection.

Same here. I have 6 MBPS download connection and it takes right around 4 minutes at the least.

Krizdo4

  • PGP
  • Posts: 43
Re: Faster Downloading Implementation Discussion
« Reply #18 on: July 04, 2007, 04:36:50 PM »
Steam Source-based games (and I think HL-based) allow a server operator to configure an HTTP download location for maps -- typically, it is some server they control. Even if the server is on the same box as the game server, it is still faster than the speed limits imposed by the game's protocol.

I don't recall if any anti-leeching stuff is provided. However, you could do the following to discourage it:
1) Use a standard file (ie: "paintball2-map-allow.txt") in the root of the HTTP maps directory that determines if that directory is okay for paintball2 in-game map leeching. If it does not exist, it is assumed that it isn't okay.
2) Use a header (ie: "X-paintball2-server-referrer") that includes the ip:port of the server we are leeching a map to play on with. This header would be sent when retrieving "paintball2-map-allow.txt" and map files, so that the server could straight out deny or generate a dynamic "allow" file based on the referring pball server (you could dynamically deny/allow access and block map downloads directly using this header).
3) Have the server respect "paintball2-map-allow.txt" and refuse to give the download HTTP location to clients if the HTTP server does not allow map downloads.
4) The client may need to check "paintball2-map-allow.txt" before a download too, since a game server may lie in order to leech bandwidth (of course, since the client sends a referring header, the HTTP server operator can always manually block requests).

If a map desired is not on the given HTTP server, the client might ask the game server for an alternate location that might possibly have it, then eventually default to the normal map download process if it is unable to complete an HTTP download.

Like HL, you could probably expand this to include all media types.

You might allow for rotation of the HTTP download servers in case for some reason or another there are different sets of maps at different locations, or the server operator feels they need to load balance a bit to allow optimum HTTP download speeds.

I like this idea. It puts the responsibility of bandwidth with the person running server. Simple access control. Would work on a LAN.

ViciouZ

  • Map Committee
  • Autococker
  • Posts: 2227
Re: Faster Downloading Implementation Discussion
« Reply #19 on: July 05, 2007, 02:28:18 PM »
Community funded central server, unlimited badwidth, cURL, proxies go kill urslef kthnx.