jitspoe's .plan

[y00tz]

16bit Linux build

Heh, judging by all your recent posts at QuakeSrc, I was thinking you were shooting for a 64bit build...

[jitspoe]

Haha, well, percentage wise, a 16bit version is probably almost in as high of demand as a 64bit linux version.  I'd throw them both in the less than 1% category, but, hey, I'm a nice guy.  I've actually got it working now and played a couple maps with it.  Unfortunately there are no X-Fi drivers for Linux, so I can't tell if the sound works or not, but everything else seems to be fine, minus the annoying mouse acceleration that I can almost-but-not-quite get rid of.

[jitspoe]

I'm getting pretty close on the fast file downloading system.  I actually have something that "works", but I was trying to make use of the existing Quake2 netchan code.  Turns out that system discards out of order and redundant packets.  Since the download system might be sending out more than one packet on a given sequence number, some of those packets get dropped unnecessarily.  I'm currently in the process of switching over to connectionless or "out of band" packets that will behave independently of the game packets.

[Apocalypse]

Sounds great Jits hope you get it as good as possible.

[ViciouZ]

I'm getting pretty close on the fast file downloading system.  I actually have something that "works", but I was trying to make use of the existing Quake2 netchan code.  Turns out that system discards out of order and redundant packets.  Since the download system might be sending out more than one packet on a given sequence number, some of those packets get dropped unnecessarily.  I'm currently in the process of switching over to connectionless or "out of band" packets that will behave independently of the game packets.

Any estimate yet of the effective maximum download speed this will provide? 1000kb/s is all good in theory, but how fast is it going in practice?

[jitspoe]

That all depends on the server and client.  I imagine the dedicated servers could handle 100-300k/s to broadband clients.  Also, the theoretical limit is more like 40,000k/s now.

[ViciouZ]

40,000k/s

That'll be good for a few years yet then xD

[KnacK]

40,000k/s

That'll be good for a few years yet then xD

and maybe by then I'll be off of dialup.......

[ViciouZ]

And we'll all be on OC-192s!

[jitspoe]

Calrathan got out his networking book and I've implemented something that looks more like TCP now.  It's not really any faster, unfortunately, but local transfers run ~800KB/s-2MB/s.  Top transfer rate from cable modem to cable modem was 42KB/s.  I still have a bunch of tedious little things to do to clean up the code and add error checking and stuff.  Unfortunately a tornado knocked the power out, so I won't be getting anything done tonight.

afk, tornado.

[tenchu]

well anything is better than 10

[jitspoe]

I'm getting pretty close.  The "fun stuff" is pretty much all gone now.   I just have tedious things left to do.  I think what I'll do is provide the fast transfer functionality in the next build, but with a separate command, so we can test file transfers on different servers and see if there are any adverse effects.  If everything goes well, I'll make the fast transfer mode the default (so when you connect to a server with a new map, it uses the fast download system).

In addition to all the security checks, I still need to add some method to limit the transfer rate for servers that may not have sufficient bandwidth to transfer files quickly and run a server.

[y00tz]

I hope you'll have the beta server jammed pack with maps so we can test this.

[maxpower]

Just empty 10 meg maps! YEA that would be sweet

[Calrathan]

I have successfully downloaded an 800mb DivX movie over the paintball2 fast download system.  Thanks to jitspoe, I now have a covert channel for sneaking prelease builds of games out from work.

[jitspoe]

Well, I've been making slow progress.  I haven't had any really long coding sessions lately as I've been busy with things in real life (just got a level 10 upgrade for my automag - gonna play some real paintball this weekend, heading home next weekend, went to AWA last weekend, went to a big CCGA lan party the weekend before, etc.)

The actual file transfer portion of the new download system has been done for a while, but there are lots of tedious little things to do.  I've gotten most of them done.  I'll just dump what I've got on my "todo" list -- I have a special text file just for this where I keep notes of snippets that need to be checked/added/deleted, things to do, etc.  If I don't write stuff down right away, I forget about it, which leads to bugs, security holes, etc.  So here you go:

Com_Printf("Fileid mismatch: %d != %d\n", (int)fileid, (int)cls.download3fileid);
   keep track of last fileid so that doesn't pop up at the end of a download.

      if (fileid == cls.download3lastfileid) // fileid of a previous download.  Ignore it
         return;

   // Update rate
   timediff = realtime - cls.download3lastratecheck;

   if (timediff > 1000) // check every second
   {
      float rate;

      rate = 1000.0f * (float)cls.download3bytessincelastratecheck / (float)timediff;

      if (cls.download3rate)
         cls.download3rate = 0.75f * cls.download3rate + 0.25f * rate;
      else
         cls.download3rate = rate;

      cls.download3bytessincelastratecheck = 0;
   }

download3lastratecheck - need to set this to the start time

CL_StopCurrentDownload(); gets called at the end and beginning of each download, so the previous fileid would get overwritten with -1 if multiple

files are downloaded in a row.

#define DOWNLOAD3_MAXWINDOWSIZE 128


         //if (windowsize < num_chunks)
         if (windowsize < DOWNLOAD3_MAXWINDOWSIZE)
#define DOWNLOAD3_DEBUG

#define DOWNLOAD3_MINRESENDWAIT 1000 // 1 second before attempting to re-send a packet.






      CL_RequestNextDownload(); - not sure if I need my own function for this or not.

ditch download3_delay?

chunk_offset = atoi(Cmd_Argv(2)); // downloaded offset - todo, not used.  This is handled in the dl3confirm

Need to make the client request hr4 files if highres textures are enabled.

Fix COM_StripExtension and other extension function to get everthing after the LAST period.

md5sum the whole file?

disable sv_maxfps check during downloads - no longer applicable with out of band?

if we go the tcp/window route, remove the code to actively check download3's as fast as possible.

test LoadFileOrAlternate()


   //float         download3_delay;   // time/window between sending chunks (increases as packets are lost, decreases as

packets are received)
   //int            download3_lastsent;
   //int            download3_lastdrop; // time the last packet was dropped.
//   sv_client->download3_lastsent = Sys_Milliseconds();
//   newcl->download3_delay = DOWNLOAD3_STARTDELAY;
//   cl->download3_delay *= 1.09f;

Need to make the download system download both skm and skp files.

//SV_NextDownload_f();

be sure to free all data structures

Put stuff in the lastfaileddownload table for failed download 3's so they don't spam the console.

disable transfer rate display with old mode, so it doesn't show bogus values. (done? test.)

//#define DOWNLOAD3_DEBUG

cl_download3 or cl_fastdownload cvar to enable/disable new download system

Server side rate control

sv_fastdownload or sv_download3 option to enable/disable fast download system

Add check so client knows if server supports download3

May have some duplication.  Sometimes I think of the same thing twice.  I wouldn't really bother reading it.  I just felt like posting it.  I guess it's too late at this point.

[y00tz]

I have a special text file just for this where I keep notes of snippets

I wish my todo-text was special  

[KnacK]

This is for you y00tz:

Code: [Select]

#define todo-text
#include special


[jitspoe]

Getting closer:

cl_download3 or cl_fastdownload cvar to enable/disable new download system

#ifdef USE_DOWNLOAD3
cvar_t *sv_fast_download; // jitdownload
#endif

#ifdef USE_DOWNLOAD3
   sv_fast_download      = Cvar_Get("sv_fast_download", "0", 0); // jitdownload - todo - enable by default
#endif

#ifdef USE_DOWNLOAD3
cvar_t   *cl_fast_download; // jitdownload
#endif

#ifdef USE_DOWNLOAD3
   cl_fast_download =   Cvar_Get("cl_fast_download", "0", 0); // jitdownload - todo - enable by default, archive
#endif

#ifdef USE_DOWNLOAD3
   if (cl_fast_download->value)
   {
      CL_Download3_f();
      return;
   }
#endif

#ifdef USE_DOWNLOAD3
extern   cvar_t   *cl_fast_download; // jitdownload
#endif






Server side rate control

sv_fastdownload or sv_download3 option to enable/disable fast download system

Add check so client knows if server supports download3

#define DOWNLOAD3_DEBUG

      CL_RequestNextDownload(); - not sure if I need my own function for this or not.

Need to make the client request hr4 files if highres textures are enabled.

Fix COM_StripExtension and other extension function to get everthing after the LAST period.
//SV_NextDownload_f();

Compile and test on linux


md5sum the whole file? - done? test again.
   md5sum = Com_MD5Checksum(sv_client->download, sv_client->downloadsize);
   MSG_WriteLong(&sv_client->netchan.message, md5sum); // md5 checksum of the file, for validation.
   cls.download3md5sum = MSG_ReadLong(&net_message);
      // Verify that the file downloaded correctly
   Com_Printf("MD5 sum check on downloaded file failed.  %d != %d.  Re-downloading.\n", md5sum, cls.download3md5sum);
   Cbuf_AddText(va("download3 %s\n", cls.downloadname));

test LoadFileOrAlternate()

be sure to free all data structures

Need to make the download system download both skm and skp files.

//#define DOWNLOAD3_DEBUG

[jitspoe]

Closer yet...

Need to make the client request hr4 files if highres textures are enabled.

Fix COM_StripExtension and other extension function to get everthing after the LAST period.

Compile and test on linux


md5sum the whole file? - done? test again.
   md5sum = Com_MD5Checksum(sv_client->download, sv_client->downloadsize);
   MSG_WriteLong(&sv_client->netchan.message, md5sum); // md5 checksum of the file, for validation.
   cls.download3md5sum = MSG_ReadLong(&net_message);
      // Verify that the file downloaded correctly
   Com_Printf("MD5 sum check on downloaded file failed.  %d != %d.  Re-downloading.\n", md5sum, cls.download3md5sum);
   Cbuf_AddText(va("download3 %s\n", cls.downloadname));

test LoadFileOrAlternate()

be sure to free all data structures

Need to make the download system download both skm and skp files.

//#define DOWNLOAD3_DEBUG

make svc_download3start dynamic, retrieved through extension system

Try to standardize extension system with other Q2 engine mods.

If using download3, remove attempts to download all the file types - this is handled server side.