MIXING SERVER DOWNLOADS WITH TORRENT DOWNLOADS

Jose — Fri, 22 Feb 2008 00:39:21 +0100

Let's say you want to jumpstart a new torrent download by using prior downloaded data, eg, from an earlier different torrent (eg, different sized chunks) or from a server. Why would you want to do this? Well, so that you don't have to sit through an entire torrent. I mean, if you downloaded half the file from the server but the server is out of commission now or you think you can get a better download rate from a new torrent you saw, you may want to reuse the part of the file you already downloaded so as to save time and bandwidth. OR say you have a file that got corrupted (maybe you were playing around with dd or something), but you think it is still mostly in tact (you know it's corrupted because an md5sum of it gives a bad value, or it's a file you can view with an application and it looks corrupted).

How do you use this data to jumpstart a torrent download? Like this (using KTorrent as an example):

-- First, find a torrent that is of the file you want to complete or fix. I presume you already have the torrent link since that is presumably what motivated this exercise, but I'll state this step to be complete.

-- Next, make a backup of the partial (or corrupt) data you want to leverage in order to be safe. Why? Because it's possible to make a mistake somewhere and delete this data. If you really are doing this to save time and take advantage of a partial download (eg, if you have a slow Internet connection), why take chances? Just make the backup.. and give the backup a different name just to be safe (eg, if the backup is named the same but is in a different directory, you still might wipe it out a bit easier with KTorrent than if it was named differently).

-- Name one copy just like the file to be downloaded via the torrent. Ie, take the torrent link and keep the last portion. Then drop the ".torrent" ending. This is the name of the file. Eg, for the current running example, "LinpusLinux-LiveCD-i386.iso" would be the name of the file. This is how you would name your partial (or corrupt) file. If you aren't sure of the name, just start up the torrent (see step below) and see what new files appear in the directory. The new one w/o the .torrent ending is the name you want.

-- Copy this properly named file, ie, the partial download (or the corrupt file) you want to finish (or fix), to a clean directory with nothing else in it. [This step may not be necessary, but let's make our lives simple for now by eliminating variables.]

-- Make sure you are not currently downloading the torrent you want to download. [This step too may be optional, but just follow it. I gave up trying to test all possibilities I could think off, so I am describing one procedure that works.]

-- Start a torrent (eg, on KTorrent, File -> Paste torrent url, or perhaps just click on the torrent link from a website and open it with KTorrent). When prompted, save the torrent to the same directory where you have the partial (corrupt) data you want to complete (fix). [You made the back up, right?]

-- KTorrent will start performing a data integrity check if it finds a file with the same name as the torrent file to be downloaded. So it sees the partial download data that was named as expected; rebuilds it's knowledge of the chunks it has and the chunks it needs (that's the integrity check); and starts up the torrent download from the peers to fill in *only* the missing (ie, missing or corrupted) chunks! If you cancel the integrity check midway, KTorrent will just start downloading all the chunks I think (so you will not be saving time and will be redownloading data you already have stored in the proper place). However, I don't think it trashes the file. Thus I think you can stop/remove the torrent download, and try opening up the torrent again (this time allowing the integrity check to procede to the end). KTorrent just writes into the file at the proper location where a chunk fits. It doesn't truncate; thus, it will just write good data over good data or good data over bad data, but it won't clobber anything of value.. I think. In any case, if you do mess up or remove the .torrent file and data, you have that trusty backup and can restart from step one, right?

Any time you want, you can do an integrity check (which is a neat feature).

Also, you may find that you don't have to do all the steps above, but I make no guarantees. In fact, I make no guarantees that if you do follow the steps above exactly as described that you won't cause your computer and files to explode. Reader beware.

Anyone that wants to do more testing and report on it, go ahead.

One more thing, don't panic if KTorrent says something like torrent file or data is corrupt. That message can appear for any of numerous reasons. It just means KTorrent got confused with the torrent link you gave it (eg, I gave a bad url one time repeatedly and didn't realize it) or you (re)move the data file (ie, the file being downloaded) from where it expects it to be and it panicked. Also, if you remove the .torrent file without telling it or rename the directory or something like that, it may also produce that nasty sounding message or maybe even a worse one.

Other torrent apps besides KTorrent may have a similar feature. If not file a feature request

Happy downloading and hope this feature saves you time or is at least useful in some way.

[Years ago, before I ever tried a torrent app, I thought about hacking wget to implement a more flexible way to download so that I could download only precise chunks easily (using http partial request header). I never got around to finishing that project (or studying wget to see where to make the incision). Now, I can probably just use KTorrent or maybe link to a torrent library or something (I'll investigate the library possibility later). Goes to show you that if you react too slowly, the wheel will get implemented for you. How horrible, right? FOSS rulz!]

Downloading from most

Jose — Tue, 19 Feb 2008 21:48:28 +0100

Downloading from most central servers is usually faster and sometimes much faster than d/l from a torrent but not always.

Go to this page
http://linuxtracker.org/index.php?page=torrents and test the download of something like mythbuntu (near the bottom as of the time of this posting) that had 8 people seeding and no one leeching. It would be very fast because all 8 could be dedicated just to you. You will probably saturate your download speed as if you were tapping a decent server. OTOH, Linpus Linux had 1 seed and around 12 leeching. There what you get is a download speed of several tens of kilobytes/sec if you have been on for a while or about ten if you have been on for a short time (ymmv as it depends on many factors including what kind of Internet connection you have).

The reason I decide to seed this one was that the actual linpus servers were already quite slow. For much of my download they were around 5 Kbytes/sec; however, for most iso downloads I do I really prefer the main servers or large mirrors over torrents if the torrent would be slow or modest (low ration of seeders to leechers).

********

One more thing about the protocols, ignore DHT. I think it's a way to bypass the tracker system, but something I read recently suggested that it wasn't used that much and/or wasn't completely functionals or something.

In case it wasn't clear, I

Jose — Tue, 19 Feb 2008 17:13:01 +0100

In case it wasn't clear, I have very limited experience specifically with bit torrents and P2P. I did, however, read some of the online links I posted and have read some on P2P networks elsewhere (and know some about networks and software in general). I also have at least acquired *something* and seeded *something* through KTorrent.

This all certainly makes me an expert , I know, but just to be safe, remember to bring the salt shaker.

I think linuxtracker.org

Jose — Tue, 19 Feb 2008 16:54:52 +0100

I think linuxtracker.org uses FOSS to handle the tracking (and maybe even the website). If you own your own website or the hosting company allows you to use such software, you don't need linuxtracker.org or anyone else. In fact, you can even provide those services for others.

That's something nuxified/libervis network might want to consider providing for the community (not that there is a need for it but it might ultimately help attract more eyeballs to this website).

Torrents in a nutshell

Jose — Tue, 19 Feb 2008 16:22:31 +0100

The wikipedia article is useful http://en.wikipedia.org/wiki/BitTorrent_(protocol) and the specification for those that are curious will be maintained here http://wiki.theory.org/BitTorrentSpecificati

Nuxified GNU/Linux Help Forums - Torrents in a nutshell - Comments

MIXING SERVER DOWNLOADS WITH TORRENT DOWNLOADS

Downloading from most

In case it wasn't clear, I

I think linuxtracker.org

Torrents in a nutshell