Torrents in a nutshell

The wikipedia article is useful http://en.wikipedia.org/wiki/BitTorrent_(protocol) and the specification for those that are curious will be maintained here http://wiki.theory.org/BitTorrentSpecification. [Bit Torrent, Inc went closed source/protocol on us not long ago, so the wiki spec is likely the one you want to keep up with and not the original spec as published by Bit Torrent, Inc.]

Basically bit torrent is a specific P2P protocol that is understood by clients such as KTorrent, (micro)Torrent, Azureus, and others. [I am only familiar with KTorrent which I know is FOSS].

This P2P protocol will generally use a "tracker" to keep tabs on the users (peers) that are participating in the sharing. You give up some bits to others automatically while they give up some bits to you. So that the system works efficiently and reliably, the file that is being shared (you can "seed" any file at all that you want to share with others) is shared in specific sized chunks and each of these chunks has a hash/checksum associated with it and distributed to all peers so that each peer can verify when it has correctly finished downloading a chunk from another peer. Thus the system has protection from those that want to poison other peers.

Well, you can distribute poison (corrupted files, trojans, etc) as the initial seeder. Assuming you trust the initial seeder though, then you can have confidence that participating peers (including those that graduate from leechers to seeders) cannot contaminate the file chunks they give you (your bit torrent client would reject the chunk as it only accepts chunk that match the checksums in the original .torrent file).

Let me say right now that I just seeded Linpus Linux http://linuxtracker.org/download.php?id=7521efbf6e7e6fb7a1cbac220eec0dbe3d87ebd8&f=LinpusLinux-LiveCD-i386.iso.torrent and what everyone should always do after a binary download from anywhere (at least if you expect to run it) is to check the md5sum of the file. The value for this torrent/iso file is "61cd5d05f853d8c6d71da973b5c2d987" which you can verify is what the Linpus website shows. If they are not showing that, either I am lying, made a mistake, or they changed their binary at some point between the time I downloaded it a few days ago and when you checked (so if you come across this thread in the far off future, expect the md5sums to disagree because they likely will have upgraded their download offering, necessarily leading to a new md5sum for it).

On Linux check the md5 of a new download by using the md5sum cli "command". Eg, "md5sum LinpusLinux-LiveCD-i386.iso" (and the result in this case should be "61cd5d05f853d8c6d71da973b5c2d987 ..."). If you don't have md5sum installed, get it or ask for help. If you get a bad md5sum checksum, there was a problem along the way, and you will likely have to redo the download. [Maybe you haven't finished downloading the file entirely yet, or you are looking at the wrong md5 sum or downloaded the wrong file, or you did "wget -c http://..." from a site that changed binaries on you over the time you were downloading (an issue if you downloaded the file through a slow modem connection over many days as happened to me once)... or maybe you just have a flaky system or were extremely unlucky (eg, a very very strong solar flare may have changed the bits on your computer's hardware memory).]

OK, on a modern distro, you should hopefully have a bit torrent client installed and when you click on a ".torrent" link on a webpage, you should be offered the option to save to disk or to open it in the bit torrent client. In any case, you are safe saving it to disk and then from inside your bit torrent client opening it up.

What is a .torrent file?

It's a file with a very simple structure (you will realize that if you can manage to make sense of the specification linked at the top). Basically there is some administrative info at the top of the .torrent file and then you get a long list of hash sums (sha-1 I think) for each of the chunks in the file to be shared.

What do you do with a .torrent file? How do you participate in a torrent? How do you start a torrent (to share your own files)?

The person that wants to create a new torrent has to create such a .torrent file and then upload it to a "tracker". KTorrent created my .torrent file for the Linpus .iso I had downloaded from the Linpus website (because that .iso was the file I wanted to share as a torrent). To create the .torrent file in KTorrent, I clicked on File -> New. In the dialogue that popped up, I had to determine the chunk size (I chose 1MB). After pointing to the file on disk for which I wanted to generate the .torrent I was basically done. KTorrent then created the .torrent file. Actually, I also supplied a tracker url to KTorrent when building the .torrent file, but I think that may not have been necessary (I'll have to check later).

At this point you find a website (a server actually since the port for bit torrent is not 80, but they probably have a website, too) that provides tracker services, eg, linuxtracker.org. You have to set up an account with them. Once logged in, you are given the option to upload a torrent file and you are also shown the url for their tracker (not sure if that url is necessary, but I did put that url into KTorrent when setting up to build the .torrent .. it was http://linuxtracker.org/announce.php or something like that; the real tracker url you get later on, the one that actually works, looks different though -- see following paragraphs).

When using linuxtracker.org's webform to upload a .torrent file, I had to fill some basic information and point to the .torrent file on disk. The tracker then uses the .torrent file I just uploaded in order to coordinate the peers. The tracker distributes that .torrent file to all peers that click on the .torrent link (like the one provided at the top of this post). Also the bit torrent protocol requires that your torrent client keep a dialogue with the tracker. That is how you know (your torrent client app knows) about new peers (I think). Remember the .torrent file is fairly small and basically holds the name of the file, the number of chunks and sizes, etc, as well as the hash sums for each chunk. So at 1mb, this Linpus torrent has between 600 and 700 chunks.

Anyway, once I uploaded the .torrent file, KTorrent requires that you open up the torrent (File -> Open the .torrent file you already saved to disk if you saved to disk (after uploading to linuxtracker, not the original .torrent you created) or click on the .torrent link in a web browser and choose to open with KTorrent) as if you were going to download it even though you will have the file you are going to seed already on the hard drive.

Where does this torrent link come from when you are the initial seeder? Well, what I did after uploading the .torrent file was to do a search for "Linpus" on linuxtracker.org. That gave me one result only (I got zero results intially because the torrent was not active.. so put "All" in the field right next to the search box in order to see inactive torrents, too). The torrent link is the one right to the right of the name (it will end in .torrent).

Do this download attempt even if KTorrent indicates you are already seeding that torrent. In reality you are not (a bug?) because that entry was created before you had actually logged in and set things up through linuxtracker (maybe other trackers don't require this step, in which case KTorrent would not have a bug). Once you click on the link to download the torrent, it will appear as a new torrent on your status indicator, and it will instantaneously be "downloaded" entirely. The difference with this new torrent is that it will work while the initial one will not. You can go into the menu and stop/delete the initial torrent to clear up your status window (you will also note that the tracker status (the "Trackers" tab) for that initial link will indicate that the tracker is not communicating with KTorrent; this is unlike the case when you opened up the torrent after the upload to linuxtracker, as that torrent will be communicating with linuxtracker.org and it has a different announce url).

Hopefully I didn't lose anyone with this last part of the discussion. If you have problems seeding, then you will know what I mean; otherwise, don't worry too much about what I just said. Torrent clients besides KTorrent should work differently. I found this out from reading on the linuxtracker forums of people that were having trouble seeding up a file and they were using KTorrent. Anyway, the key is to open up/click on the torrent link you get from linuxtracker *after* you have uploaded your .torrent file to them (try this even if you don't use KTorrent). The torrent you created initially (to upload) can be reused and uploaded to other trackers. I think the tracker modifies the copy of the .torrent file you upload to suit their needs (eg, to put a tracker url inside that corresponds to their service)... So don't think you can use the .torrent file until after you upload it to a tracker [but your bit torrent client may do this transparently.. I am guessing.. if the tracker works with anonymous seeders or something and you provided a tracker url when creating the .torrent ???$##$@^#%$%??].

When you click on the .torrent link, you need not be logged on. In fact, I think it's preferable not to be logged on. I wasn't, and the reason was because of something I read on the forums. Things may work out if you are logged on though. [To participate in a torrent that is already set up on linuxtracker.org, you need not have anything to do with linuxtracker; you only log on if you want to use them as a tracker for a torrent you want to set up.]

The info in the .torrent file, in conjunction with the tracker (whose url will be inside that file somewhere I believe), gives torrent clients all the info needed to start finding peers and uploading/downloading (u/l and d/l) the correct chunks.

So when seeding a file, the initial download will done instantaneously because the file is right on the hard drive and you were the only peer at that point in time. Those that come after will participate normally, getting chunks from you (if you are online) or from any other peer that has a chunk they are missing and is online participating in the torrent.

In case it wasn't clear, I have very limited experience specifically with bit torrents and P2P. I did, however, read some of the online links I posted and have read some on P2P networks elsewhere (and know some about networks and software in general). I also have at least acquired *something* and seeded *something* through KTorrent.

This all certainly makes me an expert , I know, but just to be safe, remember to bring the salt shaker.

Let's say you want to jumpstart a new torrent download by using prior downloaded data, eg, from an earlier different torrent (eg, different sized chunks) or from a server. Why would you want to do this? Well, so that you don't have to sit through an entire torrent. I mean, if you downloaded half the file from the server but the server is out of commission now or you think you can get a better download rate from a new torrent you saw, you may want to reuse the part of the file you already downloaded so as to save time and bandwidth. OR say you have a file that got corrupted (maybe you were playing around with dd or something), but you think it is still mostly in tact (you know it's corrupted because an md5sum of it gives a bad value, or it's a file you can view with an application and it looks corrupted).

How do you use this data to jumpstart a torrent download? Like this (using KTorrent as an example):

-- First, find a torrent that is of the file you want to complete or fix. I presume you already have the torrent link since that is presumably what motivated this exercise, but I'll state this step to be complete.

-- Next, make a backup of the partial (or corrupt) data you want to leverage in order to be safe. Why? Because it's possible to make a mistake somewhere and delete this data. If you really are doing this to save time and take advantage of a partial download (eg, if you have a slow Internet connection), why take chances? Just make the backup.. and give the backup a different name just to be safe (eg, if the backup is named the same but is in a different directory, you still might wipe it out a bit easier with KTorrent than if it was named differently).

-- Name one copy just like the file to be downloaded via the torrent. Ie, take the torrent link and keep the last portion. Then drop the ".torrent" ending. This is the name of the file. Eg, for the current running example, "LinpusLinux-LiveCD-i386.iso" would be the name of the file. This is how you would name your partial (or corrupt) file. If you aren't sure of the name, just start up the torrent (see step below) and see what new files appear in the directory. The new one w/o the .torrent ending is the name you want.

-- Copy this properly named file, ie, the partial download (or the corrupt file) you want to finish (or fix), to a clean directory with nothing else in it. [This step may not be necessary, but let's make our lives simple for now by eliminating variables.]

-- Make sure you are not currently downloading the torrent you want to download. [This step too may be optional, but just follow it. I gave up trying to test all possibilities I could think off, so I am describing one procedure that works.]

-- Start a torrent (eg, on KTorrent, File -> Paste torrent url, or perhaps just click on the torrent link from a website and open it with KTorrent). When prompted, save the torrent to the same directory where you have the partial (corrupt) data you want to complete (fix). [You made the back up, right?]

-- KTorrent will start performing a data integrity check if it finds a file with the same name as the torrent file to be downloaded. So it sees the partial download data that was named as expected; rebuilds it's knowledge of the chunks it has and the chunks it needs (that's the integrity check); and starts up the torrent download from the peers to fill in *only* the missing (ie, missing or corrupted) chunks! If you cancel the integrity check midway, KTorrent will just start downloading all the chunks I think (so you will not be saving time and will be redownloading data you already have stored in the proper place). However, I don't think it trashes the file. Thus I think you can stop/remove the torrent download, and try opening up the torrent again (this time allowing the integrity check to procede to the end). KTorrent just writes into the file at the proper location where a chunk fits. It doesn't truncate; thus, it will just write good data over good data or good data over bad data, but it won't clobber anything of value.. I think. In any case, if you do mess up or remove the .torrent file and data, you have that trusty backup and can restart from step one, right?

Any time you want, you can do an integrity check (which is a neat feature).

Also, you may find that you don't have to do all the steps above, but I make no guarantees. In fact, I make no guarantees that if you do follow the steps above exactly as described that you won't cause your computer and files to explode. Reader beware.

Anyone that wants to do more testing and report on it, go ahead.

One more thing, don't panic if KTorrent says something like torrent file or data is corrupt. That message can appear for any of numerous reasons. It just means KTorrent got confused with the torrent link you gave it (eg, I gave a bad url one time repeatedly and didn't realize it) or you (re)move the data file (ie, the file being downloaded) from where it expects it to be and it panicked. Also, if you remove the .torrent file without telling it or rename the directory or something like that, it may also produce that nasty sounding message or maybe even a worse one.

Other torrent apps besides KTorrent may have a similar feature. If not file a feature request

Happy downloading and hope this feature saves you time or is at least useful in some way.

[Years ago, before I ever tried a torrent app, I thought about hacking wget to implement a more flexible way to download so that I could download only precise chunks easily (using http partial request header). I never got around to finishing that project (or studying wget to see where to make the incision). Now, I can probably just use KTorrent or maybe link to a torrent library or something (I'll investigate the library possibility later). Goes to show you that if you react too slowly, the wheel will get implemented for you. How horrible, right? FOSS rulz!]