It's been a while since I've given a status update for my Google Summer of Code project to create a BitTorrent proxy for downloading packages using APT, so here it is.
I've been working hard on integrating support for APT into the DebTorrent program. I've almost got it working perfectly, now it's just a matter of testing to make sure all is well. The functionality works like this:
- DebTorrent listens on a port for HTTP requests
- an http://localhost:port/mirror_name/debian/... entry is added to APT's sources.list file (similar to how apt-cacher works for proxying)
- an apt-get update will then send HTTP requests to the DebTorrent
program
- DebTorrent proxies these requests, downloading files it doesn't have and saving them in a cache before passing them on to APT
- DebTorrent recognizes requests for Packages files, and also uses them to start the torrents for those files
- an apt-get install will send an HTTP request to DebTorrent for a
package (.deb) file
- rather than getting package files from the mirror, DebTorrent finds the file in one of its running torrents and enables it for download
- the package file is downloaded (either from other peers, or using the backup HTTP downloader that gets it from the mirror if no peers can be found)
- once the download is complete, DebTorrent passes the package file to APT
There are two things I really like about this. One of the best is the backup HTTP downloader. It insures that if you're an early adopter and there are no peers, or if the package you're requesting is rare and can't be found in any connected peers, the download will still occur in a reasonable amount of time (taking no more, or less, mirror bandwidth than if you had just been using APT directly). The other thing I like is that you get the BitTorrent-style peer-to-peer downloading, with simple HTTP proxying thrown in for free. You can run DebTorrent on a single computer on your network, and have the others connect to it to initiate downloads and request packages.
I haven't done any serious time testing, but I estimate it currently only takes twice as long to use as a regular APT update and download from a mirror. Most of that slowdown is because it currently only processes a single request at a time from APT, which is not very efficient for BitTorrent systems where downloading from multiple peers is how the highest download speeds are achieved. I have been talking to the APT maintainer, Michael Vogt, about a better way to do this, probably by adding a new APT transport method for DebTorrent (i.e. debtorrent:// instead of http://). This will not only speed up the downloads, but also hopefully provide better feedback to the APT user, as currently it will seem nothing is happening until the download comes in all at once at the end.
My work has unfortunately been slowed by other commitments, and bugs. I have two papers to submit to a conference by Monday, but after that I should be back full time on DebTorrent. I also spent a long time tracking down and fixing a bug in the underlying BitTornado code (which lead to much rejoicing at 3am), only to find it was fixed in upstream's CVS. (Doh!)