This feed contains pages in the "Debian" category. All these posts relate somehow to the Debian project and the work that I do with them.

I have just uploaded an important update to apt-p2p that is highly recommended for all users.

At some point over the last 48 hours bittorrent nodes have started to infiltrate the apt-p2p DHT (you may have noticed many ValueError tracebacks in your log file). The (mainline) bittorrent DHT is very similar in protocol to apt-p2p, and so the nodes are able to partially communicate and pollute each others routing table. I didn't think this would ever happen, as there wouldn't seem to be a reason for them to ever come into contact, but somehow it did.

I've made some minor changes that exploit the differences between the two protocols to exclude bittorrent nodes from the apt-p2p routing table, and to drop any requests from bittorrent nodes (which should prevent apt-p2p nodes from polluting the bittorrent DHT). However, it is very important that all users upgrade to this new version to prevent any further mixing of the DHTs.

Posted Sun Jun 22 07:24:45 2008 Tags:

This new version of DebTorrent finalizes the work I previously uploaded to experimental. This new version is the first in unstable to use unique piece numbers to create long-lived torrents for testing and unstable. It also comes with a new version of the apt-transport-debtorrent package, which can now tell APT not to cache any package files downloaded with DebTorrent, which should save some disk space for some people.

The locations of the long-lived torrent files and piece files that DebTorrent uses has also changed, as they are now hosted on my merkel page and should be more up to date. However, I was not able to get the ssh trigger for the sync from ftp-master setup, nor am I able to access the projectb database through pygresql. If anyone knows how I can get those working, please email me.

Here is the changelog for debtorrent:

  • Display the torrent identifier on the local status page rather than the info hash (Closes: #465339)
  • Add support for the new No Content message to apt-transport-debtorrent so that apt treats debtorrent sources as local
  • Make the pieces and unique piece number locations config options, and use the new ones at
  • Decompress the files needed to create torrents so they are created and started faster (this and the previous Closes: #463676)

And for apt-transport-debtorrent:

  • Add a config option to make debtorrent sources local (Closes: #477383)
  • Upgrade the recommends on debtorrent to 0.1.7.
  • Add a config file for APT to load, and a manpage describing the options.
Posted Fri Jun 20 07:30:59 2008 Tags:

I have updated apt-p2p to version 0.1.3, which is available now in unstable and my repository (see my previous post introducing apt-p2p).

This update brings mostly speed improvements to finding peers to download from, and performing the download. Nodes in the DHT are now checked for responsiveness more thoroughly before adding them to the routing table, and rechecked more frequently afterwards. This should prevent bad nodes from propagating through the system, and departing nodes remaining in the system long after they have left. Downloading from peers is also quicker, as bad peers are dropped with a quicker timeout value and after a small number of errors. There was also a bug revealed by about 10% of the mirrors that caused downloads to fail, and was fixed in version 0.1.2.

This update is STRONGLY recommended, as keeping unresponsive nodes out of the DHT will make everyone's experience better. If you tried apt-p2p and had some problems, I hope you'll consider trying again now. Unfortunately, these are enhancements I could not predict needing before releasing to the public, as the problems they solve are all caused by having a larger number of users, some of which are unresponsive.

There are still a large number of peers (maybe 50% of them) that are unreachable, and so can not share any files with other peers. I don't have a good way to check yet, but you can go to sites such as this one to probe your client for you. Just enter the remote IP address and port (available on the status page) of your client in the form http://ipaddress:port/ (e.g. If the check returns any HTTP headers (even a 404 Not Found response) then you're fine, but if it doesn't return anything then your peer is firewalled or NATted and should probably be fixed.

Here are the changelog entries for 0.1.2 and 0.1.3:

  • Speed up downloading from peers
    • Set a new peer's ranking values so they don't get an unfair advantage.
    • Reduce the HTTP connection timeout to 10s.
    • Drop peers after a limited number of errors.
  • Speed up the DHT requests when nodes fail
    • Schedule a re-ping message after adding a new node.
    • When a node fails, schedule a future ping to check again.
    • Send periodic finds to nodes that are stale.
    • Increase the stored value redundancy to 6.
    • Increase the concurrency of DHT requests to 8.
    • Add early termination and ignoring slow responses to recursive DHT actions when timeouts occur.
  • Remove the debconf note about port forwarding (Closes:#479492)
    • Add a NEWS entry for port forwarding
  • Fixed a bug in the HTTP downloader that caused errors with some mirrors that always close the connections (Closes: #479455)
Posted Sat May 10 16:28:12 2008 Tags:

After many long months of planning and work, I have completed another peer-to-peer downloader for Debian. If you've been keeping track, that makes 2 now. This one is called apt-p2p, and as of yesterday it is available in unstable.

The functionality is very similar to the first one I wrote, DebTorrent, so if you've used that one you should feel very comfortable. After installation you just add a localhost:9977 to the start of your sources.list entries (see the man page). The only difference is the port number (DebTorrent uses 9988 by default) and that you can use it on all your sources.list entries, whether they be deb, deb-src, official Debian archive, or any other archive. Then an apt-get update gets it started, and you can begin installing packages. Point your web browser to http://localhost:9977 to see what's going on.

IMPORTANT: as with any P2P program, it works much better if you open a port through your NAT or firewall. Without this crucial step, you won't be able to share with any peers, and your lookup of peers to download from may take longer. Make sure to forward both TCP and UDP ports 9977, or whichever port you set in the config file. For more details, see the port forwarding section of the FAQ.

The similarities with DebTorrent are all external, so let's look at how it differs from DebTorrent internally:

  • it's very general, it doesn't matter what you're trying to download, a source package or a Packages file, from constantly-updated unstable or a year-old stable, for i386 or hppa architectures (DebTorrent only works for .deb files, is only supported for the official archive, and breaks downloaders into groups by architectures)
  • it doesn't require anything other than what's available to apt (DebTorrent uses piece hashes of large files, and ordering information, both of which are downloaded separately)
  • it can be very fast when downloading from mirrors (mirror downloads with DebTorrent are not so fast)
  • the code is simple, and makes use of available code, such as Twisted, Khashmir, and python-apt, which should make future enhancements and maintenance easier (DebTorrent is large and monolithic)
  • requires less memory and CPU power (50% to 75% less memory than DebTorrent)

Here's some technical details for those interested:

  • makes use of hashes to uniquely identify files
  • uses a Distributed Hash Table (DHT) to find peers
  • also stores piece hashes in the DHT for efficient downloading
  • uses HTTP/1.1 requests to download from peers
  • no peers available causes it to fallback to a download from the mirror

Though I know it can be fast, I'm not yet sure if the peer lookup in the DHT will be quick enough to keep up with the downloading. All my tests so far show that it is, but until there are a number of peers out there trying it, I can't be sure. Also, I have some improvements in mind to enhance the speed, in particular the wait for a timeout to occur, so this may improve in the future. If you see a delay in downloading where apt seems to be stalled saying 'Waiting for headers', for now be patient and see what happens. It may be that apt-p2p is downloading in the background (it does this sometimes), or that it's waiting for a lookup in the DHT to complete. If it hangs for more than a minute, or there's errors in the log file, please file a bug so that I can look into it.

Finally, for you DebTorrent fans don't worry, I haven't given up on it. Stay tuned for more info on it coming soon.

Posted Sun May 4 07:10:26 2008 Tags:

I have uploaded a new version of DebTorrent to experimental and to my personal repository. This new version implements the unique piece numbers, which has been planned for quite some time now. Well, it's finally here.

To summarize, unique piece numbers keep torrent's alive longer, by assigning files unique piece numbers that never get reused in that torrent. New files get new piece numbers added to the end, but peers in the old torrent can still share most of the old files with peers in the new torrent (in fact, it's the same torrent, but old and new peers have different ideas about what it contains).

The creation of torrents from Packages files has also changed, as now 2 torrents are created, one for whatever arch the Packages file was for, and one for the Architecture:all files. Since the Arch:all files are the same for all architectures, this also allows for more sharing of common files between peers on different architectures. This change required a change in the cache directories, which is described in more detail in the NEWS file, but is handled almost automatically. The only thing to do is the make sure to do an apt-get update after upgrading so that the torrents can be restarted.

There was also some changes in the statistics reporting. In the client, the uploaded and downloaded statistics will now persist over restarts so that you can see how much you've done over a longer period. The tracker status page also got an update, and now shows the total uploaded and downloaded bytes for each torrent, as well as some more descriptive names for some of the torrents.

Due to the large changes in this release, I opted for uploading to experimental (and my repository) for now, so that it can be tested a bit. Please do test it, and let me know of any bugs, problems or concerns that you may have.

Here is the changelog:

  • Add support for unique piece numbers
    • increases duration of oft-updated torrents so that more peers can participate
    • currently supported only by debian testing and unstable
    • see for more info
  • Switch to using 2 torrents per Packages file: one for architecture- specific files, and one for architecture-independent files
    • also added a new script splitcachefor_all to ease the upgrade
  • Use python-debian for all reading of RFC 822 type files
    • also requires python-apt
  • Add torrent names to the tracker display
  • Make the download/upload statistics persist over restarts
  • Report more and better statistics on the tracker's info page
Posted Mon Jan 28 07:17:30 2008 Tags:

It's been a while, but a new update to DebTorrent is available in unstable. There is also a new version of the helper apt-transport-debtorrent (0.2.0) to go with it, which was mostly the cause of the long delay (due to a pending apt transition to testing). It is now highly recommended to install both, read below for new reasons why.

One of the previous problems with debtorrent was the incorrect status updates displayed by apt, due to it not being aware of pieces of large files that have already downloaded. Using these new versions together will help to improve the update status messages shown during a download. Now, apt-get update will display a status at the bottom that will look something like:

DebTorrent: 837MiB left at 319 KiB/s (46m03s)

You may not be able to see it all if there are multiple updates to different mirrors under way. In aptitude, the status will appear on a line by itself at the bottom of the list of downloaded files. The status line will update about once per second, and should give a good indication of how much time is really left. The other (incorrect) status information presented by apt/aptitude will still be there but you can ignore it. I've only tested apt and aptitude, so I'm not sure how (or if) synaptic, adept or gnome-apt will display these messages, though they are general apt status messages, usually used for displaying the status while logging into an FTP server.

This should also fix another problem that debtorrent can have with downloading large files (usually on slower connections). Since apt can't see the download happening for a long time, the connection would sometimes time out. With these status updates the connection is constantly active, so the timeouts shouldn't be a problem when downloading large packages (although they may still occur when doing an apt-get update, as there are no status updates during that process).

I did implement a better solution, as suggested by MichaƂPolitowski on the wiki. This solution uses the creation of sparse files to let apt know how much of a file has actually been downloaded. However, apt doesn't understand how a single process can download multiple files in prarallel, so it only follows the last file that was started, and complains when a file finishes that was not the last one that started. I've disabled that for now while I see if it's possible to modify apt to work better in this situation.

Here are the changes for debtorrent:

  • Update to support apt debtorrent transport version 0.2
    • send piece downloaded status messages (currently disbaled due to apt not liking it)
    • send general status update messages
  • Fix some minor packaging issues
  • Upgrade to standards version 3.7.3 (no changes)
  • Remove the unneeded binary-arch rule
  • Changes the XS-Vcs-* headers to Vcs-*
  • Moved Homepage from description to Source package fields

And for apt-transport-debtorrent:

  • Upgrade transport version to 0.2
    • Create sparse files based on status messages (102) from DebTorrent
    • Send general status updates to apt (103) from DebTorrent
  • Remove support for Range headers as they may confuse the sparse file allocation
  • Fix typo in long description
  • Upgrade to standards version 3.7.3 (no changes)
  • Changes the XS-Vcs-* headers to Vcs-*
  • Moved Homepage from description to Source package fields
Posted Thu Jan 10 06:22:43 2008 Tags:

Since the conclusion of the Google Summer of Code, the DebTorrent project has been without a sponsor. I was going to email debian-mentors to see if anyone was interested, but I decided to first post to my blog to see if there are any Debian Developers reading this who have the time and are especially interested in sponsoring it.

Here are some of the details of the package:

It is currently maintained using an svn-buildpackage subversion repository, but I might be interested in switching to git if a sponsor is interested.

Posted Sun Oct 14 22:01:25 2007 Tags: