[libtorrent] Support large number of torrents

[Chen, Yung-Min]

Hi,

I am developing an application that shares files among a cluster of machines in a local network using Libtorrent. Each machine in the cluster may receive batches of files from external sources at any time. The goal is to have all files received in a batch on each machine automatically propagate to every other machine as soon as they are received, so that every machine will have the same set of files received on all machines within reasonable time.

My implementation creates one torrent for each batch of files received on a machine; the machine then seeds the torrent immediately for other machines to download. There is no user interaction; everything happens automatically. The torrent size varies from 25 Mbytes up to ~ 2GB each. The number of machines (peers) are not big (20~25), but the number of torrents (seeding and downloading) could be large on each machine since each machine receives a batch of files about every couple of minutes. I am using local service discovery and auto-managed session with default settings parameters.

The file distribution works very well at the beginning, but as time goes on some torrents take a few minutes to propagate to all systems in a Gbit network connection. I am new to Libtorrent and would like to solicit your advice on the session settings and tuning parameters I should set to resolve the slow distribution problem described above. Any suggestion will be very much appreciated.

Thanks in advance!
Yung

[Arvid Norberg]

Post by Chen, Yung-Min
Hi,
I am developing an application that shares files among a cluster of
machines in a local network using Libtorrent. Each machine in the cluster
may receive batches of files from external sources at any time. The goal is
to have all files received in a batch on each machine automatically
propagate to every other machine as soon as they are received, so that
every machine will have the same set of files received on all machines
within reasonable time.
My implementation creates one torrent for each batch of files received on
a machine; the machine then seeds the torrent immediately for other
machines to download. There is no user interaction; everything happens
automatically. The torrent size varies from 25 Mbytes up to ~ 2GB each. The
number of machines (peers) are not big (20~25), but the number of torrents
(seeding and downloading) could be large on each machine since each machine
receives a batch of files about every couple of minutes. I am using local
service discovery and auto-managed session with default settings parameters.

libtorrent scales quite well with many torrents. I regularly test with
100'000 torrents or so.

Post by Chen, Yung-Min
The file distribution works very well at the beginning, but as time goes
on some torrents take a few minutes to propagate to all systems in a Gbit
network connection. I am new to Libtorrent and would like to solicit your
advice on the session settings and tuning parameters I should set to
resolve the slow distribution problem described above. Any suggestion will
be very much appreciated.

when you say "some torrents take a few minutes to propagate", are you
referring to the .torrent files themeselves? (which I assume you've built
something to do), or are you talking about their content taking longer and
longer to be transferred to all nodes?

If it's the latter, please enable logging and understand where the time is
spent. My suspicion is that, with auto-managed torrents, the startup time
may not be negligible. You could try to make new torrents "force started",
i.e. make them not auto-managed, and once they complete, set the
auto-managed flag.

I believe currently, when adding a new torrent, it doesn't always have the
highest priority to start immediately. "recalculate auto managed torrents"
is called on a delay to better handle torrents added in batches, it's
possible this delay become noticeable to you, or that there is a
performance issue with it. If you discover any issues causing this delay to
be unreasonably long (I think it should be no more than 1 second), patches
are welcome!