[libtorrent] Slow download speeds

[Luboš Doležel]

Hi,

maybe I just missed something but I've noticed one problem.

I develop a GPLed download manager for Linux (fatrat.sf.net) and I use
libtorrent as a base library for Bittorrent support.

It is great when the torrent is well seeded. Download speed is high...it
works great.

But when there is a lot of ugly leech-only users the download speed sucks. I
understand the way the Bittorrent works but just Azureus downloads the same
torrent *much* faster.

My line's max download speed is 16 kB/s...Azureus gets over 10 kB/s in a
minute. Libtorrent never gets over 5 kB/s. The line is *not* slowed down
because of upload. It seems that Azureus has some better algorithm or
what...

Wait, I think I have found the source problem. Libtorrent tries to connect
about 50 people after it gets IPs from the tracker. After a few moments the
number of connected peers is about 20 (IP is unreachable etc.)...after 10
minutes it goes down to 10 (peer has closed connection etc.)

But it does not try to connect any more until it reannounces...why?! Azureus
connects 100 peers and tries to keep this numer high. I have not limited
libtorrent's max connections in any way.
I hopelessly tried to do torrent_handle::set_ratio(1) but no luck.

How to boost libtorrent to make it download faster (like Azureus)?

Sorry for such a long message ;-)

Thanks,
Lubos Dolezel

[Matt Sicker]

Post by Luboš Doležel
Hi,
maybe I just missed something but I've noticed one problem.
I develop a GPLed download manager for Linux (fatrat.sf.net) and I use
libtorrent as a base library for Bittorrent support.
It is great when the torrent is well seeded. Download speed is high...it
works great.
But when there is a lot of ugly leech-only users the download speed sucks. I
understand the way the Bittorrent works but just Azureus downloads the same
torrent *much* faster.
My line's max download speed is 16 kB/s...Azureus gets over 10 kB/s in a
minute. Libtorrent never gets over 5 kB/s. The line is *not* slowed down
because of upload. It seems that Azureus has some better algorithm or
what...
Wait, I think I have found the source problem. Libtorrent tries to connect
about 50 people after it gets IPs from the tracker. After a few moments the
number of connected peers is about 20 (IP is unreachable etc.)...after 10
minutes it goes down to 10 (peer has closed connection etc.)
But it does not try to connect any more until it reannounces...why?! Azureus
connects 100 peers and tries to keep this numer high. I have not limited
libtorrent's max connections in any way.
I hopelessly tried to do torrent_handle::set_ratio(1) but no luck.
How to boost libtorrent to make it download faster (like Azureus)?
Sorry for such a long message ;-)
Thanks,
Lubos Dolezel

Is it possible that the decentralised tracker is in effect for Azureus? If so,
disable that and check for that as well. Also, did you try to set the minimum
peers setting?

[Luboš Doležel]

Post by Matt Sicker
Also, did you try to set the minimum
peers setting?

I haven't found such function in documentation :-(

Luboš Doleşel

[Arvid Norberg]

Post by Luboš Doležel
Hi,
maybe I just missed something but I've noticed one problem.
I develop a GPLed download manager for Linux (fatrat.sf.net) and I
use libtorrent as a base library for Bittorrent support.
It is great when the torrent is well seeded. Download speed is
high...it works great.
But when there is a lot of ugly leech-only users the download speed
sucks. I understand the way the Bittorrent works but just Azureus
downloads the same torrent *much* faster.
My line's max download speed is 16 kB/s...Azureus gets over 10 kB/s
in a minute. Libtorrent never gets over 5 kB/s. The line is *not*
slowed down because of upload. It seems that Azureus has some
better algorithm or what...
Wait, I think I have found the source problem. Libtorrent tries to
connect about 50 people after it gets IPs from the tracker. After a
few moments the number of connected peers is about 20 (IP is
unreachable etc.)...after 10 minutes it goes down to 10 (peer has
closed connection etc.)
But it does not try to connect any more until it
reannounces...why?! Azureus connects 100 peers and tries to keep
this numer high. I have not limited libtorrent's max connections in
any way.
I hopelessly tried to do torrent_handle::set_ratio(1) but no luck.
How to boost libtorrent to make it download faster (like Azureus)?

If you find something out that may cause this I'm more than willing
to try it.

It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.

If there are many other azureus peers in the swarm, it is also
possible that they exchange peers with each other. As far as I
understand, azureus uses it's own protocol to talk to other az clients.

As Matt already pointed out, the dht in azureus may also have an
effect here. The plan is to add support for the mainline dht in
libtorrent when I'm finished with the things I'm working on right now.

But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.

Post by Luboš Doležel
Sorry for such a long message ;-)

--
Arvid Norberg

[Philippe Gagnon]

While I understand your concerns about respecting the specifications
and the other peers, sometimes it's not always the right thing to do.
I totally agree here about your decision not to hammer trackers by
using their proposed reannounce interval.

But I disagree with the philosophy around peer wire communication. It
might do more good to the torrent in general to nudge peers into
keeping in touch with the client. From my point of view, while it
wouldn't be as nice to said peers bandwidth, it might do more good to
both the client (for the down speed would be faster) and the swarm in
general (since a peer with a higher download speed is a peer that
acquires data faster, thus, more distributed copies / an eventual seed
faster) .

Also DHT only kicks in if Azureus fails to connect to the given
tracker for a torrent, so I guess it's not a real issue when comparing
download speeds with libtorrent.

P.S. I have noticed that forcing the share ratio to 1.2 seems to
increase my leeching speed drastically. Might want to try this out.

Post by Arvid Norberg

Post by Luboš Doležel
Hi,
maybe I just missed something but I've noticed one problem.
I develop a GPLed download manager for Linux (fatrat.sf.net) and I
use libtorrent as a base library for Bittorrent support.
It is great when the torrent is well seeded. Download speed is
high...it works great.
But when there is a lot of ugly leech-only users the download speed
sucks. I understand the way the Bittorrent works but just Azureus
downloads the same torrent *much* faster.
My line's max download speed is 16 kB/s...Azureus gets over 10 kB/s
in a minute. Libtorrent never gets over 5 kB/s. The line is *not*
slowed down because of upload. It seems that Azureus has some
better algorithm or what...
Wait, I think I have found the source problem. Libtorrent tries to
connect about 50 people after it gets IPs from the tracker. After a
few moments the number of connected peers is about 20 (IP is
unreachable etc.)...after 10 minutes it goes down to 10 (peer has
closed connection etc.)
But it does not try to connect any more until it
reannounces...why?! Azureus connects 100 peers and tries to keep
this numer high. I have not limited libtorrent's max connections in
any way.
I hopelessly tried to do torrent_handle::set_ratio(1) but no luck.
How to boost libtorrent to make it download faster (like Azureus)?

If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.
If there are many other azureus peers in the swarm, it is also
possible that they exchange peers with each other. As far as I
understand, azureus uses it's own protocol to talk to other az clients.
As Matt already pointed out, the dht in azureus may also have an
effect here. The plan is to add support for the mainline dht in
libtorrent when I'm finished with the things I'm working on right now.
But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.

Post by Luboš Doležel
Sorry for such a long message ;-)

--
Arvid Norberg
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id865&opclick
_______________________________________________
Libtorrent-discuss mailing list
https://lists

[Luboš Doležel]

Post by Arvid Norberg
If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.

torrent_handle::force_reannounce temporarily helps, but Azureus doesn't
announce more often. The problem lays somewhere else.

Good example: there is a very popular torrent, it had over 30000 peers
yesterday (yes, 30000) and after every announce libtorrent tries to connect
cca 50 random peers, with 20% success. But what about those 30000 others?
Libtorrent won't try to connect any of them - it will just try another 50
after next annnounce.
If you are interested in the torrent itself, it was "Lost s02e10"...


But I will keep this in mind and have a look if there's some way to

Post by Arvid Norberg
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.

Disconnected peers are probably dead anyway, it won't boost the connected
peer number...

[Arvid Norberg]

Post by Arvid Norberg
If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.
torrent_handle::force_reannounce temporarily helps, but Azureus
doesn't announce more often. The problem lays somewhere else.
Good example: there is a very popular torrent, it had over 30000
peers yesterday (yes, 30000) and after every announce libtorrent
tries to connect cca 50 random peers, with 20% success. But what
about those 30000 others? Libtorrent won't try to connect any of
them - it will just try another 50 after next annnounce.
If you are interested in the torrent itself, it was "Lost s02e10"...

Most trackers limits the number of peers they send to 50. That means
the client doesn't know about the other 29950 peers.

Post by Arvid Norberg
But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.
Disconnected peers are probably dead anyway, it won't boost the
connected peer number...

That's how I've been reasoning too.

I will keep invastigating this though, to see if it can be made more
efficient.

--
Arvid Norberg

[M***@massaroddel.de]

Post by Arvid Norberg

Post by Arvid Norberg
If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.
torrent_handle::force_reannounce temporarily helps, but Azureus
doesn't announce more often. The problem lays somewhere else.
Good example: there is a very popular torrent, it had over 30000
peers yesterday (yes, 30000) and after every announce libtorrent
tries to connect cca 50 random peers, with 20% success. But what
about those 30000 others? Libtorrent won't try to connect any of
them - it will just try another 50 after next annnounce.
If you are interested in the torrent itself, it was "Lost s02e10"...

Most trackers limits the number of peers they send to 50. That means
the client doesn't know about the other 29950 peers.

Post by Arvid Norberg
But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.
Disconnected peers are probably dead anyway, it won't boost the
connected peer number...

That's how I've been reasoning too.
I will keep invastigating this though, to see if it can be made more
efficient.

Even if you do not connect to those peers - they should connect to you ove=
r time.
Your peer count should increase anyway. Do you have incomming connections?
On popular torrents i always reach my peer limit (600). And with the way l=
ibtorrent
behaves - 90% are incomming connection. I think this is caused by the fact=
that libtorrent
does not reconnect if it gets disconnected by the peer. But it does not ma=
tter much
because the peer will connect back to me sooner or later.

MassaRoddel

[Luboš Doležel]

Post by M***@massaroddel.de
Even if you do not connect to those peers - they should connect to you
over time.
Your peer count should increase anyway. Do you have incomming connections?
On popular torrents i always reach my peer limit (600). And with the way
libtorrent
behaves - 90% are incomming connection. I think this is caused by the fact
that libtorrent
does not reconnect if it gets disconnected by the peer. But it does not
matter much
because the peer will connect back to me sooner or later.

Yeah, they should and they would but...
I live in a small town -> public IPs are too expensive here (only one ISP ->
big prices). So I have to suffer behind NAT like most WiFi users in this
area.
Looking forward to IPv6...

Luboš Doleşel

[M***@massaroddel.de]

Even if you do not connect to those peers - they should connect to y=

ou over time.

Your peer count should increase anyway. Do you have incomming connec=

tions?

On popular torrents i always reach my peer limit (600). And with the=

way libtorrent

behaves - 90% are incomming connection. I think this is caused by th=

e fact that libtorrent

does not reconnect if it gets disconnected by the peer. But it does =

not matter much

because the peer will connect back to me sooner or later.
Yeah, they should and they would but...
I live in a small town -> public IPs are too expensive here (only one IS=

P -> big prices). So I have

to suffer behind NAT like most WiFi users in this area.
Looking forward to IPv6...
Lubo=C5=A1 Dole=C5=BEel

In that case, can you forward your listen port to the router? (assuming on=
ly one NAT
level) AZ might do this automatically.

[Luboš Doležel]

Post by M***@massaroddel.de
In that case, can you forward your listen port to the router? (assuming
only one NAT
level) AZ might do this automatically.

No I can't. The router is not UPnP enabled and the ISP doesn't officially
provide port forwarding.
But the admin is quite good, I'll ask him.

Lubos Dolezel

[Alejandro Santos]

If every client behaves like that, waiting for other peers to connect
to you, then no one would connect no anyone and there would be no
peers to download data from nor upload data to.

I don't know how azureous nor other clients work, but i belive they
must have some sort of reconnection algorithm. For example, if a peer
closes the connection then wait some time interval and try to
reconnect to it, retrying up to a fixed number of times with an
appropiate interval between retrys. If the time interval and the
maximum retry count are fair enough, that won't be hammering.

Alejandro.

Post by Arvid Norberg

Post by Arvid Norberg
If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.
torrent_handle::force_reannounce temporarily helps, but Azureus
doesn't announce more often. The problem lays somewhere else.
Good example: there is a very popular torrent, it had over 30000
peers yesterday (yes, 30000) and after every announce libtorrent
tries to connect cca 50 random peers, with 20% success. But what
about those 30000 others? Libtorrent won't try to connect any of
them - it will just try another 50 after next annnounce.
If you are interested in the torrent itself, it was "Lost s02e10"...

Most trackers limits the number of peers they send to 50. That means
the client doesn't know about the other 29950 peers.

Post by Arvid Norberg
But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.
Disconnected peers are probably dead anyway, it won't boost the
connected peer number...

That's how I've been reasoning too.
I will keep invastigating this though, to see if it can be made more
efficient.

Even if you do not connect to those peers - they should connect to you over time.
Your peer count should increase anyway. Do you have incomming connections?
On popular torrents i always reach my peer limit (600). And with the way libtorrent
behaves - 90% are incomming connection. I think this is caused by the fact that libtorrent
does not reconnect if it gets disconnected by the peer. But it does not matter much
because the peer will connect back to me sooner or later.
MassaRoddel
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id865&opclick
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/list

[M***@massaroddel.de]

Post by Alejandro Santos
If every client behaves like that, waiting for other peers to connect
to you, then no one would connect no anyone and there would be no
peers to download data from nor upload data to.

You try to connect to all peers but if those peers disconnect you - you can assume a
reason why they did so. And you try to connect to them again anyway after the tracker
gives you the same peers again some announce intervalls later.

[Alejandro Santos]

As you say, there are many reasons a peer might disconnect from you.
There is no way to know the truth. But after 10,15 or 20 minutes the
situation might change. If after 2 retries the peer disconnects from
you, that should be enough to assume the peer have no interest in
keeping your connection.

Post by M***@massaroddel.de

Post by Alejandro Santos
If every client behaves like that, waiting for other peers to connect
to you, then no one would connect no anyone and there would be no
peers to download data from nor upload data to.

You try to connect to all peers but if those peers disconnect you - you can assume a
reason why they did so. And you try to connect to them again anyway after the tracker
gives you the same peers again some announce intervalls later.
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

--
Alejandro.

[Daniel Wang]

force_reannounce ontimer can ALWAYS help the quick speeds.
we can add this force_reannounce_ontimer(interval) function in
libtorrent.
so in high UI application can solve this.and the libtorrent follows
the BITTORRENT protocal standard all the same.

Post by Arvid Norberg
If you find something out that may cause this I'm more than willing
to try it.
It is true that libtorrent will only announce itself with the tracker
using the interval given by the tracker. This was mostly done with
some respect of tracker owners, and becuase that's what the protocol
says. I believe that azureus may do more announces. You could try to
do a torrent_handle::force_reannounce() to see if that would help.
torrent_handle::force_reannounce temporarily helps, but Azureus
doesn't announce more often. The problem lays somewhere else.
Good example: there is a very popular torrent, it had over 30000
peers yesterday (yes, 30000) and after every announce libtorrent
tries to connect cca 50 random peers, with 20% success. But what
about those 30000 others? Libtorrent won't try to connect any of
them - it will just try another 50 after next annnounce.
If you are interested in the torrent itself, it was "Lost s02e10"...

Most trackers limits the number of peers they send to 50. That means
the client doesn't know about the other 29950 peers.

Post by Arvid Norberg
But I will keep this in mind and have a look if there's some way to
keep the number of connected peers high. libtorrent will not try to
reconnect to a peer that disconnected either, because it will assume
the other end disconnected for a reason. It basically tries to play
nicely instead of hammering.
Disconnected peers are probably dead anyway, it won't boost the
connected peer number...

That's how I've been reasoning too.

I will keep invastigating this though, to see if it can be made more
efficient.

--
Arvid Norberg




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_idv37&alloc_id865&op=click
_______________________________________________
Libtorrent-discuss mailing list
Libtorrent-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss




---------------------------------
Yahoo! Photos – Showcase holiday pictures in hardcover
Photo Books. You design it and we’ll bind it!

[Luboš Doležel]

Post by Daniel Wang
force_reannounce ontimer can ALWAYS help the quick speeds.
we can add this force_reannounce_ontimer(interval) function in
libtorrent.
so in high UI application can solve this.and the libtorrent follows
the BITTORRENT protocal standard all the same.
**

Hello,

I've tried to boost the download speed by force_reannounce every 10 minutes.
I helped to keep the peer number high.
But the only result is UL/DL ratio almost 4 and only 90 MiB downloaded after
8 hours.
Upload was still at the limit I've set.

I can't explain this behavior...

Luboš Doleşel

[Arvid Norberg]

Post by Luboš Doležel
Hello,
I've tried to boost the download speed by force_reannounce every 10
minutes. I helped to keep the peer number high.
But the only result is UL/DL ratio almost 4 and only 90 MiB
downloaded after 8 hours.
Upload was still at the limit I've set.
I can't explain this behavior...

The reasons behind upload and download ratios is pretty complex. I
believe the biggest influence might be if you are behind a NAT or
not, and then how common it is to be behind a NAT in the swarm. The
more peers you have, the better are the chances that the best of them
are better. Also, if you have a symmetric internet connection, many
peers will download faster from you than they can upload to you
because they have asymmetric connections.

I have also experienced the situation where you participate in the
torrent for hours, only uploading and the download is crawling.
Without being an expert in the field, my guess would be that this
could be the effect of a torrent whose swarm is growing too fast,
being too popular. It could also be the effect of a very slow seed so
that people drop off after having 50% download.

I don't know though, there are probably millions of explanations.

--
Arvid Norberg

[Luboš Doležel]

Post by Arvid Norberg
The reasons behind upload and download ratios is pretty complex. I
believe the biggest influence might be if you are behind a NAT or
not, and then how common it is to be behind a NAT in the swarm. The
more peers you have, the better are the chances that the best of them
are better. Also, if you have a symmetric internet connection, many
peers will download faster from you than they can upload to you
because they have asymmetric connections.

Yes, I am behind NAT.

I have also experienced the situation where you participate in the

Post by Arvid Norberg
torrent for hours, only uploading and the download is crawling.
Without being an expert in the field, my guess would be that this
could be the effect of a torrent whose swarm is growing too fast,
being too popular. It could also be the effect of a very slow seed so
that people drop off after having 50% download.
I don't know though, there are probably millions of explanations.
--
Arvid Norberg

Well, Azureus has downloaded 90 MB in 2 hrs (the same torrent). Is there
some way to make libtorrent more agressive - to send more piece requests?

Luboš Doleşel