Discussion:
[libtorrent] Web seed request size
Artem Petrov
2015-12-11 10:23:05 UTC
Permalink
Hi! Arvid, thank you for fast reply.

I'm using libtorrent 1.0.7, and it is released from RC_1_0 branch.

Behavior, which you had described in previous mail is valid for master
branch, however for RC_1_0 piece picker works in a different way: in case
when almost all pieces are downloaded and the other parts are in
downloading state piece picker will return some random block from list of
downloading blocks, which was already requested by some other connection.
As result, this single block will be requested from web seed.

Do you have plan and when is it supposed to release desired behaviour?

P.S. The previous private mails are below separated with lines.

Best Regards,
Artem

----------------------------------------------------

The short answer is, that is a good idea and I would expect it to already
be the case. I assume you have some empirical experience of it not being
the case, which probably means there's a bug.

from memory, this is how I recall it to work:

when prefer whole pieces is enabled, and the preferred size is greater than
a piece, the piece picker will primarily look for pieces that have no
requests to them. If it finds one, it will then try to expand to pieces
before it and pieces after it, and requests those.

If all pieces have requests to them, it will start looking for the highest
priority piece and pick as many contiguous blocks as possible from it.

Once the piece picker is done, the actual coalescing of adjacent block
requests happens in web_peer_connection::write_request().

If you'd like to look into the case you have in mind, I would strongly
encourage you to take a look at test/test_piece_picker.cpp. There's a
section that exercises the contiguous block logic, but I'm sure there are
cases not covered there. Having a test case that expose the problem you're
seeing is the best way of having it fixed and stay fixed.

--------------------------------------------------------

Hello,

I'm interesting in optimizing block sizes which are requested
from web seeds. I took a look at the thread
https://sourceforge.net/p/libtorrent/mailman/message/34373739/.
<https://sourceforge.net/p/libtorrent/mailman/message/34373739/>
prefer_whole_pieces
logic really gives a nice advantage by requesting all pieces at once, but
in case when one 16Kb block from a piece was obtained from other peer type
all the remaining piece will be downloaded via 16Kb blocks, no matter from
what peer.

So the question is: is it good idea to modify piece picker to request the
largest possible block of partially downloaded piece from peer with
prefer_whole_pieces=1?

E.g. if the first 16 Kb block of 2 Mb piece was downloaded from any peer we
can make range request for downloading remaining 2032 Kb at once
from web peer.

Do you see some side effects of this "improvement"?
------------------------------------------------------------------------------
Arvid Norberg
2015-12-20 01:20:55 UTC
Permalink
Post by Artem Petrov
Hi! Arvid, thank you for fast reply.
I'm using libtorrent 1.0.7, and it is released from RC_1_0 branch.
Behavior, which you had described in previous mail is valid for master
branch, however for RC_1_0 piece picker works in a different way: in case
when almost all pieces are downloaded and the other parts are in
downloading state piece picker will return some random block from list of
downloading blocks, which was already requested by some other connection.
As result, this single block will be requested from web seed.
This is the master behavior as well. This is the end-game mode in
bittorrent.

When the piece picker fails to find a free block to request from a peer, it
will request exactly one 16 kiB block that's already being downloaded from
another peer.

It won't pick another end-game-mode block until the previous one is
completely downloaded (i.e. it won't pipeline end-game blocks). This should
keep the wasted download within bounds. The end game mode normally wouldn't
last very long though. The peers(s) that requested those blocks in the
first place pipelined them, and in this case those peers are likely stalled.

Do you experience continuous end-game mode? if so, do you know why?
Post by Artem Petrov
Do you have plan and when is it supposed to release desired behaviour?
Is it possible that web seeds behave differently in end-game mode?
--
Arvid Norberg
------------------------------------------------------------------------------
Artem Petrov
2016-01-04 15:27:57 UTC
Permalink
On Fri, Dec 11, 2015 at 5:23 AM, Artem Petrov <artempetroff <at>
gmail.com>
Post by Artem Petrov
Hi! Arvid, thank you for fast reply.
I'm using libtorrent 1.0.7, and it is released from RC_1_0 branch.
Behavior, which you had described in previous mail is valid for master
branch, however for RC_1_0 piece picker works in a different way: in case
when almost all pieces are downloaded and the other parts are in
downloading state piece picker will return some random block from list of
downloading blocks, which was already requested by some other connection.
As result, this single block will be requested from web seed.
This is the master behavior as well. This is the end-game mode in
bittorrent.
When the piece picker fails to find a free block to request from a peer, it
will request exactly one 16 kiB block that's already being downloaded from
another peer.
It won't pick another end-game-mode block until the previous one is
completely downloaded (i.e. it won't pipeline end-game blocks). This should
keep the wasted download within bounds. The end game mode normally wouldn't
last very long though. The peers(s) that requested those blocks in the
first place pipelined them, and in this case those peers are likely stalled.
Do you experience continuous end-game mode? if so, do you know why?
Post by Artem Petrov
Do you have plan and when is it supposed to release desired
behaviour?
Is it possible that web seeds behave differently in end-game mode?
Hi, thanks for reply.

Regarding endgame mode - it is not continuous for me. But I had made
some measurements on 165 MB file (there was 5 web seeds and non-web
seed, piece length - 2 MB) as result, I had the following results (from
libtorrent logs):
Total requests count: 987
Total 16 kiB requests count: 885 (4 in endgame mode)

As you can observe - there are a lot of 16 kiB requests even not in
endgame mode.

When we have only web seeds - numbers are the next:
Total requests count: 156
Total 16 kiB requests count: 76 (76 in endgame mode)

So, single blocks are requested from web seeds even in normal mode.
I think that other behaviour (e.g. whole combining blocks) in end game
mode for web seeds is dangerous.
But according to numbers above: seems that problem is not in the end
game mode.
Do you a suggestion what are other reasons for that?

Thanks you in advance.




------------------------------------------------------------------------------
Arvid Norberg
2016-01-05 06:44:09 UTC
Permalink
There may be an issue with the affinity to pick whole pieces vs. the
affinity to pick form already started pieces. I'll have to look into it
some more.
Post by Artem Petrov
On Fri, Dec 11, 2015 at 5:23 AM, Artem Petrov <artempetroff <at>
gmail.com>
Post by Artem Petrov
Hi! Arvid, thank you for fast reply.
I'm using libtorrent 1.0.7, and it is released from RC_1_0 branch.
Behavior, which you had described in previous mail is valid for
master
Post by Artem Petrov
branch, however for RC_1_0 piece picker works in a different way: in
case
Post by Artem Petrov
when almost all pieces are downloaded and the other parts are in
downloading state piece picker will return some random block from
list of
Post by Artem Petrov
downloading blocks, which was already requested by some other
connection.
Post by Artem Petrov
As result, this single block will be requested from web seed.
This is the master behavior as well. This is the end-game mode in
bittorrent.
When the piece picker fails to find a free block to request from a
peer, it
will request exactly one 16 kiB block that's already being downloaded
from
another peer.
It won't pick another end-game-mode block until the previous one is
completely downloaded (i.e. it won't pipeline end-game blocks). This
should
keep the wasted download within bounds. The end game mode normally
wouldn't
last very long though. The peers(s) that requested those blocks in the
first place pipelined them, and in this case those peers are likely
stalled.
Do you experience continuous end-game mode? if so, do you know why?
Post by Artem Petrov
Do you have plan and when is it supposed to release desired
behaviour?
Is it possible that web seeds behave differently in end-game mode?
Hi, thanks for reply.
Regarding endgame mode - it is not continuous for me. But I had made
some measurements on 165 MB file (there was 5 web seeds and non-web
seed, piece length - 2 MB) as result, I had the following results (from
Total requests count: 987
Total 16 kiB requests count: 885 (4 in endgame mode)
As you can observe - there are a lot of 16 kiB requests even not in
endgame mode.
Total requests count: 156
Total 16 kiB requests count: 76 (76 in endgame mode)
So, single blocks are requested from web seeds even in normal mode.
I think that other behaviour (e.g. whole combining blocks) in end game
mode for web seeds is dangerous.
But according to numbers above: seems that problem is not in the end
game mode.
Do you a suggestion what are other reasons for that?
Thanks you in advance.
------------------------------------------------------------------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
--
Arvid Norberg
------------------------------------------------------------------------------
Arvid Norberg
2016-01-07 22:29:36 UTC
Permalink
Could you try this patch and let me know if it helps?

diff --git a/src/piece_picker.cpp b/src/piece_picker.cpp
index 57980c9..b4d0a9d 100644
--- a/src/piece_picker.cpp
+++ b/src/piece_picker.cpp
@@ -1989,7 +1989,7 @@ namespace libtorrent
// them. In order for this to have an affect, also
disable
// prefer whole pieces (otherwise partial pieces
would be de-prioritized)
options |= prioritize_partials;
- prefer_contiguous_blocks = 0;
+// prefer_contiguous_blocks = 0;

ret |= picker_log_alert::partial_ratio;
}
Post by Arvid Norberg
There may be an issue with the affinity to pick whole pieces vs. the
affinity to pick form already started pieces. I'll have to look into it
some more.
Post by Artem Petrov
On Fri, Dec 11, 2015 at 5:23 AM, Artem Petrov <artempetroff <at>
gmail.com>
Post by Artem Petrov
Hi! Arvid, thank you for fast reply.
I'm using libtorrent 1.0.7, and it is released from RC_1_0 branch.
Behavior, which you had described in previous mail is valid for
master
Post by Artem Petrov
branch, however for RC_1_0 piece picker works in a different way: in
case
Post by Artem Petrov
when almost all pieces are downloaded and the other parts are in
downloading state piece picker will return some random block from
list of
Post by Artem Petrov
downloading blocks, which was already requested by some other
connection.
Post by Artem Petrov
As result, this single block will be requested from web seed.
This is the master behavior as well. This is the end-game mode in
bittorrent.
When the piece picker fails to find a free block to request from a
peer, it
will request exactly one 16 kiB block that's already being downloaded
from
another peer.
It won't pick another end-game-mode block until the previous one is
completely downloaded (i.e. it won't pipeline end-game blocks). This
should
keep the wasted download within bounds. The end game mode normally
wouldn't
last very long though. The peers(s) that requested those blocks in the
first place pipelined them, and in this case those peers are likely
stalled.
Do you experience continuous end-game mode? if so, do you know why?
Post by Artem Petrov
Do you have plan and when is it supposed to release desired
behaviour?
Is it possible that web seeds behave differently in end-game mode?
Hi, thanks for reply.
Regarding endgame mode - it is not continuous for me. But I had made
some measurements on 165 MB file (there was 5 web seeds and non-web
seed, piece length - 2 MB) as result, I had the following results (from
Total requests count: 987
Total 16 kiB requests count: 885 (4 in endgame mode)
As you can observe - there are a lot of 16 kiB requests even not in
endgame mode.
Total requests count: 156
Total 16 kiB requests count: 76 (76 in endgame mode)
So, single blocks are requested from web seeds even in normal mode.
I think that other behaviour (e.g. whole combining blocks) in end game
mode for web seeds is dangerous.
But according to numbers above: seems that problem is not in the end
game mode.
Do you a suggestion what are other reasons for that?
Thanks you in advance.
------------------------------------------------------------------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
--
Arvid Norberg
--
Arvid Norberg
------------------------------------------------------------------------------
Artem Petrov
2016-01-15 11:40:07 UTC
Permalink
Post by Arvid Norberg
Could you try this patch and let me know if it helps?
diff --git a/src/piece_picker.cpp b/src/piece_picker.cpp
index 57980c9..b4d0a9d 100644
--- a/src/piece_picker.cpp
+++ b/src/piece_picker.cpp
<at> <at> -1989,7 +1989,7 <at> <at> namespace libtorrent
// them. In order for this to have an affect, also
disable
// prefer whole pieces (otherwise partial pieces
would be de-prioritized)
options |= prioritize_partials;
- prefer_contiguous_blocks = 0;
+// prefer_contiguous_blocks = 0;
ret |= picker_log_alert::partial_ratio;
}
Hi,

Just finished testing.
Regarding this patch - it was patch for master, but I'm using libtorrent
1.0.7, so I had adapted this patch for RC_1_0 in the following way:


diff --git a/src/piece_picker.cpp b/src/piece_picker.cpp
index a688a00..e488ae3 100644
--- a/src/piece_picker.cpp
+++ b/src/piece_picker.cpp
@@ -1510,7 +1510,7 @@ namespace libtorrent
// threshold for the number of partials
if (int(m_downloads.size()) > m_num_pad_files + num_peers * 3 / 2)
options |= prioritize_partials;

- if (options & ignore_whole_pieces) prefer_whole_pieces = 0;
+ //if (options & ignore_whole_pieces) prefer_whole_pieces = 0;

// only one of rarest_first and sequential can be set.
TORRENT_ASSERT(((options & rarest_first) ? 1 : 0)


and it wasn't helped.
I'm not sure, that it's a correct adaptation for RC_1_0 due to huge
differences between master and RC_1_0. In case if
I'm wrong, could you please correct me?

Thanks,
Artem
Arvid Norberg
2016-01-15 23:34:07 UTC
Permalink
Post by Artem Petrov
Post by Arvid Norberg
Could you try this patch and let me know if it helps?
diff --git a/src/piece_picker.cpp b/src/piece_picker.cpp
index 57980c9..b4d0a9d 100644
--- a/src/piece_picker.cpp
+++ b/src/piece_picker.cpp
<at> <at> -1989,7 +1989,7 <at> <at> namespace libtorrent
// them. In order for this to have an affect,
also
Post by Arvid Norberg
disable
// prefer whole pieces (otherwise partial
pieces
Post by Arvid Norberg
would be de-prioritized)
options |= prioritize_partials;
- prefer_contiguous_blocks = 0;
+// prefer_contiguous_blocks = 0;
ret |= picker_log_alert::partial_ratio;
}
Hi,
Just finished testing.
Regarding this patch - it was patch for master, but I'm using libtorrent
hm.. ok. youre patch looks good. If you could provide me with an example
.torrent file that reproduces this behavior, I'm confident I could fix it
(feel free to mail it directly to me).
--
Arvid Norberg
Arvid Norberg
2016-01-17 17:35:51 UTC
Permalink
Post by Arvid Norberg
Post by Artem Petrov
Just finished testing.
Regarding this patch - it was patch for master, but I'm using libtorrent
hm.. ok. youre patch looks good. If you could provide me with an example
.torrent file that reproduces this behavior, I'm confident I could fix it
(feel free to mail it directly to me).
Ok, here's another patch against RC_1_0. Would you mind giving it a try?

apply like this: patch -p1 <prefer-whole-pieces-fix.diff
--
Arvid Norberg
Arvid Norberg
2016-01-17 20:13:09 UTC
Permalink
it's also available as a pull request here:
https://github.com/arvidn/libtorrent/pull/396
Post by Arvid Norberg
Post by Arvid Norberg
Post by Artem Petrov
Just finished testing.
Regarding this patch - it was patch for master, but I'm using libtorrent
hm.. ok. youre patch looks good. If you could provide me with an example
.torrent file that reproduces this behavior, I'm confident I could fix it
(feel free to mail it directly to me).
Ok, here's another patch against RC_1_0. Would you mind giving it a try?
apply like this: patch -p1 <prefer-whole-pieces-fix.diff
--
Arvid Norberg
--
Arvid Norberg
Artem Petrov
2016-01-20 13:21:28 UTC
Permalink
Post by Arvid Norberg
https://github.com/arvidn/libtorrent/pull/396
On Sun, Jan 17, 2016 at 12:35 PM, Arvid Norberg <arvid.norberg <at>
gmail.com>
Post by Arvid Norberg
On Fri, Jan 15, 2016 at 6:34 PM, Arvid Norberg <arvid.norberg <at>
gmail.com>
Post by Arvid Norberg
On Fri, Jan 15, 2016 at 6:40 AM, Artem Petrov <artempetroff <at>
gmail.com>
Post by Arvid Norberg
Post by Artem Petrov
Just finished testing.
Regarding this patch - it was patch for master, but I'm using libtorrent
hm.. ok. youre patch looks good. If you could provide me with an example
.torrent file that reproduces this behavior, I'm confident I could fix it
(feel free to mail it directly to me).
Ok, here's another patch against RC_1_0. Would you mind giving it a try?
apply like this: patch -p1 <prefer-whole-pieces-fix.diff
--
Arvid Norberg
Hi!
Thanks for reply and for fix. Now situation is much better: on regular
rates - there are about 30-50% of 16 KiB requests and all of them in
endgame mode, but 16KiB requests can be observed with same conditions on
hight download rates (10+ MB/sec.).

Can you assume what is related to?


Regards, Artem
Arvid Norberg
2016-01-21 00:54:43 UTC
Permalink
Post by Artem Petrov
[...]
Hi!
Thanks for reply and for fix. Now situation is much better: on regular
rates - there are about 30-50% of 16 KiB requests and all of them in
endgame mode, but 16KiB requests can be observed with same conditions on
hight download rates (10+ MB/sec.).
Can you assume what is related to?
in libtorrent 1.1 (RC_1_1 branch) there is an alert category that posts
picker_log_alerts, a form of somewhat verbose logging of which parts of the
piece picker were activated for individual invocations. With this logging,
it would be somewhat simple to figure out what's going on. in 1.0.x
however, probably something similar would have to be added.
--
Arvid Norberg
Artem Petrov
2016-02-09 07:33:18 UTC
Permalink
Post by Arvid Norberg
in libtorrent 1.1 (RC_1_1 branch) there is an alert category that posts
picker_log_alerts, a form of somewhat verbose logging of which parts of the
piece picker were activated for individual invocations. With this logging,
it would be somewhat simple to figure out what's going on. in 1.0.x
however, probably something similar would have to be added.
Ok, thank you for you efforts, current solution is pretty good.

Continue reading on narkive:
Loading...