[libtorrent] Full write queue performance

[Murray S. Kucherawy]

Scenario: libtorrent-rasterbar 1.0.8, via python bindings; write queue is
INT_MAX in size

If we fill the write queue and then more I/O comes in that needs to go to
disk, what happens to the torrent client? I'm wondering if we would begin
explicitly rejecting pieces, let the socket read buffers back up, block
until the write queue drains some, or something else.

Thanks,

-MSK

[Arvid Norberg]

Post by Murray S. Kucherawy
Scenario: libtorrent-rasterbar 1.0.8, via python bindings; write queue is
INT_MAX in size
If we fill the write queue and then more I/O comes in that needs to go to
disk, what happens to the torrent client?

If a peer connection cannot stuff another piece down the disk write queue,
it will apply back-pressure to the socket (each peer connection has a
bitmask indicating whether it's stalled and what it is stalled on, waiting
for data on the socket, waiting to be able to write to disk or waiting for
bandwidth quota if there's a rate limit set).

Basically the peer will stop reading from the socket while it's blocked.
This will cause the socket receive buffer to fill up, sending increasingly
small advertised receive windows back to the sender and eventually close
the window. At that point the sender will stop sending more data until the
window is opened again. The sender should have a limit on how much data it
stuffs down its socket send buffer and once that fills up, it should stop
reading more data to be sent, until things clear up.

It would be quite wasteful to have a write queue of 2 GB though, most
likely.

I'm wondering if we would begin

Post by Murray S. Kucherawy
explicitly rejecting pieces,

You don't reject incoming pieces, you reject incoming requests for pieces.
You "reject" incoming pieces by not requesting them. Which will happen as
the download rate drops, and the number of outstanding requests increase,
the delay-bandwidth target will decrease and the number of outstanding
requests we're aiming to have will decrease.

[Murray S. Kucherawy]

Post by Arvid Norberg

Post by Murray S. Kucherawy
Scenario: libtorrent-rasterbar 1.0.8, via python bindings; write queue is
INT_MAX in size
If we fill the write queue and then more I/O comes in that needs to go to
disk, what happens to the torrent client?

If a peer connection cannot stuff another piece down the disk write queue,
it will apply back-pressure to the socket (each peer connection has a
bitmask indicating whether it's stalled and what it is stalled on, waiting
for data on the socket, waiting to be able to write to disk or waiting for
bandwidth quota if there's a rate limit set).
Basically the peer will stop reading from the socket while it's blocked.
This will cause the socket receive buffer to fill up, sending increasingly
small advertised receive windows back to the sender and eventually close
the window. At that point the sender will stop sending more data until the
window is opened again. The sender should have a limit on how much data it
stuffs down its socket send buffer and once that fills up, it should stop
reading more data to be sent, until things clear up.
It would be quite wasteful to have a write queue of 2 GB though, most
likely.

Could be that the backing store isn't able to keep up for some operational
reason.

Are other torrents on the same client impacted when the disk write queue is
full? Or to put it another way, would I expect only downloading torrents
to start degrading, or could this affect torrents going in the other
direction as well?

-MSK

[Arvid Norberg]

Post by Arvid Norberg
[...]

Post by Arvid Norberg
If a peer connection cannot stuff another piece down the disk write

queue,

Post by Arvid Norberg
it will apply back-pressure to the socket (each peer connection has a
bitmask indicating whether it's stalled and what it is stalled on,

waiting

Post by Arvid Norberg
for data on the socket, waiting to be able to write to disk or waiting

for

Post by Arvid Norberg
bandwidth quota if there's a rate limit set).
Basically the peer will stop reading from the socket while it's blocked.
This will cause the socket receive buffer to fill up, sending

increasingly

Post by Arvid Norberg
small advertised receive windows back to the sender and eventually close
the window. At that point the sender will stop sending more data until

the

Post by Arvid Norberg
window is opened again. The sender should have a limit on how much data

it

Post by Arvid Norberg
stuffs down its socket send buffer and once that fills up, it should stop
reading more data to be sent, until things clear up.
[...]

Could be that the backing store isn't able to keep up for some operational
reason.
Are other torrents on the same client impacted when the disk write queue is
full? Or to put it another way, would I expect only downloading torrents
to start degrading, or could this affect torrents going in the other
direction as well?

There is only one disk queue. If it fills up, no other peer will be able to
put more data on it.

Older versions of libtorrent had one queue per peer, it just makes it hard
to cap the memory usage with that system.

[Murray S. Kucherawy]

Post by Arvid Norberg
There is only one disk queue. If it fills up, no other peer will be able to
put more data on it.

OK. If a process has two client instances, one used for downloading and
one for uploading, and the download client's write queue fills, would this
impact the upload client at all?

-MSK

[Murray S. Kucherawy]

Post by Arvid Norberg
There is only one disk queue. If it fills up, no other peer will be able to

Post by Arvid Norberg
put more data on it.

OK. If a process has two client instances, one used for downloading and
one for uploading, and the download client's write queue fills, would this
impact the upload client at all?

Or maybe here's a better way to ask:

Suppose I have a service that's greedy, by which I mean any time it's told
about a torrent, it will add it to its download client until it has a
copy. Once it has a copy, it removes that torrent from the download client
and adds it to another one dedicated to serving uploads. You've explained
that this will eventually mean torrents on the download client will slow
down or even stop as the disk tries to keep up. So two other questions:

1) Could that overload situation on the download client affect anything on
the upload client?

2) Would it be better to avoid adding torrents to the download client when
the disk queue is full, or is it fine to just keep adding them? Or from
the perspective of the other members of the swarm: Is a swarm weakened by a
peer that can't actually participate because it can't accept any pieces?

-MSK

[Arvid Norberg]

Post by Murray S. Kucherawy

Post by Arvid Norberg
There is only one disk queue. If it fills up, no other peer will be able

to

Post by Arvid Norberg

Post by Arvid Norberg
put more data on it.

OK. If a process has two client instances, one used for downloading and
one for uploading, and the download client's write queue fills, would

this

Post by Arvid Norberg
impact the upload client at all?

Suppose I have a service that's greedy, by which I mean any time it's told
about a torrent, it will add it to its download client until it has a
copy. Once it has a copy, it removes that torrent from the download client
and adds it to another one dedicated to serving uploads. You've explained
that this will eventually mean torrents on the download client will slow
down or even stop as the disk tries to keep up.

I don't think the properties you describe lead to that. The disk being
slower than the network connection will.

Post by Murray S. Kucherawy
1) Could that overload situation on the download client affect anything on
the upload client?

If they share disk and it's the bottleneck, yes.

Post by Murray S. Kucherawy
2) Would it be better to avoid adding torrents to the download client when
the disk queue is full, or is it fine to just keep adding them? Or from
the perspective of the other members of the swarm: Is a swarm weakened by a
peer that can't actually participate because it can't accept any pieces?

You don't accept pieces in bittorrent, you request them. Once you've
reached your disk capacity, there should be an equilibrium where your
request rate oscillate around your disk capacity. I can't think of any
terrible consequences, but if you can't upload, you're dead weight to the
swarm.

[Murray S. Kucherawy]

Post by Murray S. Kucherawy

Post by Murray S. Kucherawy
Suppose I have a service that's greedy, by which I mean any time it's

told

Post by Murray S. Kucherawy
about a torrent, it will add it to its download client until it has a
copy. Once it has a copy, it removes that torrent from the download

client

Post by Murray S. Kucherawy
and adds it to another one dedicated to serving uploads. You've

explained

Post by Murray S. Kucherawy
that this will eventually mean torrents on the download client will slow
down or even stop as the disk tries to keep up.

I don't think the properties you describe lead to that. The disk being
slower than the network connection will.

Ah, interesting. That leads me to another point that might help me solve
my problem:

Does a downloading client ask for a piece from multiple peers at the same
time, or does a piece request time out (and possibly the peer gets snubbed)
before that client will ask for that piece from a different peer?

-MSK

[Arvid Norberg]

Post by Murray S. Kucherawy

Post by Murray S. Kucherawy

Post by Murray S. Kucherawy
Suppose I have a service that's greedy, by which I mean any time it's

told

Post by Murray S. Kucherawy
about a torrent, it will add it to its download client until it has a
copy. Once it has a copy, it removes that torrent from the download

client

Post by Murray S. Kucherawy
and adds it to another one dedicated to serving uploads. You've

explained

Post by Murray S. Kucherawy
that this will eventually mean torrents on the download client will

slow

Post by Murray S. Kucherawy

Post by Murray S. Kucherawy
down or even stop as the disk tries to keep up.

I don't think the properties you describe lead to that. The disk being
slower than the network connection will.

Ah, interesting. That leads me to another point that might help me solve
Does a downloading client ask for a piece from multiple peers at the same
time, or does a piece request time out (and possibly the peer gets snubbed)
before that client will ask for that piece from a different peer?

pieces (or blocks really) are requested from multiple peers in parallel,
but normally not the *same* block from multiple peers (there are some edge
cases where that happens).

piece requests don't really time out, peers do (except for the end-game
mode). You can find more details here:
http://blog.libtorrent.org/2011/11/block-request-time-outs/

[Arvid Norberg]

Post by Murray S. Kucherawy
[...]
OK. If a process has two client instances, one used for downloading and
one for uploading, and the download client's write queue fills, would this
impact the upload client at all?

If they access the same disk, and the disk is the bottleneck; yes.