[libtorrent] Libtorrent(python) Raspberry overload

Discussion:

Mircea Stanciu

2016-08-26 08:23:01 UTC

He guys! I started using Libtorrent inside a python script on a
Raspberry Pi3. It performs admirably while downloading, but when doing a
hash-check one of the 4 core of the CPU jumps to 100% usage, making
other programs run sluggish.

The tuning page recommends a few settings for low memory systems
http://www.rasterbar.com/products/libtorrent/tuning.html.

I ended up changing a few settings with code like:
ses = lt.session()
session_settings = ses.settings()

session_settings.optimize_hashing_for_speed = False
session_settings.download_rate_limit = 100*1000
session_settings.asio_disable_iocp = True

ses.set_settings(session_settings)

I tried other settings, but for no effect. I can see a decrees on
download speed, but still, the CPU gets maxed out. What settings could I
use to tame Libtorrent for my little Raspberry Pi3
Kind regards,

--
---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
------------------------------------------------------------------------------

m***@beroal.in.ua

2016-08-27 08:14:51 UTC

Permalink

Hi. None of the options you tweaked seems to address CPU usage. Anyway,
a hash check requires many computations, so high CPU usage is
unavoidable. Instead, you need to limit the share of CPU used by
"libtorrent" by means of your OS. In Linux, you can impose a soft limit
by "nice" and a hard limit by the "|cgroup|" feature.

Post by Mircea Stanciu
He guys! I started using Libtorrent inside a python script on a
Raspberry Pi3. It performs admirably while downloading, but when doing a
hash-check one of the 4 core of the CPU jumps to 100% usage, making
other programs run sluggish.
The tuning page recommends a few settings for low memory systems
http://www.rasterbar.com/products/libtorrent/tuning.html.
ses = lt.session()
session_settings = ses.settings()
session_settings.optimize_hashing_for_speed = False
session_settings.download_rate_limit = 100*1000
session_settings.asio_disable_iocp = True
ses.set_settings(session_settings)
I tried other settings, but for no effect. I can see a decrees on
download speed, but still, the CPU gets maxed out. What settings could I
use to tame Libtorrent for my little Raspberry Pi3
Kind regards,

------------------------------------------------------------------------------

Arvid Norberg

2016-08-27 15:05:33 UTC

Permalink

I imagine there are mainly two possibilities for other things being slow on
your system:

1. there is CPU contention
2. there is disk I/O (memory/swap) contention

You can solve (1) by adjusting the priority of the process.

You can solve (2) with a setting called "file_checks_delay_per_block", this
will slow down file checking. However, it turns out this feature was
removed in libtorrent 1.1. Maybe I should add it back in. Do you think it's
useful?

Post by m***@beroal.in.ua
Hi. None of the options you tweaked seems to address CPU usage. Anyway,
a hash check requires many computations, so high CPU usage is
unavoidable. Instead, you need to limit the share of CPU used by
"libtorrent" by means of your OS. In Linux, you can impose a soft limit
by "nice" and a hard limit by the "|cgroup|" feature.

------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

--
Arvid Norberg
------------------------------------------------------------------------------

t***@infinite-source.de

2016-08-27 15:24:15 UTC

Permalink

There also is ionice, although i'm not sure if that affects mmaped IO.

Post by Arvid Norberg
I imagine there are mainly two possibilities for other things being slow on
1. there is CPU contention
2. there is disk I/O (memory/swap) contention
You can solve (1) by adjusting the priority of the process.
You can solve (2) with a setting called "file_checks_delay_per_block", this
will slow down file checking. However, it turns out this feature was
removed in libtorrent 1.1. Maybe I should add it back in. Do you think it's
useful?

------------------------------------------------------------------------------

Mircea Stanciu

2016-08-29 06:24:39 UTC

Permalink

I did try "file_checks_delay_per_block", just threw values at it
(something over 300 halts the check all together, under that no
perceivable effect). I found some tweaks here,
https://sourceforge.net/p/libtorrent/mailman/message/29400204/ ,under
min_memory_usage() class. Still, maxed out CPU.

After more testing it turned out that the biggest killer here is the
download speed. The system simply can't write to the SD Card as fast as
the torrent downloads, so a download speed limit of max 200kbps is a
must to avoid overwhelming.

Isolating the torrent process on one core from the OS and a download
speed limit doesn't stress the system that hard. It's a shame to loose
the download speed tho....

Post by t***@infinite-source.de
There also is ionice, although i'm not sure if that affects mmaped IO.

------------------------------------------------------------------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

------------------------------------------------------------------------------

Michael Mckeown

2016-08-29 10:08:17 UTC

Permalink

The faster you download the faster you hash and the more cpu you use, I'm experimenting with the 3 crypto options at the moment, openssl gives you a noticeable 2-3% cpu reduction, haven't tried libgcrypt yet.

In settings_pack turn off disable_hash_checks to see that it really is just the hashing.

Looks to me like the way things work was fine a while back when download speeds were typically poor but if you have a 10mb torrent protocol download limit as I have with my ISP then it's a 15-20% cpu usage per downloading torrent then if I go on to a VPN that takes the ISP throttling away the cpu will happily go up to 90%+ usage.

On 29/08/2016 07:24, Mircea Stanciu wrote:

I did try "file_checks_delay_per_block", just threw values at it
(something over 300 halts the check all together, under that no
perceivable effect). I found some tweaks here,
https://sourceforge.net/p/libtorrent/mailman/message/29400204/ ,under
min_memory_usage() class. Still, maxed out CPU.

After more testing it turned out that the biggest killer here is the
download speed. The system simply can't write to the SD Card as fast as
the torrent downloads, so a download speed limit of max 200kbps is a
must to avoid overwhelming.

Isolating the torrent process on one core from the OS and a download
speed limit doesn't stress the system that hard. It's a shame to loose
the download speed tho....

On 8/27/2016 6:24 PM, the8472.lt-***@infinite-source.de<mailto:the8472.lt-***@infinite-source.de> wrote:

There also is ionice, although i'm not sure if that affects mmaped IO.

On 27.08.2016 17:05, Arvid Norberg wrote:

I imagine there are mainly two possibilities for other things being slow on
your system:

1. there is CPU contention
2. there is disk I/O (memory/swap) contention

You can solve (1) by adjusting the priority of the process.

You can solve (2) with a setting called "file_checks_delay_per_block", this
will slow down file checking. However, it turns out this feature was
removed in libtorrent 1.1. Maybe I should add it back in. Do you think it's
useful?

On Sat, Aug 27, 2016 at 4:14 AM, <***@beroal.in.ua><mailto:***@beroal.in.ua> wrote:

Hi. None of the options you tweaked seems to address CPU usage. Anyway,
a hash check requires many computations, so high CPU usage is
unavoidable. Instead, you need to limit the share of CPU used by
"libtorrent" by means of your OS. In Linux, you can impose a soft limit
by "nice" and a hard limit by the "|cgroup|" feature.

On 26.08.16 11:23, Mircea Stanciu wrote:

He guys! I started using Libtorrent inside a python script on a
Raspberry Pi3. It performs admirably while downloading, but when doing a
hash-check one of the 4 core of the CPU jumps to 100% usage, making
other programs run sluggish.

The tuning page recommends a few settings for low memory systems
http://www.rasterbar.com/products/libtorrent/tuning.html.

I ended up changing a few settings with code like:
ses = lt.session()
session_settings = ses.settings()

session_settings.optimize_hashing_for_speed = False
session_settings.download_rate_limit = 100*1000
session_settings.asio_disable_iocp = True

ses.set_settings(session_settings)

I tried other settings, but for no effect. I can see a decrees on
download speed, but still, the CPU gets maxed out. What settings could I
use to tame Libtorrent for my little Raspberry Pi3
Kind regards,

------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
Libtorrent-***@lists.sourceforge.net<mailto:Libtorrent-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

------------------------------------------------------------------------------
_______________________________________________
Libtorrent-discuss mailing list
Libtorrent-***@lists.sourceforge.net<mailto:Libtorrent-***@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Arvid Norberg

2016-08-30 06:44:58 UTC

Permalink

Post by Michael Mckeown
The faster you download the faster you hash and the more cpu you use, I'm
experimenting with the 3 crypto options at the moment, openssl gives you a
noticeable 2-3% cpu reduction, haven't tried libgcrypt yet.
In settings_pack turn off disable_hash_checks to see that it really is just the hashing.
Looks to me like the way things work was fine a while back when download
speeds were typically poor but if you have a 10mb torrent protocol download
limit as I have with my ISP then it's a 15-20% cpu usage per downloading
torrent then if I go on to a VPN that takes the ISP throttling away the cpu
will happily go up to 90%+ usage.

You talk about the CPU usage as if you have already concluded that this is
the cause of your problems. Did you try to lower the priority of the
process?

You may want to try to make your disk cache smaller too, in case your
download rate exceeds the disk write rate (for too long).

--
Arvid Norberg
------------------------------------------------------------------------------