Discussion:
[libtorrent] Running a secondary DHT
Bedeho Mender
2016-09-21 16:09:17 UTC
Permalink
We are considering running a secondary DHT network for our client in order
to facilitate much faster peer discovery of peers which supports our
extension. Our plan is to run a second session/dht-node in the client, and
manually do get_peers/announce requests for torrents on our primary session.

The problem we are facing is that a malicious peer could easily inject
mainline peers into our secondary DHT. This is further exacerbated by the
fact that the two DHTs are indistinguishable at the protocol level, hence
mainline peers will respond to any requests which are made to them, further
introducing more mainline peers.

Do you have any suggestions for how to avoid this cross contamination?

Obviously no way to distinguish mainline peers from our peers, unless we
connect directly with them. However, we would at the very least want to
avoid mainline peers unwittingly contributing to further contamination. Do
we have to modify the DHT node to ignore all responses from peers which
have a v-key that does not match what we expect?

Best
Bedeho
ᐧ
------------------------------------------------------------------------------
t***@infinite-source.de
2016-09-21 16:46:23 UTC
Permalink
Post by Bedeho Mender
We are considering running a secondary DHT network for our client in order
to facilitate much faster peer discovery of peers which supports our
extension. Our plan is to run a second session/dht-node in the client, and
manually do get_peers/announce requests for torrents on our primary session.
That optimization seems fairly dubious to me.

What are your time-to-result needs?

Theoretical limits for a RTT around the globe is around 160ms (speed of
light in an optical fiber).

With some optimizations (using adaptive timeouts and a
recently-seen/low-RTT cache for lookups) I can get lookups to return the
first result on the public DHT in 400ms on average.

- The 8472

------------------------------------------------------------------------------
Bedeho Mender
2016-09-21 18:23:13 UTC
Permalink
When our client is initiating a download of a torrent, it has to quickly
figure out
how many, if any, peers exist on the given swarm which supports our bep10
extension.
As is, we have to basically actively search through the swarm, doing
multiple DHT
queries and PEX, and at some point just make a decision. That is a
relativley slow
and complex process, in particular because the overwhelming majority of
swarms
will not have our nascent extension.

This is a significant problem, so if you have some simpler and cleaner
solution in mind,
then please let us know.

Bedeho
Post by Bedeho Mender
Post by Bedeho Mender
We are considering running a secondary DHT network for our client in
order
Post by Bedeho Mender
to facilitate much faster peer discovery of peers which supports our
extension. Our plan is to run a second session/dht-node in the client,
and
Post by Bedeho Mender
manually do get_peers/announce requests for torrents on our primary
session.
That optimization seems fairly dubious to me.
What are your time-to-result needs?
Theoretical limits for a RTT around the globe is around 160ms (speed of
light in an optical fiber).
With some optimizations (using adaptive timeouts and a
recently-seen/low-RTT cache for lookups) I can get lookups to return the
first result on the public DHT in 400ms on average.
- The 8472
------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------------------------
t***@infinite-source.de
2016-09-21 18:53:36 UTC
Permalink
It looks like you're deep down some XY-problem rabbit hole.

You want to do X: whatever your extension does

As a subgoal/implementation strategy you want Y: Test whether swarms
have peers supporting the same extension.

Which you want to implement as Z: Having a completely separate DHT to
signal support for that extension


I don't know whether X actually *requires* Y since I don't know what
your extension does. But you don't need Z to achieve Y.

For example you could additionally announce to a derived infohash, e.g.
sha1(infohash + "my-extension") to have a pool of contacts supporting
your extension.

- The 8472
Post by Bedeho Mender
When our client is initiating a download of a torrent, it has to quickly
figure out
how many, if any, peers exist on the given swarm which supports our bep10
extension.
As is, we have to basically actively search through the swarm, doing
multiple DHT
queries and PEX, and at some point just make a decision. That is a
relativley slow
and complex process, in particular because the overwhelming majority of
swarms
will not have our nascent extension.
This is a significant problem, so if you have some simpler and cleaner
solution in mind,
then please let us know.
Bedeho
Post by Bedeho Mender
Post by Bedeho Mender
We are considering running a secondary DHT network for our client in
order
Post by Bedeho Mender
to facilitate much faster peer discovery of peers which supports our
extension. Our plan is to run a second session/dht-node in the client,
and
Post by Bedeho Mender
manually do get_peers/announce requests for torrents on our primary
session.
That optimization seems fairly dubious to me.
What are your time-to-result needs?
Theoretical limits for a RTT around the globe is around 160ms (speed of
light in an optical fiber).
With some optimizations (using adaptive timeouts and a
recently-seen/low-RTT cache for lookups) I can get lookups to return the
first result on the public DHT in 400ms on average.
- The 8472
------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------------------------
Bedeho Mender
2016-09-21 19:47:37 UTC
Permalink
I just wanted to spare people my X, which does require Y, but your
suggestion seems much better than our plan for Z, thanks!
ᐧ
Post by t***@infinite-source.de
It looks like you're deep down some XY-problem rabbit hole.
You want to do X: whatever your extension does
As a subgoal/implementation strategy you want Y: Test whether swarms
have peers supporting the same extension.
Which you want to implement as Z: Having a completely separate DHT to
signal support for that extension
I don't know whether X actually *requires* Y since I don't know what
your extension does. But you don't need Z to achieve Y.
For example you could additionally announce to a derived infohash, e.g.
sha1(infohash + "my-extension") to have a pool of contacts supporting
your extension.
- The 8472
Post by Bedeho Mender
When our client is initiating a download of a torrent, it has to quickly
figure out
how many, if any, peers exist on the given swarm which supports our bep10
extension.
As is, we have to basically actively search through the swarm, doing
multiple DHT
queries and PEX, and at some point just make a decision. That is a
relativley slow
and complex process, in particular because the overwhelming majority of
swarms
will not have our nascent extension.
This is a significant problem, so if you have some simpler and cleaner
solution in mind,
then please let us know.
Bedeho
Post by Bedeho Mender
Post by Bedeho Mender
We are considering running a secondary DHT network for our client in
order
Post by Bedeho Mender
to facilitate much faster peer discovery of peers which supports our
extension. Our plan is to run a second session/dht-node in the client,
and
Post by Bedeho Mender
manually do get_peers/announce requests for torrents on our primary
session.
That optimization seems fairly dubious to me.
What are your time-to-result needs?
Theoretical limits for a RTT around the globe is around 160ms (speed of
light in an optical fiber).
With some optimizations (using adaptive timeouts and a
recently-seen/low-RTT cache for lookups) I can get lookups to return the
first result on the public DHT in 400ms on average.
- The 8472
------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------
------------------
Post by Bedeho Mender
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------
------------------
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss
------------------------------------------------------------------------------
Continue reading on narkive:
Loading...