[libtorrent] Possible DHT extension?

Discussion:

s***@libero.it

2016-05-13 19:47:08 UTC

Hi Arvid, can I have your opinion about some questions about dht_put/get functions?

Right now we are able to put an item (let's say immutable one) on the DHT and the hash of the entry is returned to us for later retrieval. So far, so good. But if I place an item on the DHT only I am able to retrieve it later. I'm searching for a way to put the item on the DHT and later somebody else retrieves it, without knowing the hash of the whole entry, but just knowing the "identifier" string as below.
Let me explain it better.

I do have an entry wich is a list made of three string (as an example)

Entry{ "identifier_string" , "string2" , "string3"}

Right now it is not possibile to return from the put operation the SHA1 of the identifier string for later retrieval, I get the SHA1 of the flat, bencoded entry.
If that was able, a second person later, can retrive the whole entry just by querying for the SHa1 of the identifier string.

p.s.: if you rename the "identifier_string", "keyword" there you got a why to do some keyword searching on the DHT.

Thanks in advance,
Erjon

t***@infinite-source.de

2016-05-13 20:34:28 UTC

Permalink

If your motivation is keyword search then that is pretty much an
anti-goal of the put/get design.

Keyword search is a very difficult problem for several reasons:

* spam / other malicious actors
* human language follows a zipf distrubution, kademlia assumes that keys
are roughly uniformly distributed.
* intersection or union operations
* partial matching
* multi-ownership, no single entity should own a particular keyword
* result-set-sizes, a consequence of multi-ownership

Of course there's plenty of research and solutions to those problems,
but they're not trivial, especially when you have to implement all of them.

The get/put feature set was derived from the idea of tackling an easier
problem and just having single-owner data structures on the dht, where
the mutable put would give you an updatable root node in your data
structure pointing to the remaining entries, implemented as immutable
values.

This basic building block could be used to implement feeds for example,
which was the original motivation.

Single-ownership is guaranteed by including the public key in the
mutable entries. Since those are effectively random there are no
collisions even if human-readable text is used as salt.

- The 8472

Post by s***@libero.it
Hi Arvid, can I have your opinion about some questions about dht_put/get functions?
Right now we are able to put an item (let's say immutable one) on the DHT and the hash of the entry is returned to us for later retrieval. So far, so good. But if I place an item on the DHT only I am able to retrieve it later. I'm searching for a way to put the item on the DHT and later somebody else retrieves it, without knowing the hash of the whole entry, but just knowing the "identifier" string as below.
Let me explain it better.
I do have an entry wich is a list made of three string (as an example)
Entry{ "identifier_string" , "string2" , "string3"}
Right now it is not possibile to return from the put operation the SHA1 of the identifier string for later retrieval, I get the SHA1 of the flat, bencoded entry.
If that was able, a second person later, can retrive the whole entry just by querying for the SHa1 of the identifier string.
p.s.: if you rename the "identifier_string", "keyword" there you got a why to do some keyword searching on the DHT.
Thanks in advance,
Erjon
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

s***@libero.it

2016-05-13 20:54:18 UTC

Permalink

Please, correct me if I'm wrong.

In a mutable item I can use a string as salt for the item, but I cannot have a
list of entry's that maps that string to.
Using different salt I can have different values stored for the same key but
these values are unique. Can be updated with CAS, but, cannot exist a list of
values that can be identified with the same salt.

this code
[code]
map<string, set<entry>> m_association_string_to_set
[/code]
can never belong to DHT because of single-ownership?

Or in a more completed way
[code]
std::map<node_id,set<node_id>> m_keyword_to_infohash_map
[/code]

where each node_id in the set is the hash of the put_dht_item of entry std::
list(entry).

Thanks

----Messaggio originale----
Data: 13/05/2016 22.34
Ogg: Re: [libtorrent] Possible DHT extension?
If your motivation is keyword search then that is pretty much an
anti-goal of the put/get design.
* spam / other malicious actors
* human language follows a zipf distrubution, kademlia assumes that keys
are roughly uniformly distributed.
* intersection or union operations
* partial matching
* multi-ownership, no single entity should own a particular keyword
* result-set-sizes, a consequence of multi-ownership
Of course there's plenty of research and solutions to those problems,
but they're not trivial, especially when you have to implement all of them.
The get/put feature set was derived from the idea of tackling an easier
problem and just having single-owner data structures on the dht, where
the mutable put would give you an updatable root node in your data
structure pointing to the remaining entries, implemented as immutable
values.
This basic building block could be used to implement feeds for example,
which was the original motivation.
Single-ownership is guaranteed by including the public key in the
mutable entries. Since those are effectively random there are no
collisions even if human-readable text is used as salt.
- The 8472

Post by s***@libero.it
Hi Arvid, can I have your opinion about some questions about dht_put/get

functions?

Post by s***@libero.it
Right now we are able to put an item (let's say immutable one) on the DHT

and the hash of the entry is returned to us for later retrieval. So far, so
good. But if I place an item on the DHT only I am able to retrieve it later.
I'm searching for a way to put the item on the DHT and later somebody else
retrieves it, without knowing the hash of the whole entry, but just knowing the
"identifier" string as below.

Post by s***@libero.it
Let me explain it better.
I do have an entry wich is a list made of three string (as an example)
Entry{ "identifier_string" , "string2" , "string3"}
Right now it is not possibile to return from the put operation the SHA1 of

the identifier string for later retrieval, I get the SHA1 of the flat, bencoded
entry.

Post by s***@libero.it
If that was able, a second person later, can retrive the whole entry just

by querying for the SHa1 of the identifier string.

Post by s***@libero.it
p.s.: if you rename the "identifier_string", "keyword" there you got a why

to do some keyword searching on the DHT.

Post by s***@libero.it
Thanks in advance,
Erjon

------------------------------------------------------------------------------

Post by s***@libero.it
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data

untouched!

Post by s***@libero.it
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

t***@infinite-source.de

2016-05-13 21:04:59 UTC

Permalink

NO: map<string, set<entry>>
YES: map<tuple<pubkey,string>, set<entry>>

That's pretty much what I already described in my initial reply.
Including the pubkey ensures single-ownership.

Strings cannot be owned by a single person, because collisions would
happen. Who would own the word "Video" for example?

- The 8472

Post by s***@libero.it
Please, correct me if I'm wrong.
In a mutable item I can use a string as salt for the item, but I cannot have a
list of entry's that maps that string to.
Using different salt I can have different values stored for the same key but
these values are unique. Can be updated with CAS, but, cannot exist a list of
values that can be identified with the same salt.
this code
[code]
map<string, set<entry>> m_association_string_to_set
[/code]
can never belong to DHT because of single-ownership?
Or in a more completed way
[code]
std::map<node_id,set<node_id>> m_keyword_to_infohash_map
[/code]
list(entry).
Thanks

Post by s***@libero.it
Hi Arvid, can I have your opinion about some questions about dht_put/get

functions?

Post by s***@libero.it
Right now we are able to put an item (let's say immutable one) on the DHT

the identifier string for later retrieval, I get the SHA1 of the flat, bencoded
entry.

Post by s***@libero.it
If that was able, a second person later, can retrive the whole entry just

by querying for the SHa1 of the identifier string.

Post by s***@libero.it
p.s.: if you rename the "identifier_string", "keyword" there you got a why

to do some keyword searching on the DHT.

Post by s***@libero.it
Thanks in advance,
Erjon

------------------------------------------------------------------------------

untouched!

Jakob Petsovits

2016-05-13 22:50:06 UTC

Permalink

Post by t***@infinite-source.de
NO: map<string, set<entry>>
YES: map<tuple<pubkey,string>, set<entry>>
That's pretty much what I already described in my initial reply.
Including the pubkey ensures single-ownership.
Strings cannot be owned by a single person, because collisions would
happen. Who would own the word "Video" for example?

Well, ideally you'd put together an application/document-specific prefix
and the actual string to make e.g. "myapp:jpetso:Video" and then search
for (the hash of) that to get resolved to a set of DHT item hashes
and/or Ed25519 public keys. Nobody owns that string, but several owners
can advertise content for it. An unsigned string has to be ownerless
anyway. It's up to you to figure out whether and how to handle the
results, similar to torrents where the DHT allows you to discover peers
but you decide who to connect to and download from.

Right now it's already possible to resolve an arbitrary hash to a set of
IP/port locations, one might argue that opening this up to generic hash
results makes it more flexible and opens up new use cases. You don't
have to solve string pattern matching in order for a hash ->
potentially_incomplete_set<hash> function to be useful.

Concerns of spamming and limited input string space (resulting in a
small number of nodes being responsible for items) is definitely an
issue though.

- Jakob

Post by t***@infinite-source.de

Post by s***@libero.it
Hi Arvid, can I have your opinion about some questions about dht_put/get

functions?

Post by s***@libero.it
Right now we are able to put an item (let's say immutable one) on the DHT

the identifier string for later retrieval, I get the SHA1 of the flat, bencoded
entry.

Post by s***@libero.it
If that was able, a second person later, can retrive the whole entry just

by querying for the SHa1 of the identifier string.

Post by s***@libero.it
p.s.: if you rename the "identifier_string", "keyword" there you got a why

to do some keyword searching on the DHT.

Post by s***@libero.it
Thanks in advance,
Erjon

------------------------------------------------------------------------------

untouched!

t***@infinite-source.de

2016-05-13 23:28:54 UTC

Permalink

Could you give some good example what kind of use-case it would enable
that's not a human-language search?

Post by Jakob Petsovits

Well, ideally you'd put together an application/document-specific prefix
and the actual string to make e.g. "myapp:jpetso:Video" and then search
for (the hash of) that to get resolved to a set of DHT item hashes
and/or Ed25519 public keys. Nobody owns that string, but several owners
can advertise content for it. An unsigned string has to be ownerless
anyway. It's up to you to figure out whether and how to handle the
results, similar to torrents where the DHT allows you to discover peers
but you decide who to connect to and download from.
Right now it's already possible to resolve an arbitrary hash to a set of
IP/port locations, one might argue that opening this up to generic hash
results makes it more flexible and opens up new use cases. You don't
have to solve string pattern matching in order for a hash ->
potentially_incomplete_set<hash> function to be useful.
Concerns of spamming and limited input string space (resulting in a
small number of nodes being responsible for items) is definitely an
issue though.
- Jakob

Post by t***@infinite-source.de

Post by s***@libero.it
Hi Arvid, can I have your opinion about some questions about dht_put/get

functions?

Post by s***@libero.it
Right now we are able to put an item (let's say immutable one) on the DHT

the identifier string for later retrieval, I get the SHA1 of the flat, bencoded
entry.

Post by s***@libero.it
If that was able, a second person later, can retrive the whole entry just

by querying for the SHa1 of the identifier string.

Post by s***@libero.it
p.s.: if you rename the "identifier_string", "keyword" there you got a why

to do some keyword searching on the DHT.

Post by s***@libero.it
Thanks in advance,
Erjon

------------------------------------------------------------------------------

untouched!

Arvid Norberg

2016-05-14 00:07:56 UTC

Permalink

Post by Jakob Petsovits

Post by t***@infinite-source.de
[...]
That's pretty much what I already described in my initial reply.
Including the pubkey ensures single-ownership.
Strings cannot be owned by a single person, because collisions would
happen. Who would own the word "Video" for example?

[...]
Right now it's already possible to resolve an arbitrary hash to a set of
IP/port locations, one might argue that opening this up to generic hash
results makes it more flexible and opens up new use cases. You don't
have to solve string pattern matching in order for a hash ->
potentially_incomplete_set<hash> function to be useful.
Concerns of spamming and limited input string space (resulting in a
small number of nodes being responsible for items) is definitely an
issue though.

The mechanisms for adding content to the DHT that exist today are protected
to some degree:

1. announce_peer is pretected by doing a "3-way handshake", using
write_token. You can only add your IP if you control receiving packets to
it.
2. putting an immutable blob is protected from from overwriting someone
else's data (it can extend the life of someone else's data though).
3. immutable items are also protected from overwriting someone else's data,
assuming the private key isn't compromised.

Now, given salt, it's possible to create a publicly known keypair and use
the salt as the real key. However, that requires you to deliberately
circumvent the design, and hopefully you'll think twice.

--
Arvid Norberg

s***@libero.it

2016-05-14 12:32:38 UTC

Permalink

I was asking if there was a way to do things without breaking the design. Right
now I'm using the mutable item store for RSS feeds and its working properly.
Each user has its own private/public key and in no way I'm willing to break the
desing.

Anyway do you beleive that this feature could be implemented in the future?

Thanks

----Messaggio originale----
Data: 14/05/2016 2.07

net>

Ogg: Re: [libtorrent] R: Re: Possible DHT extension?

Post by Jakob Petsovits

The mechanisms for adding content to the DHT that exist today are protected
1. announce_peer is pretected by doing a "3-way handshake", using
write_token. You can only add your IP if you control receiving packets to
it.
2. putting an immutable blob is protected from from overwriting someone
else's data (it can extend the life of someone else's data though).
3. immutable items are also protected from overwriting someone else's data,
assuming the private key isn't compromised.
Now, given salt, it's possible to create a publicly known keypair and use
the salt as the real key. However, that requires you to deliberately
circumvent the design, and hopefully you'll think twice.
--
Arvid Norberg
------------------------------------------------------------------------------
Mobile security can be enabling, not merely restricting. Employees who
bring their own devices (BYOD) to work are irked by the imposition of MDM
restrictions. Mobile Device Manager Plus allows you to control only the
apps on BYO-devices by containerizing them, leaving personal data untouched!
https://ad.doubleclick.net/ddm/clk/304595813;131938128;j
_______________________________________________
Libtorrent-discuss mailing list
https://lists.sourceforge.net/lists/listinfo/libtorrent-discuss

t***@infinite-source.de

2016-05-14 20:58:02 UTC

Permalink

Post by s***@libero.it
I was asking if there was a way to do things without breaking the design. Right
now I'm using the mutable item store for RSS feeds and its working properly.
Each user has its own private/public key and in no way I'm willing to break the
desing.
Anyway do you beleive that this feature could be implemented in the future?

Only if something changes about the use-case or the proposed design that
avoids the problems mentioned by everyone. Just waiting for the future
doesn't magically improve things.

Post by s***@libero.it
Thanks

----Messaggio originale----
Data: 14/05/2016 2.07

net>

Ogg: Re: [libtorrent] R: Re: Possible DHT extension?

Post by Jakob Petsovits