[libtorrent] Single UDP socket and multi-homed clients

[Steven Siloti]

Arvid,

I did some research and found the handy socket option IPV6_PKTINFO which
allows recvmsg()/sendmsg() to get/set the interface index and
destination/source address of the datagram. This is just what we need to
support multi-homed clients while only using a single UDP socket
listening on IN6ADDR_ANY. Of course it's not supported by asio so we'll
have to call the native send/recv functions directly, but this shouldn't
be too much work.

For the DHT my plan is to iterator over the list of local addresses at
startup and construct dht::node instances with the following rules:
A single IPv4 node is created for each interface that has at least one
non-loopback IPv4 address associated with it.
A node is created for each IPv6 address with global scope. We may want
to add some logic to avoid creating nodes for temporary addresses if we can.

Multi-home with IPv4 on a single interface will not not be supported
because NAT would make it a nightmare. Each IPv4 interface will get its
own external_ip instance while IPv6 will use the local address directly.
While NAT with IPv6 technically exists (NAT66), I don't think it's
widely used enough to warrant supporting it.

Incoming DHT messages would be routed to the appropriate node based on
the receiving interface/address. All outgoing messages will pass the
node's interface/address to sendmsg(). This is important both to ensure
that responses use the same interface/address the request was
sent/received on and to detect when an interface/address becomes
invalid. If a sendmsg() call returns one of the errors defined in RFC
3542 Section 6.6 then the corresponding DHT node(s) are destroyed.

Detecting when new local addresses become available is more challenging.
I don't know of any standard way to get notifications about network
changes so we'll have to write separate implementations for each
platform. The monitoring will likely have to be done in a separate thread.

UTP sockets would be bound to the interface/address the SYN or STATE
packet is received on. Outgoing SYN packets would not specify an
interface/address to use (i.e. let the OS decide) while all others would
specify the bound interface/address.

Similarly, UDP tracker connect requests would not specify an
interface/address to use while announce/scrape requests would specify
the interface/address the connect response was received on.

Does this sound reasonable?

[Arvid Norberg]

This post might be inappropriate. Click to display it.

[Steven Siloti]

Post by Arvid Norberg
The documentation I can find for IPV6_PKTINFO is not entirely clear. I
found this document [1] which suggests that a UDP socket can be "bound" to
use a specific outgoing address and interface.
The IPV6_PKTINFO option on z/OS appears to take an in6_pktinfo struct
directly [2]. However, msdn [3] and the linux man pages [4] suggests that
this option is only used to enable receiving more information from incoming
packets, and possibly being able to specify it per packet in sendmsg.
[1]
https://www-01.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.hale001/ipv6d0151011588.htm
[2]
https://www-01.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.hale001/ipv6d0151000725.htm?lang=en
[3]
https://msdn.microsoft.com/en-us/library/windows/desktop/hh285667(v=vs.85).aspx
[4] http://man7.org/linux/man-pages/man7/ip.7.html
I get the impression that the scope-id returned in recvfrom() indicates
which interface it was received over, and if that same address is passed
verbatim to sendto(), that interface will be used for the outgoing packets.
Maybe IBM is special in this regard though.
It seems a bit messy to set control messages in msghdr struct, which is why
I got a bit optimistic reading that IBM document saying you can just set a
socket option to specify which interface to use. But I don't get the
impression that's how that option normally works.

Being able to set an interface directly using the IPV6_PKTINFO option is
definitely an IBM specific extension. As far as I can tell this option
was originally specified in RFC 2292 and there it is simply a boolean
which enables sending/receiving the in6_pktinfo struct in the message
control data. Linux, Windows, and OSX all follow the RFC in this regard.

Post by Arvid Norberg
It would probably make sense to have one utp_socket_manager tied to each
external IP. There's actually a feature to bind outgoing sockets to
specific interfaces which (at least on windows, I believe) will cause the
connection to be tied to the corresponding interface. The idea is that
round-robining outgoing interfaces can load balance between multiple NICs.

I don't see a problem with that, although it seems like managing
interface utilization should be done by the OS.

Post by Arvid Norberg
I've been thinking that if may actually make sense to announce multiple
times, once for each external IP. You may get some bias for incoming
connections, but on the other hand, you may want all of your incoming paths
to be advertised to other peers.

Agreed.

Post by Arvid Norberg

Post by Steven Siloti
Does this sound reasonable?

The current track I'm on has one socket per interface to listen on. One
case I have in mind is if you only want to listen on a subset of your
existing interfaces, tying sockets to those may be a simple way of
achieving that. I believe that, across OSX, BSD, Windows and linux, there
are simpler ways of tying a socket to a specific interface (although, not
portable).
On windows, my understanding is that if you bind a socket to a local IP,
outgoing TCP SYN or UDP packets will go out over that interface,
disregarding the routing table (on vista and later) [5].
On linux, one can use SO_BINDTODEVICE.
On OSX and BSD, one can use IP_BOUND_IF.

One socket per interface, and per global IPv6 address, is fine if you're
committed to doing load balancing within libtorrent.

Post by Arvid Norberg
[5]
http://stackoverflow.com/questions/2065495/using-a-specific-network-interface-for-a-socket-in-windows/2080495#2080495
If that's enough to make outgoing packets go over the right interface, I'm
1. it allows for configuring specifically which interfaces/addresses to
listen on (and which ones not to listen on)
2. It integrates well with boost.asio.