Meshing using Apple Wireless Direct Link (AWDL)
19 August 2019 by Neil AlexanderWireless without borders
I was mostly prompted to write this post in response to a Hacker News thread recently, which announced the release of an open-source AirDrop implementation called OpenDrop, from the same team at Seemoo Lab who produced an open-source implementation of Apple Wireless Direct Link (AWDL) protocol called OWL. AWDL is the secret sauce behind AirDrop, peer-to-peer AirPlay and some other Apple wireless technologies. Even though everything covered in this post was done some time ago, I have never spent the time to document it.
With a few exceptions, most wireless networks in the world operate in “infrastructure mode” which is where a wireless access point serves one or more wireless clients. Think of your Wi-Fi at home, at work or in a coffee shop. However, as implied by the name, reliable and usable infrastructure Wi-Fi is often only available in certain physical locations with “good infrastructure”. If you wanted to connect some devices together anywhere not served by an infrastructure Wi-Fi network, or in a location where you can’t suddenly plug in a wireless access point, you may not have many options (Bluetooth aside).
AWDL is designed to avoid this problem by extending the 802.11 wireless standard to allow client devices to communicate directly with each other, without the help of the central wireless access point. You can walk out into a field with a couple of iPhones or Macs and they can use AWDL to discover each other and exchange data, peer-to-peer. Even better is that nearby devices that are connected to different infrastructure Wi-Fi networks can still communicate with each other using AWDL!
The science
Normally, when connected to a wireless access point, wireless clients remain locked to the specific radio channel that the AP is using. AWDL works by instructing the wireless adapter in the device to “hop” between channels so that it can not only remain connected to the wireless access point, but can also listen to other nearby devices.
Devices announce their presence and information about their services on a “social channel” for other devices to hear, effectively creating peer-to-peer service discovery. Once two devices have decided that they want to communicate directly, they agree to jump to another channel for real data exchange so that they don’t interrupt existing Wi-Fi networks or, indeed, the social channel. These “hops” between wireless channels happen so quickly that there’s very little disruption to what the user is doing with their Wi-Fi connection already (except for some minor wireless performance degradation - to be covered later).
A number of papers have been published by the OWLink team on the inner workings of the AWDL protocol, which can be found here. In particular, this paper from Mobicom 2018 contains a significant amount of detail about the AWDL protocol itself, channel hopping techniques and security considerations, amongst other things.
Mesh opportunities
Yggdrasil is designed to create a mesh network automatically out of interconnected nodes - the idea being that all nodes can route to all other nodes on the mesh network by routing through other nodes.
Today, many of these connections happen between nodes across the Internet, since the community is still relatively small and geographically dispersed. A node joining the Yggdrasil network needs to only peer with a single device that is already connected to the wider network in order to participate in the fully-routable mesh.
However, it’s not the goal of Yggdrasil to remain something that we just toy with over the Internet. We want to build a protocol that can scale globally and work ad-hoc, even in places where infrastructure might not be particularly strong otherwise. We think that one of Yggdrasil’s greatest strengths is that it is very close to zero-configuration, beyond giving it a very small number of configuration options, and it should scale well too in principle.
Yggdrasil can already discover potential peers on the same network segment by
using multicast service discovery, which sounds a lot like what AWDL does on the
social channel. You can configure which interfaces Yggdrasil beacons on with the
MulticastInterfaces configuration directive.
I wanted to know if we could blend the two so that Yggdrasil could automatically discover other nearby devices and initiate peering connections with them using AWDL.
Getting started
Macs are a good target for developing and testing AWDL-aware applications as
AWDL is exposed to userspace through a network adapter called awdl0. It sits
there with a link-local IPv6 address, you can run tcpdump or Wireshark on it
to listen to AWDL traffic and you can even ping multicast group addresses on the
interface and get responses from other nearby devices, e.g. using ping6
ff02::1%awdl0! However, Apple devices don’t always keep AWDL alive and
listening all of the time.
On macOS, the AWDL driver is only woken up when either AirDrop is being
actively used in Finder, or where a NetService has been created (usually
through Objective-C or Swift) which requests peer-to-peer networking. AWDL is
normally kept alive long enough to satisfy connectivity for these sessions and
then will be sent back to sleep after a period of idleness.
On iOS, the story is somewhat similar to above, except that AWDL is often woken
up as soon as the device is unlocked if AirDrop is enabled. The NetService API
otherwise functions the same way.
tvOS is the outlier in that it seems to wake up and listen to AWDL randomly, even when the device is otherwise asleep, presumably because it is advertising the ability to receive incoming AirPlay sessions to nearby devices.
From a user perspective, the awdl0 interface looks entirely unremarkable. It
behaves largely like any other ethernet interface, carrying regular IPv6
traffic. In the background it’s a bit more complicated, as the AWDL driver
performs traffic filtering for security reasons, namely, to stop someone sat
next to you in the airport from browsing your file shares. Regular listening
sockets won’t accept connections over AWDL unless a specific socket option was
configured on the socket before it started listening.
Multicast traffic, however, does largely get passed through the filter untouched. Bingo.
Waking up AWDL
The NetService API is effectively a wrapper around multicast DNS-SD, which in
Apple’s colourful language, is affectionately known as Bonjour. The API has the
added benefit of being able to tell the operating system to wake up the AWDL
driver pretty much on demand on behalf of “peer-to-peer” services.
So all we would need to do to wake up AWDL is to call the NetService API,
publish a service that requests peer-to-peer functionality and let the operating
system do the hard work for us. Yggdrasil, being written in Go, didn’t have any
concept of NetService but thankfully we were able to use Cgo to do this
instead.
We wrote a Cgo function which calls the NetService API and advertises our new
fake service, _yggdrasil._tcp, which causes the operating system to wake up
the AWDL driver. Amazingly this worked.
Yggdrasil doesn’t actually use DNS-SD - we currently use a custom-formatted multicast beacon on a different multicast group. It is planned to eventually migrate to something more standard, like DNS-SD, for service discovery. However, in this instance, registering a fake DNS-SD service was just enough to wake up AWDL.
Peering automatically
Once the driver is active, the regular Yggdrasil multicast beacons on the
ff02::114 multicast group address seem to be passed through to the driver
normally and the Yggdrasil nodes running on each machine start to hear each
other’s calls.
The only thing that remained to be done was to configure the sockets with the
aforementioned socket option to allow them to communicate over the AWDL
interface. This socket option is called SO_RECV_ANYIF and is defined in
sys/socket.h on Darwin as 0x1104.
We configure the socket option on our TCP peering socket:
err = unix.SetsockoptInt(int(fd), syscall.SOL_SOCKET, 0x1104, 1)
if err != nil {
...
}
Now that the Yggdrasil nodes can hear each other’s advertisements over the
awdl0 interface, the regular automatic peering process kicks in and a TCP
session is opened between the two devices, creating a peering. The net result?
AWDL peerings!
$ sudo yggdrasilctl getSwitchPeers
bytes_recvd bytes_sent coords endpoint ip port proto
1 244278 313907 [3 5 5 2 1] fe80::xxxx:xxxx:xxxx:xxxx%awdl0 xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx 1 tcp
To further cement the experiment, we can actually disconnect the two devices
from each other, or connect to different Wi-Fi networks automatically, and the
peering over the awdl0 interface still continues to function!
An iperf3 test over Yggdrasil using the new AWDL link looks fairly good - the
devices are sat next to each other:
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 15.4 MBytes 129 Mbits/sec
[ 5] 1.00-2.00 sec 16.9 MBytes 141 Mbits/sec
[ 5] 2.00-3.00 sec 15.9 MBytes 133 Mbits/sec
[ 5] 3.00-4.00 sec 17.6 MBytes 147 Mbits/sec
[ 5] 4.00-5.00 sec 16.8 MBytes 141 Mbits/sec
[ 5] 5.00-6.00 sec 16.2 MBytes 136 Mbits/sec
[ 5] 6.00-7.00 sec 12.5 MBytes 105 Mbits/sec
[ 5] 7.00-8.00 sec 12.7 MBytes 106 Mbits/sec
[ 5] 8.00-9.00 sec 14.9 MBytes 125 Mbits/sec
[ 5] 9.00-10.00 sec 13.5 MBytes 113 Mbits/sec
Observations and iOS
As the iperf3 test above shows, the link performance is actually quite good!
It routinely exceeds 100mbps, although this is between only two devices. I have
not been able to test this with Yggdrasil nodes running over AWDL in any
particular density due to only having a limited number of Macs to hand.
One thing that I did notice though is that, while AWDL is active, my wireless connection to my home Wi-Fi network does reduce in speed somewhat. This is to be expected, given that the wireless chipset is hopping between channels rather than spending all of its time on a single channel.
Sadly we weren’t able to reproduce this test using iOS Testflight builds of Yggdrasil. On iOS, we implement Yggdrasil as a VPN service which is subject to a number of probably reasonable restrictions imposed by the OS, which presumably exist to stop VPN extensions from spying on you.
We were able to create a NetService from within the VPN extension and the
service beacons were advertised as expected, however, we weren’t able to
initiate any other kind of connections over the awdl0 interface. After a chat
with an engineer at Apple, it turns out that the awdl0 interface isn’t scoped
for use within a VPN extension, thus squashing our hopes and dreams of being
able to sprinkle this kind of magic onto our iOS port of Yggdrasil. We have a
feature request radar open with Apple in the hope that they may be able to
change this restriction in the future.
But we were able to get this to work on macOS and that, itself, is quite awesome.
Conclusion
Yggdrasil doesn’t enable AWDL by default because of the reduction in wireless
performance that AWDL being active can cause. Therefore, to enable AWDL peering,
you must add the awdl0 interface specifically into the MulticastInterfaces
configuration option in yggdrasil.conf. However, we do have working support
for connecting Macs together and meshing automatically using AWDL, and you can
enable it very easily if you wish to experiment!
We’d love to hear if you are peering Yggdrasil nodes using AWDL, or have performed any more extensive tests of how it performs in real-world scenarios - join us on our Matrix channel!