Introduction to P2P messaging using CADET and GNUnet++

GNUnet is GNU's (yes, that GNU in GNU/Linux) framework for p2p applications. CADET (Confidential Ad-hoc Decentralized End-to-end Transport) is GNUnet's transport layer protocol. Think it like a replacement of TCP/IP. Practically it provides the following some over plain TCP/IP:

• Encrypted communication
• NAT traversal
• 512-bit port range
• Firewall bypass
• Censorship resistance
• Mesh routing (nodes not necessarily need connected to the Internet)

This post explains the basic concept you need to know to use CADET and shows how to send messages using GNUnet++.

Preparation

To use CADET you need a working GNUnet installation. Hopefully you already have one. If not, you can follow the instructions in ArchWiki for a quick setup (yes, even if you are on Ubuntu, the basic idea is the same. Highly recomment to get the latest release from source. GNUnet sometimes changes protocol details, this keeps your node up to data with others). Once you have GNUnet installed, you can start the GNUnet with the following command:

❯ gnunet-arm -s

You are good if nothing is printed to the console. That starts the entire GNUnet stack. You can check if CADET is up using gnunet-arm -I | grep cadet. If you see something like the following, then you are good to go:

❯ gnunet-arm -I | grep cadet
cadet (binary='gnunet-service-cadet', status=started)

Now let's install GNUnet++. GNUnet++ is my C++ wrapper around GNUnet's C API. It's experimental, but makes developing GNUnet applications much, much easier. You can install from source. Note that you need a C++20 compiler (likely GCC 11 or Clang 14) to compile GNUnet++. The dependencies are listed in the README.

❯ git clone https://github.com/marty1885/gnunetpp.git
❯ cd gnunetpp
❯ mkdir build
❯ cd build
❯ cmake ..
❯ make -j
❯ sudo make install

Try running some examples in GNUnet++ to confirm both GNUnet and GNUnet++ are working fine. It shouldn't hang or print any error messages. But a very short period of nothing is normal if you just started GNUnet seconds ago. We'll talk about what's going on in the examples later.

❯ ./examples/gnunetpp-cadet list-tunnel
Tunnel: R3J2CPAGEFPMZFM8N821N97WVTZCTSKCDPMNY1HSR1YVE2A1WPT0 [ ENC: Ok               , CON: New     ] 0 CHs, 1 CONNs
Tunnel: S97E3GF56K9YHYTVBAJZ8H4Y5EKRXQM2YMM9Z2HB6Y7R0B2N2R20 [ ENC: Ok               , CON: New     ] 0 CHs, 3 CONNs
Tunnel: Y924NSHMMZ1N1SQCE5TXF93ED6S6JY311K0QT86G9WJC68F6XVZ0 [ ENC: Ok               , CON: New     ] 0 CHs, 0 CONNs

CADET Basics

The internet is broken

As many of network programmers know. No doubt, the modern internet is "broken" in may ways, from annoyance to serious privacy and freedom issues.

• Censorship: Governments and corporations can block access to certain websites. And it's easy. Either get DNS providers to delist the domain or get ISPs to block the IP address.
• Privacy: ISPs can track who you are talking to (the service you are connecting to), when, and the DNS queries you make.
• NAT/Firewall: In some environments, your access to the Internet is limited. Some places only allow SSL traffic going out. Sometimes you are behind a NAT that prevents you from hosting a server. And sometimes you are behind a firewall that logs all your traffic. These are no fun to deal with.
• Non-standard network: Some networks are not directly connected to the Internet. Maybe you have to Bluetooth chain your way, or LoRa. Each of these networks have their own APIs and protocols. And it's a pain to write a program that works on all of them.
• (The lack of) default security: TCP/IP, even IPv6 is designed way back in a time where security of computers is solved by "don't let malicious people go near a computer". But that's not the case anymore. That's why HTTPS is the baseline nowadays. But base protocols are still not secure. Upstream can inject fake packets and downstream would be non the wiser.
• Centralized control: ISP usually have a monopoly over an area. That means a) if their core network goes down, everyone in their network goes down too. b) if they want to block a certain website, they can do so for everyone. Yikes

You may say: "Sure, but the government is just protecting the citizens. And I'm a law abiding citizen. Why should I care?" To that my reply is: Yes, but try saying the same thing when your government screwed up big time and you are in a protest. Knowing who you talk to on a day to day basis is probably fine.. ish. But when you are defending your rights, you don't want that information to be leaked and be used to target your allies.

GNUnet is designed to fix this. And CADET is GNUnet's replacement for TCP/IP. It's a protocol that is designed to be secure, private, and censorship resistant. It's also designed to work on any network. Only SSL is allowed? No problem, use the HTTPS plugin. You only have Bluetooth? Bluetooth daisy-chain to the rescue.

How CADET works

Disclaimer: I learned most of the following from the CADET paper and the GNUnet source code. I'm not a P2P expert. So if you find any mistakes, please let me know.

It's difficult to explain how CADET works in detail as it's still evolving. And more importantly, requires a lot of background knowledge. But I'll try to give a high level overview. GNUnet is is fromed by a set of computers that runs GNUnet (the thing we just installed earlier). Every node on the network has it's own peer identity and peers. Communicating directly to a peer is like sending messages on LAN in TCP/IP, no CADET routing is involved. You can still send messages to a peer using CADET, CADET just doesn't do much. The following image is the stack of GNUnet. Transport deals with direct communication between peers. While CADET deals with routing and delivering messages between peers in a secure and private way.

For distant nodes to communicate, CADET discovers a chain of nodes that can eventually reach the destination. This is a path. And multiple paths could exist between any two nodes. While sending a message, CADET picks one path for that message. And another for the next. And so on. This way no single entity can block all messages to a node. And backup paths are formed in case one path goes down. CADET also encrypts the messages so that no one can read them.

Peer identity, ports, channels, and tunnels

You might have seen a lot of confusing terms above. Let's go over them one by one.

A tunnel is a encrypted connection between two nodes. Think it as a virtual Ethernet cable. Many streams of traffic can go through a single tunnel. Furthermore, only one tunnel is created between two nodes. Different channels (we'll talk about next) are multiplexed over a single tunnel. The gnunetpp-cadet list-tunnel command we ran earlier shows the tunnels that is currently active.

A channel is a stream of traffic between two nodes. IT's like a socket in TCP/IP. A channel is created by a client and the server. The client can send messages to the server. And the server can send messages to the client. A channel is multiplexed over a tunnel thus, a single tunnel can have multiple channels.

A path is a chain of nodes that can be used to send messages between two nodes (by using a tunnel). A tunnel can have multiple paths. When a message is sent, a path is chosen from the available paths. You can see the paths to known nodes using gnunetpp-cadet list -p. Some nodes may have multiple paths and some have none. If none, it's likely CADET hasn't discovered a path to that node or the node is offline (among other reasons).

Your peer identity is a unique identifier for your node. It's a public key that is used to identify you among other things. It's the first line shown in ./examples/gnunetpp-cadet list

CADET vs libp2p

Why use CADET instead of libp2p? libp2p is a more mature solution then CADET. However, it's abilities are more limited. libp2p like it's name suggests focuses on P2P communication. CADET, besides P2P messaging, does mesh routing, censorship resistance, congestion control, and more. CADET also designed to work in very restricted situations.

Besides, GNUnet is not just a P2P library. It's a whole framework. No need to reinvent the wheel. GNUnet also provides a DHT just like libp2p. But also supports file sharing, decentralized name resolution and much more. I'm working on wrapping more GNUnet features in GNUnet++. Stay tuned.

Sending and receiving messages

Client side

Now let's write some code. We'll start with a simple example that sends a messages from stdin (usually your keyboard) to a server. Hopefully the following code is self-explanatory. If not, I'll explain later anyway. But notice that we are using C++ coroutiens instead of plain std::cin. This is very important. GNUnet itself is single threaded and fully async. So we cannot block the event loop at any point (on the same thread at least). Otherwise sending and receiving messages will not work.

#include <gnunetpp/gnuetpp-cadet.hpp>
#include <gnunetpp/gnunetpp.hpp>
using namespace gnunetpp;

cppcoro::task<> service(const GNUNET_CONFIGURATION_Handle* cfg)
{
    auto cadet = std::make_shared<CADET>(cfg);
    auto myself = crypto::to_string(crypto::myPeerIdentity(cfg));
    auto channel = cadet->connect(myself, "default", {});
    while (true) {
        auto msg = co_await scheduler::readStdin();
        channel->send(msg, {GNUNET_MESSAGE_TYPE_CADET_CLI});
    }
}

int main()
{
    gnunetpp::start(service);
}

And compile it with:

❯ g++ cadet_client.cpp -o cadet_client -std=c++20 -lgnunetpp -lgnunetcadet -lgnunetcore -lgnunetutil

Now let's start the server. Let's go with the example server in GNUnet++'s examples. We'll build one later. It's a simple echo server. It will print whatever you send to it (if the message type is GNUNET_MESSAGE_TYPE_CADET_CLI. Open another terminal and navigate to GNUnet++'s build directory. Run the following command and observe it starts listening on the "default" port.

❯ ./examples/gnunetpp-cadet server
Listening on Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0 port 'default'

Now let's run the client we just compiled. Type something and hit enter. You should see the same thing printed on the server side. Then Ctrl+C to exit.

❯ ./cadet_client
Hello, world!
^C

On the server side, you should see something like the following:

Hello, world!
* Connection closed for Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0

Nice! It works! I'll explain what we just wrote. First, we create an instance of CADET. It is our window into the CADET service. Then we get our own peer identity. As CADET uses peer identity to establish connection, we need to know our own peer identity to connect to our selves (i.e. There's no localhost in CADET).

auto cadet = std::make_shared<CADET>(cfg);
auto myself = crypto::to_string(crypto::myPeerIdentity(cfg));

Next we create a channel to ourselves on the "default" port. Important! CADET ports are just strings. The 3rd argument is a list of message type we want to receive. CADET ignores messages unless we separately tell it to receive them. In this case we don't care about any message. So we pass an empty list.

auto channel = co_await cadet->connect(myself, "default", {});

Finally we read stuff from stdin and send it

while (true) {
    auto msg = co_await scheduler::readStdin();
    channel->send(msg, {GNUNET_MESSAGE_TYPE_CADET_CLI});
}

Also be aware that there's no "connected" event. We just call connect and it will return a channel. CADET in this regards is more like UDP than TCP. If the other side is not listening, CADET will try reconnecting until it succeeds or give up (depending on advanced flags we'll not talk about). How to know if a peer doesn't exist or just isn't reply? You don't. A peer is as good as not existing if it can't reply in a reasonable time.

Side note: To specify a peer identity other then your own. You pur a string throught crypto::peerIdentity() function. For example, to connect to the peer with the identity Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0 (or any arbitrary peer) you can do the following:

auto peer = crypto::peerIdentity("Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0");
auto channel = co_await cadet->connect(peer, port, accept_types);

Server side

A CADET server is just as easy as a client.

#include <gnunetpp/gnuetpp-cadet.hpp>
#include <gnunetpp/gnunetpp.hpp>
#include <iostream>
using namespace gnunetpp;

std::shared_ptr<CADET> cadet;
cppcoro::task<> service(const GNUNET_CONFIGURATION_Handle* cfg)
{
    cadet = std::make_shared<CADET>(cfg);
    cadet->setReceiveCallback([](const CADETChannelPtr& channel, const std::string_view data, uint16_t type) {
        std::cout << data << std::flush;
    });
    cadet->openPort("default", {GNUNET_MESSAGE_TYPE_CADET_CLI});
}

int main()
{
    gnunetpp::start(service);
}

Compile and run it with:

❯ g++ cadet_server.cpp -o cadet_server -std=c++20 -lgnunetpp -lgnunetcadet -lgnunetcore -lgnunetutil
❯ ./cadet_server

Then open another terminal and run the client we wrote earlier. You should see the same thing printed on the server side.

❯ ./cadet_client
Hello, world!

On the server side, you should see something like the following:

Hello, world!

The server program works in a similar way as the client. One noticeable difference is that we keep the CADET instance in a global variable. This is because CADET shuts down when the instance is destroyed. And since there's no coroutine to keep the instance alive, we put it in a global variable.

std::shared_ptr<CADET> cadet;

Next we set a callback to receive messages to print the received message. We also open the "default" port to listen on. The last argument is a list of message types we want to receive. In this case we only want to receive messages with type GNUNET_MESSAGE_TYPE_CADET_CLI. These types are predefined in GNUnet. You can find them in gnunet/gnunet_mq_lib.h.

cadet->setReceiveCallback([](const CADETChannelPtr& channel, const std::string_view data, uint16_t type) {
    std::cout << data << std::flush;
});
cadet->openPort("default", {GNUNET_MESSAGE_TYPE_CADET_CLI});

Server behind NAT

CADET can work behind NAT and no extra configuration is needed. Let me quickly demonstrate it. I'll run a CADET client on my laptop connected to my home WiFi. Then a server runs on my workstation connected to my phone's hotspot. There'll be some more latency now as it's not a local connection thus requires more hops routing. But it works.

In the demo I'll be using the gnunetpp-cadet example program. Just because it shows more details than the quick example above. But the code is basically the same.

❯ ./examples/gnunetpp-cadet server
* Listening on Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0 port 'default'
* Connection established for QHBSEKRBVEYEX4D0R241V5G74NDARWP7FW93RAFBS8B1DTDH74MG
Hello, world!

❯ ./examples/gnunetpp-cadet client Z6QYNW5G63KM5DZ8S9V57NXC5F6756Y98NH6CBEQRM12D6FYXHY0
Hello, world!

That's it. Simple, isn't it? Alright, that's everything in the post. I'm hoping to write more about P2P networks in the future. Hopefully eventually find a pratical, real-world use case. Signing out.