WireGuard Mesh VPN Encryption for MariaDB Galera Cluster

This tutorial is going to show you how to build a WireGuard mesh VPN and use it to encrypt replication traffic in MariaDB Galera cluster.

Prerequisites

This is not a step-by-step tutorial. To use the knowledge in this article, it’s assumed that you have a basic understanding of how to set up WireGuard VPN and MariaDB Galera Cluster.

What’s a Mesh VPN?

In computer networking:

• A mesh topology is where each node in the network has a direct connection to every other node.
• A star topology is where a central node controls the flow of connection. Nodes in the network can’t connect to each other directly but have to communicate through the central node. A typical example is a home private network.

The most common VPNs in the market use the star topology. Each VPN client has to connect to a central VPN server. And WireGuard can be set up in this way.

A mesh VPN has no central server. Every node in the mesh VPN can connect to each other directly, so there’s no single point of failure. This is also known as peer-to-peer VPN, or point-to-point VPN (not to be confused with the PPTP VPN protocol). You can use it to build a distributed secure network for your servers running in the cloud. WireGuard is an open-source ultra-fast VPN protocol that can be used to build a full-mesh network.

WireGuard is super lightweight and currently doesn’t implement UDP hole punching. If you want two nodes to connect to each other with WireGuard, one of them has to listen on a public IP address to accept requests. So its mesh networking capability isn’t comparable to Nebula or ZeroTier, which allows you to build a full-mesh VPN for nodes behind NAT devices. However, the peer-to-peer feature in WireGuard works really well for servers running in data centers and it’s very easy to set up.

What’s MariaDB Galera Cluster?

Galera Cluster is a synchronous multi-master cluster for MySQL, MariaDB, and Percona database servers to implement high-availability data redundancy. A multi-master cluster allows read and writes to any node in the cluster. Data modifications on any node are replicated to all other nodes.

Why Use WirdGuard Mesh VPN with MariaDB Galera Cluster

Because the built-in encryption mechanism in MariaDB Galera cluster isn’t robust for production environment.

In 2019, I wrote a tutorial explaining how to encrypt MariaDB Galera replication traffic on the public Internet. However, the TLS settings for Galera cluster aren’t very stable. I have seen a parameter changed back and forth.

The encrypt variable for snapshot state transfer (SST) has 5 possible values: 0, 1, 2, 3, 4. 0 means encryption is disabled.

• In 2019, the encryption mode for the mariabackup SST method should be set to 4. 1, 2 and 3 didn’t work.
• Now in 2021, the MariaDB documentation says that you should use mode 2 or 3 for the mariabackup SST method.

I didn’t expect this change and it caused the following TLS connection error when I trying to join a new node to the cluster.

[ERROR] WSREP: client_handshake error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed

It’s not easy to debug this error. And I still don’t know what’s wrong with my configurations. Whatsmore, changing the TLS settings requires restarting the entire cluster. So I gave up and started looking for alternative solutions. It occurred to me that I can use VPN to encrypt traffic and WireGuard allows me to build a peer-to-peer VPN, which is very suitable for use with the MariaDB Galera cluster.

WireGuard Mesh VPN Setup

It’s very easy to set up a mesh VPN with WireGuard. You simply need to list every peer in your WireGuard configuration file. I use /etc/wireguard/mesh.conf as the config file.

[Interface]
Address = 172.16.0.1/24
ListenPort = 51820
PrivateKey = RofXta1Y6D1mSP+KH35SUMymVWxffSZl5KQ1CyhBAlM=

[Peer]
PublicKey = ulkmsYKuoGf2G/5MGUZSrb8yBW1b+07zqzlnuP1A8MY=
AllowedIPs = 172.16.0.2/32
Endpoint = IP_Address_of_Node_2:51820
PersistentKeepalive = 25

[Peer]
PublicKey = d47QT5s5Rh75yQSvJbgzdTdma0qvin14ZvWjGx9nr0U=
AllowedIPs = 172.16.0.3/32
Endpoint = IP_Address_of_Node_3:51820
PersistentKeepalive = 25

My Galera cluster has 3 nodes, so I use also 3 nodes in my WireGuard configuration file. The private IP range 172.16.0.0/24 is used for the private network. It’s important that you add a ListenPort directive in the [Interface] section and an Endpoint directive in each [Peer] section. If you have multiple configuration files under /etc/wireguard/ directory, then only one of the config files should contain a ListenPort directive, because only one interface can bind to the 51820 UDP port.

Once you have configured all your WireGuard nodes, start the WireGuard interface on each node.

sudo systemctl start wg-quick@mesh.service

And enable auto-start at boot time.

sudo systemctl enable wg-quick@mesh.service

You should now be able to ping each VPN private IP address. If you can’t ping, check if your WireGuard is listening on UDP port 51820.

sudo ss -lnpu

Also, make sure you configure firewall to allow each node to access UDP port 51820.

Galera Cluster Setup

You simply need to delete the TLS encryption settings in your Galera configuration file. My full Galera cluster configurations is as follows.

[galera]
# Mandatory settings
wsrep_on                 = ON
wsrep_provider           = /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name       = "MariaDB Galera Cluster"
wsrep_cluster_address    = "gcomm://172.16.0.1,172.16.0.2,172.16.0.3"
binlog_format            = row
default_storage_engine   = InnoDB
innodb_autoinc_lock_mode = 2
innodb_force_primary_key = 1
innodb_doublewrite       = 1

# Allow server to accept connections on all interfaces.
bind-address = 0.0.0.0

# By default, MariaDB error logs are sent to journald, which can be hard to digest sometimes. 
# The following line will save error messages to a plain file. 
log_error = /var/log/mysql/error.log

# Optional settings
wsrep_slave_threads            = 4
innodb_flush_log_at_trx_commit = 0
wsrep_node_name                = node1
wsrep_node_address             = "172.16.0.1"

wsrep_provider_options="gcache.size = 1G; gcache.recover = yes;"

#SST method
wsrep_sst_method = mariabackup
wsrep_sst_auth = mariabackup:secret_password

[sst]
encrypt=0

Obviously, you need to change the wsrep_node_name and wsrep_node_address for each node in the Galera cluster. Also, make sure you configure firewall to allow traffic from the 172.16.0.0/24 network. If you use the UFW firewall on Ubuntu, then execute the following command on each node.

sudo ufw insert 1 allow in from 172.16.0.0/24

Wrapping Up

I hope this tutorial helped you encrypt replication traffic in MariaDB Galera cluster with WireGuard mesh VPN.