Setup an Amazon Linux 2023 EC2 Instance as a Gateway #

These notes explain how to set up a WireGuard and/or SSH gateway node for accessing a cluster without a static public IP address (homelab).

Alternatively, TailGate, ZeroTier, and similar services give free tier offerings to achieve a similar goal. However, when using them we need to take into account that the network management infrastructure is essentially “black box” managed by a less established company (compared to AWS, Azure, GCP).

EC2 Template #

To speed up launching of a new gateway instance we can create an EC2 template. Since they are region specific, we have to create a template in each region where we might need to run the gateway node.

In AWS web console make sure the right region is selected, e.g. my private cluster has better pings to ap-northeast-1 (Tokyo) and ap-northeast-3 (Osaka).

Most of the instructions below can be placed into the user script.

If we require a WireGuard tunnel, we should install the wireguard-tools package and enable IP forwarding on the node:

yum update -y
yum install wireguard-tools
sysctl net.ipv4.conf.all.forwarding=1 | tee -a /etc/sysctl.d/99-nat-forwarding.conf

WireGuard config templates #

umask 077
mkdir -p /etc/wireguard
# Optionally we can create per client dirs for keys
#mkdir -p /etc/wireguard/wg0-150-keys

# Generate server key
cd /etc/wireguard
wg genkey | tee server.key | wg pubkey > server.pub

In /etc/ssh/sshd_config, set

PasswordAuthentication no
PermitEmptyPasswords no
ClientAliveInterval 20
ClientAliveCountMax 3

Then restart the sshd service:

systemctl restart sshd
#netstat -ltnp

/etc/wireguard/gen-wg-conf.sh

#!/bin/bash

WG0_CONF="/etc/wireguard/wg0.conf"

WG_KEY="$(cat server.key)"

# Server's CIDR, to ensure routing is done properly to between multiple clients, use /24
WG_ADDRESS="A.B.C.D/24"
WG_PORT=51820

WG_PEER150_PUB="$(cat peer150.pub)"
WG_PEER150_PSK="$(cat peer150.psk)"
WG_PEER150_IP="A.B.C.D/32"
WG_PEER150_SUBNETS="X.X.X.X/24, Y.Y.Y.Y/24"

echo "# wg0 server

[Interface]
MTU = 1280
Address = $WG_ADDRESS
ListenPort = $WG_PORT
PrivateKey = $WG_KEY
PostUp = /etc/wireguard/scripts/wg0-post-up.sh
PostDown = /etc/wireguard/scripts/wg0-post-down.sh

[Peer]
PublicKey = $WG_PEER150_PUB
PresharedKey = $WG_PEER150_PSK
AllowedIPs = $WG_PEER150_IP, $WG_PEER150_SUBNETS

" > $WG0_CONF

chmod 700 $WG0_CONF

We can set iptables rules for the wg0 interface using “post-up” and “post-down” scripts.

/etc/wireguard/scripts/wg0-post-up.sh

#!/bin/bash
IPT="/sbin/iptables"
IPT6="/sbin/ip6tables"

#************************#
#** Set correct values **#
#************************#
IN_FACE="ens5" # NIC connected to the internet
WG_FACE="wg0" # WG NIC 
WG_PORT=51820 # WG udp port

## IPv4 ##
$IPT -t nat -I POSTROUTING 1 -o $IN_FACE -j MASQUERADE  # if used as an exit node
$IPT -I INPUT 1 -i $WG_FACE -j ACCEPT
$IPT -I FORWARD 1 -i $IN_FACE -o $WG_FACE -j ACCEPT
$IPT -I FORWARD 1 -i $WG_FACE -o $IN_FACE -j ACCEPT
$IPT -I INPUT 1 -i $IN_FACE -p udp --dport $WG_PORT -j ACCEPT

## IPv6 ##
# Optionally, set the same rules on IPv6

#SUB_NET_6="" # WG IPv6 sub/net (set IPv6 CIDR)
## $IPT6 -t nat -I POSTROUTING 1 -s $SUB_NET_6 -o $IN_FACE -j MASQUERADE  # if used as an exit node
## $IPT6 -I INPUT 1 -i $WG_FACE -j ACCEPT
## $IPT6 -I FORWARD 1 -i $IN_FACE -o $WG_FACE -j ACCEPT
## $IPT6 -I FORWARD 1 -i $WG_FACE -o $IN_FACE -j ACCEPT

/etc/wireguard/scripts/wg0-post-down.sh

#!/bin/bash
IPT="/sbin/iptables"
IPT6="/sbin/ip6tables"

IN_FACE="ens5" # NIC connected to the internet
WG_FACE="wg0"  # WG NIC 
WG_PORT=51820 # WG udp port

# IPv4 rules #
$IPT -t nat -D POSTROUTING -o $IN_FACE -j MASQUERADE  # if used as an exit node
$IPT -D INPUT -i $WG_FACE -j ACCEPT
$IPT -D FORWARD -i $IN_FACE -o $WG_FACE -j ACCEPT
$IPT -D FORWARD -i $WG_FACE -o $IN_FACE -j ACCEPT
$IPT -D INPUT -i $IN_FACE -p udp --dport $WG_PORT -j ACCEPT

## IPv6 ##
# If the optional IPv6 rules were set in wg0-post-up.sh, the following rules must be set as well

# IPv6 rules (uncomment) and set SUB_NET_6#
#SUB_NET_6="" # WG IPv6 sub/net
## $IPT6 -t nat -D POSTROUTING -s $SUB_NET_6 -o $IN_FACE -j MASQUERADE  # if used as an exit node
## $IPT6 -D INPUT -i $WG_FACE -j ACCEPT
## $IPT6 -D FORWARD -i $IN_FACE -o $WG_FACE -j ACCEPT
## $IPT6 -D FORWARD -i $WG_FACE -o $IN_FACE -j ACCEPT

Persistent Spot Instance #

While a prod server is normally required to stay online 24/7 without interruptions, for a personal cluster gateway, we can tolerate possible interruptions while taking advantage of more flexible pricing of EC2 spot instances.

Some things to keep in mind about spot instances (compared to more robust on-demand instances):

Generally gives good cost savings on t3.nano, t3a.nano (>50%).
AWS can interrupt spot instances at any time, so the gateway must have a stateless configuration.
For 24/7 operation, when the interruption behavior is configured properly, instead of being destroyed, the same instance can be automatically restarted again.
By attaching an elastic IP to a separately created network interface, then specifying the said interface in the spot instance’s config, we can simplify re-deployments of the gateway instance.

The following resources must exist before launching the persistent spot instance gateway:

EC2 launch template (see the previous section)
subnet in the same availability zone;
security group:
- rulse that expose the SSH port to internal network;
- if this is a WireGuard gateway: rules should expose the WireGuard port;
- if this is an SSH gateway: rules should expose the SSH port.

AWS EC2 (persistent spot instance) launch config:

{
  "MaxCount": 1,
  "MinCount": 1,
  "ImageId": "[ami-NNNN]",
  "InstanceType": "t3a.nano",
  "InstanceInitiatedShutdownBehavior": "stop",
  "KeyName": "[my ssh key previously uploaded to aws in this region]",
  "NetworkInterfaces": [
    {
      "SubnetId": "[subnet-NNNN]",
      "DeviceIndex": 0,
      "Groups": [
        "[sg-NNNN my security group that exposes the WireGuard port to public and SSH port to internal network]"
      ]
    }
  ],
  "LaunchTemplate": {
    "LaunchTemplateId": "[lt-NNNN my launch temlate]",
    "Version": "4"
  },
  "CreditSpecification": {
    "CpuCredits": "standard"
  },
  "TagSpecifications": [
    {
      "ResourceType": "instance",
      "Tags": [
        {
          "Key": "Name",
          "Value": "[name of the gateway]"
        }
      ]
    },
    {
      "ResourceType": "volume",
      "Tags": [
        {
          "Key": "Name",
          "Value": "[name of the gateway]"
        }
      ]
    },
    {
      "ResourceType": "spot-instances-request",
      "Tags": [
        {
          "Key": "Name",
          "Value": "[name of the gateway]"
        }
      ]
    },
    {
      "ResourceType": "network-interface",
      "Tags": [
        {
          "Key": "Name",
          "Value": "[name of the gateway]"
        }
      ]
    }
  ],
  "InstanceMarketOptions": {
    "MarketType": "spot",
    "SpotOptions": {
      "InstanceInterruptionBehavior": "stop",
      "SpotInstanceType": "persistent"
    }
  },
  "PrivateDnsNameOptions": {
    "HostnameType": "resource-name"
  }
}