Port Forwarding for Linux KVM guests

If you are hosting a couple of virtual machines off an Linux KVM Host, chances are, you do not only want to reach the outer network from the machines, but you might also want to publish externally on port N a service that is on machine with IP: X and port Y.

If you are already confident with iptables, you might think you already know how to achieve that.

Chances are: it's not that simple.

To begin with, any custom rule you set up, be it the PREROUTING, FORWARD or other you are thinking, will remain in place when you happen to powercycle the virtual machine. Also, tcp sessions are stuck before timing out, and this leads to a couple of set ups where you expect things to work out, until they won't.

How to Do it

If you scoured the internet hard enough, you might have come to find the following hook

#!/bin/bash

# IMPORTANT: Change the "VM NAME" string to match your actual VM Name.
# In order to create rules to other VMs, just duplicate the below block and configure
# it accordingly.
if [ "${1}" = "debian-alpha" ]; then

   # Update the following variables to fit your setup
   GUEST_IP=....
   GUEST_PORT=...
   HOST_PORT=...

   if [ "${2}" = "stopped" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${2}" = "start" ] || [ "${2}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
fi

I found it here, not sure what is the original source.

If you:

copy the script in the filesystem location: /etc/libvirt/hooks/qemu
edit the variables: GUEST_IP,GUEST_PORT,HOST_PORT
make it executable

it will probably do what you need.

But, of course, this is of very bad maintainability. Every time you will need to publish something on the outside port of the host (HOST_PORT) you will need to manually copy-paste the entire block, and configure it for the specific guest-ip:guest-port that you need to expose.

A few thoughts

I see a lot of chances for getting it wrong and, besides, adding a wrong block might result in damaging the good ones,too.

My first change at it was the following, that looks better to me:

#!/bin/bash -x

exec > >(tee /var/log/qemu-hook.log|logger -t qemu-hook -s 2>/dev/console) 2>&1
sqlite_cmd="sqlite3 ${services_db} "

vm_name=${1}
vm_state=${2}

function forward_port(){
   HOST_PORT=$1
   GUEST_IP=$2
   GUEST_PORT=$3

   if [ "${vm_state}" = "stopped" ] || [ "${vm_state}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${vm_state}" = "start" ] || [ "${vm_state}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
}


if [ "${vm_name}" = "fancyname" ]; then
   forward_port 1022 192.168.1.101 22
fi

That is: most of the iptables logic is in a shell function, that you can call repeatedly. The last three lines must be copied over for other machine from fancyname.

Here you can see that SSH traffic for the local instance 192.168.1.101 on port 22 is exposed from outside the host on port 1022.

The hook above is apparently called with 2 parameters: the VM name and the actual State, each and everytime the VM changes state. This way there are no dangling iptables rules when the Virtual Machine is off.

Also this latest version has its weak point: in a perfect world you would not need to manually edit such a script all the time.

And I'll tell you what a perfect world looks like. There should be somewhere a sqlite db with the following tables:

CREATE TABLE machine(id char(64) NOT NULL PRIMARY KEY, mac char(18), cpus int, disk text, ram text);
CREATE TABLE service(port int NOT NULL PRIMARY KEY, id_machine char(64) NOT NULL, service_host char(64) NOT NULL, service_port int NOT NULL);

In machine, there should be some general info about the machine we're talking about, so that you can recognize it.

In service you want to code the three variables: **HOST_PORT, GUEST_IP, GUEST_PORT ** and possibly HOST_PORT is unique, so that you don't mistakenly try to forward the same external port to multiple local service:port.

And let us say that you have a smart way to populate such table (spoiler alert: I do). All you have to do in the above hook is to just printout those service rows.

For example:

function machine_services()
{
  id_machine=$1
  ports=$(${sqlite_cmd} "select port from service where id_machine='${id_machine}'")
  for p in ${ports}; do
    service_host=$(${sqlite_cmd} "select service_host from service where id_machine='${id_machine}' and port='${p}'")
    service_port=$(${sqlite_cmd} "select service_port from service where id_machine='${id_machine}' and port='${p}'")
    forward_port ${p} ${service_host} ${service_port}
  done
}

The whole hook looks now like this for me:

#!/bin/bash -x

exec > >(tee /var/log/qemu-hook.log|logger -t qemu-hook -s 2>/dev/console) 2>&1
services_db=~/.kvm-machines.db
sqlite_cmd="sqlite3 ${services_db} "

vm_name=${1}
vm_state=${2}

function forward_port(){
   HOST_PORT=$1
   GUEST_IP=$2
   GUEST_PORT=$3

   if [ "${vm_state}" = "stopped" ] || [ "${vm_state}" = "reconnect" ]; then
        /sbin/iptables -D FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -D PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
   if [ "${vm_state}" = "start" ] || [ "${vm_state}" = "reconnect" ]; then
        /sbin/iptables -I FORWARD -o virbr0 -p tcp -d $GUEST_IP --dport $GUEST_PORT -j ACCEPT
        /sbin/iptables -t nat -I PREROUTING -p tcp --dport $HOST_PORT -j DNAT --to $GUEST_IP:$GUEST_PORT
   fi
}

function machine_services()
{
  id_machine=$1
  ports=$(${sqlite_cmd} "select port from service where id_machine='${id_machine}'")
  for p in ${ports}; do
    service_host=$(${sqlite_cmd} "select service_host from service where id_machine='${id_machine}' and port='${p}'")
    service_port=$(${sqlite_cmd} "select service_port from service where id_machine='${id_machine}' and port='${p}'")
    forward_port ${p} ${service_host} ${service_port}
  done
}

###

machine_services ${vm_name}

In this way, once you populare the service table with the right data, everytime you start and stop the corresponding virtual machine, the services published on the configured ports are forwarded automatically, so that you can access them from outside the Host operating system.

[git] [linux]