If you have a server with multiple interfaces – either public and/or private – your routing table might look something like this:
sh# ip route list
default via 17.10.20.1 dev eth1 metric 100
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.51
17.10.20.0/23 dev eth1 proto kernel scope link src 17.10.20.51
105.104.72.16/28 dev eth2 proto kernel scope link src 105.104.72.23
This example shows one private interface with IP 192.168.0.51, two public interfaces with IPs 17.10.20.51 and 105.104.72.23, and a default route to 17.10.20.1. This means that any traffic to/from an IP outside the interface’s subnets is sent to 17.10.20.1 — and this is where problems occur (and probably why you’re reading this article). ;-)
In this example, you have two public interfaces, each (in theory) capable of receiving traffic from the internet, but any IP falling outside of their subnets is sent to 17.10.20.1. For example, a client connects from IP 1.2.3.4 to IP 105.104.72.23 on this server. Since 1.2.3.4 is not within subnet 105.104.72.16/28, the reply is sent to 17.10.20.1, *which is on the wrong interface*. If you want to route each interface properly (sending traffic back out the way it came), you have to setup multiple routing tables, each with it’s own default gateway.
Using the same example, you might create three routing tables; an “intern” table for eth0, a “prov_1” table for eth1, and a “prov_2” table for eth2. Here’s what the creation of those tables might look like:
sh# ip rule add to 17.10.20.0/23 table prov_1 prio 5000
sh# ip rule add to 105.104.72.16/28 table prov_2 prio 2000
sh# ip rule add to 192.168.0.0/24 table intern prio 1000
sh# ip rule add from 17.10.20.0/23 table prov_1 prio 5000
sh# ip rule add from 105.104.72.16/28 table prov_2 prio 2000
sh# ip rule add from 192.168.0.0/24 table intern prio 1000
sh# ip rule list
0: from all lookup local
1000: from all to 192.168.0.0/24 lookup intern
1000: from 192.168.0.0/24 lookup intern
2000: from all to 105.104.72.16/28 lookup prov_2
2000: from 105.104.72.16/28 lookup prov_2
5000: from all to 17.10.20.0/23 lookup prov_1
5000: from 17.10.20.0/23 lookup prov_1
32766: from all lookup main
32767: from all lookup default
With these rules in place, network traffic from a client with IP 1.2.3.4, connecting to IP 105.104.72.23, will be sent to the “prov_2” table, which will have the proper default gateway for that interface. The result is that traffic coming in on one interface will go back out on the same interface, to the correct default gateway IP.
Using 105.104.72.23 and “prov_2” as an example, the routing table itself might look like this:
sh# ip route add default via 105.104.72.17 dev eth2 metric 100 table prov_2
sh# ip route add 105.104.72.16/28 dev eth2 scope link src 105.104.72.23 table prov_2
sh# ip route list table prov_2
default via 105.104.72.17 dev eth2 metric 100
105.104.72.16/28 dev eth2 scope link src 105.104.72.23
If you have just one server with multiple interfaces, and those interfaces and subnets never change, then you could simply add these ip rule/route commands to a startup script (right after networking starts), or you could use the following script. It will create the proper rules and routes for multiple networks, no matter which interface they are on (including virtual interfaces).
The script depends on `/bin/bash` and the `ipcalc` binary (available on most, if not all distributions). The top-most section of the script must be configured for your network.
# --- START OF CONFIGURATION SECTION ---
rt_tables[100]="intern"
rt_tables[200]="prov_1"
rt_tables[300]="prov_2"
NetInfo () {
# ------------------------------gateway---------metric--table---prio----fwd'ing
case "$1" in
17.10.20.0/23) echo 17.10.20.1 100 prov_1 5000 1 ;;
84.70.129.64/28) echo 84.70.129.65 100 prov_2 4000 1 ;;
105.104.78.152/29) echo 105.104.78.153 100 prov_2 3000 1 ;;
105.104.72.16/28) echo 105.104.72.17 100 prov_2 2000 1 ;;
192.168.0.0/24) echo 192.168.0.1 100 intern 1100 0 ;;
10.201.50.0/24) echo 10.201.50.1 200 intern 1200 0 ;;
esac
}
# --- END OF CONFIGURATION SECTION ---
The configuration is fairly self-explanatory — an `rt_tables` array defines the routing table numbers and names used in the `/etc/iproute2/rt_tables` file, each `case` statement line defines the gateway for that network/netmask, its metric (in case you use multiple default gateways in the same table), the table name, the rule priority (see the `ip rule list` command), and if forwarding should be turned on or off for that interface.
Download the script here, or click on the “Copy plain Code” or “Open Code in New Window” symbol to copy-paste the script.
#!/bin/bash
#
# Copyright 2012 - Jean-Sebastien Morisset - https://surniaulula.com/
#
# This script is free software; you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the Free Software
# Foundation; either version 3 of the License, or (at your option) any later
# version.
#
# This script is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
# details at http://www.gnu.org/licenses/.
#
# --- START OF CONFIGURATION SECTION ---
rt_tables[100]="intern"
rt_tables[200]="prov_1"
rt_tables[300]="prov_2"
NetInfo () {
# ------------------------------gateway---------metric--table---prio----fwd'ing
case "$1" in
17.10.20.0/23) echo 17.10.20.1 100 prov_1 5000 1 ;;
84.70.129.64/28) echo 84.70.129.65 100 prov_2 4000 1 ;;
105.104.78.152/29) echo 105.104.78.153 100 prov_2 3000 1 ;;
105.104.72.16/28) echo 105.104.72.17 100 prov_2 2000 1 ;;
192.168.0.0/24) echo 192.168.0.1 100 intern 1100 0 ;;
10.201.50.0/24) echo 10.201.50.1 200 intern 1200 0 ;;
esac
}
# --- END OF CONFIGURATION SECTION ---
RouteNet () {
# return if we don't have all 8 arguments
[ "${#@}" != 8 ] && return
# name the arguments to keep things clear
dev="$1" ip="$2" net_mask="$3" gw="$4" metric="$5" table="$6" prio="$7" fwd="$8"
echo sysctl -w net.ipv4.conf.$dev.forwarding=$fwd
for action in del add
do
cat <<EOF
ip route $action $net_mask dev $dev scope link src $ip table $table
ip route $action default via $gw dev $dev metric $metric table $table
ip rule $action from $net_mask table $table prio $prio
ip rule $action to $net_mask table $table prio $prio
EOF
done
}
ShowTables () {
echo -e "\n$1 RULES"
ip rule list
for rt in main "${rt_tables[@]}"
do
echo -e "\n$1 TABLE $rt"
ip route show table $rt
done
echo ""
}
CheckDeps () {
if [ -z "`which ip`" ]
then
echo "\"ip\" binary not found!" >/dev/stderr
exit 1
elif [ -z "`which ipcalc`" ]
then
echo "\"ipcalc\" binary not found!" >/dev/stderr
exit 1
elif [ -z "`ip rule list|grep 'from all lookup main'`" ]
then
echo "kernel does not appear to support policy routing" >/dev/stderr
exit 1
fi
}
# parse the command line, skipping stuff we don't know, until there's nothing left
while :
do
for arg in "$@"
do
case $arg in
-t) no_change="1"; shift 1;;
*) shift 1 ;;
esac
continue 2
done
break
done
CheckDeps
echo ""
[ -z "$no_change" ] && \
cat /dev/null > /etc/iproute2/rt_tables
for rt in "${!rt_tables[@]}"
do
echo "adding routing table: $rt ${rt_tables[$rt]}"
[ -z "$no_change" ] && \
echo "$rt ${rt_tables[$rt]}" >> /etc/iproute2/rt_tables
done
interfaces="`ip addr show | sed -n 's/^[0-9][0-9]*: \([^:]*\): .*$/\1/p'`"
# reset the command-line arguments with the interface names
set -- $interfaces
ShowTables "BEFORE"
echo "EXEC RULES AND ROUTES"
{
for dev in "$@"
do
for ip_mask in `ip addr show $dev | sed -n 's/^\s*inet \([0-9\.\/]*\) .*$/\1/p'`
do
ip=${ip_mask%/*}
mask=${ip_mask#*/}
[ -z "$ip" -o -z "$mask" ] && echo "error reading \"ip addr show $dev\" info" >/dev/stderr
[ "$mask" -eq "32" ] && net="$ip" || \
net="`ipcalc -n $ip/$mask|sed -n 's/^network[=:][[:space:]]*\([0-9\.]*\).*$/\1/ip'`"
[ -z "$net" ] && echo "error reading \"ipcalc -n $ip/$mask\" output" >/dev/stderr
RouteNet $dev $ip $net/$mask `NetInfo $net/$mask`
done
done
# remove possible duplicate default routes etc from multiple ips on same network
# sort reverse to exec the deletes first, then the adds
} | sort -u -r | while read line
do
echo "$line"
[ -z "$no_change" ] && \
eval $line >/dev/null 2>&1
done
ShowTables "AFTER"
ip route flush cache
echo "FORWARDING"
sysctl -a 2>/dev/null | grep '^net\.ipv4\..*\.forwarding'
echo ""
