If you have a server with multiple interfaces – either public and/or private – your routing table might look something like this:
|
1 2 3 4 5 6 7 |
sh# ip route list default via 17.10.20.1 dev eth1 metric 100 192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.51 17.10.20.0/23 dev eth1 proto kernel scope link src 17.10.20.51 105.104.72.16/28 dev eth2 proto kernel scope link src 105.104.72.23 |
This example shows one private interface with IP 192.168.0.51, two public interfaces with IPs 17.10.20.51 and 105.104.72.23, and a default route to 17.10.20.1. This means that any traffic to/from an IP outside the interface’s subnets is sent to 17.10.20.1 — and this is where problems occur (and probably why you’re reading this article). ;-)
In this example, you have two public interfaces, each (in theory) capable of receiving traffic from the internet, but any IP falling outside of their subnets is sent to 17.10.20.1. For example, a client connects from IP 1.2.3.4 to IP 105.104.72.23 on this server. Since 1.2.3.4 is not within subnet 105.104.72.16/28, the reply is sent to 17.10.20.1, *which is on the wrong interface*. If you want to route each interface properly (sending traffic back out the way it came), you have to setup multiple routing tables, each with it’s own default gateway.
Using the same example, you might create three routing tables; an “intern” table for eth0, a “prov_1” table for eth1, and a “prov_2” table for eth2. Here’s what the creation of those tables might look like:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
sh# ip rule add to 17.10.20.0/23 table prov_1 prio 5000 sh# ip rule add to 105.104.72.16/28 table prov_2 prio 2000 sh# ip rule add to 192.168.0.0/24 table intern prio 1000 sh# ip rule add from 17.10.20.0/23 table prov_1 prio 5000 sh# ip rule add from 105.104.72.16/28 table prov_2 prio 2000 sh# ip rule add from 192.168.0.0/24 table intern prio 1000 sh# ip rule list 0: from all lookup local 1000: from all to 192.168.0.0/24 lookup intern 1000: from 192.168.0.0/24 lookup intern 2000: from all to 105.104.72.16/28 lookup prov_2 2000: from 105.104.72.16/28 lookup prov_2 5000: from all to 17.10.20.0/23 lookup prov_1 5000: from 17.10.20.0/23 lookup prov_1 32766: from all lookup main 32767: from all lookup default |
With these rules in place, network traffic from a client with IP 1.2.3.4, connecting to IP 105.104.72.23, will be sent to the “prov_2” table, which will have the proper default gateway for that interface. The result is that traffic coming in on one interface will go back out on the same interface, to the correct default gateway IP.
Using 105.104.72.23 and “prov_2” as an example, the routing table itself might look like this:
|
1 2 3 4 5 6 7 |
sh# ip route add default via 105.104.72.17 dev eth2 metric 100 table prov_2 sh# ip route add 105.104.72.16/28 dev eth2 scope link src 105.104.72.23 table prov_2 sh# ip route list table prov_2 default via 105.104.72.17 dev eth2 metric 100 105.104.72.16/28 dev eth2 scope link src 105.104.72.23 |
If you have just one server with multiple interfaces, and those interfaces and subnets never change, then you could simply add these ip rule/route commands to a startup script (right after networking starts), or you could use the following script. It will create the proper rules and routes for multiple networks, no matter which interface they are on (including virtual interfaces).
The script depends on /bin/bash and the ipcalc binary (available on most, if not all distributions). The top-most section of the script must be configured for your network.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# --- START OF CONFIGURATION SECTION --- rt_tables[100]="intern" rt_tables[200]="prov_1" rt_tables[300]="prov_2" NetInfo () { # ------------------------------gateway---------metric--table---prio----fwd'ing case "$1" in 17.10.20.0/23) echo 17.10.20.1 100 prov_1 5000 1 ;; 84.70.129.64/28) echo 84.70.129.65 100 prov_2 4000 1 ;; 105.104.78.152/29) echo 105.104.78.153 100 prov_2 3000 1 ;; 105.104.72.16/28) echo 105.104.72.17 100 prov_2 2000 1 ;; 192.168.0.0/24) echo 192.168.0.1 100 intern 1100 0 ;; 10.201.50.0/24) echo 10.201.50.1 200 intern 1200 0 ;; esac } # --- END OF CONFIGURATION SECTION --- |
The configuration is fairly self-explanatory — an rt_tables array defines the routing table numbers and names used in the /etc/iproute2/rt_tables file, each case statement line defines the gateway for that network/netmask, its metric (in case you use multiple default gateways in the same table), the table name, the rule priority (see the ip rule list command), and if forwarding should be turned on or off for that interface.
Download the script here, or click on the “Copy plain Code” or “Open Code in New Window” symbol to copy-paste the script.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 |
#!/bin/bash # # Copyright 2012 - Jean-Sebastien Morisset - https://surniaulula.com/ # # This script is free software; you can redistribute it and/or modify it under # the terms of the GNU General Public License as published by the Free Software # Foundation; either version 3 of the License, or (at your option) any later # version. # # This script is distributed in the hope that it will be useful, but WITHOUT # ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS # FOR A PARTICULAR PURPOSE. See the GNU General Public License for more # details at http://www.gnu.org/licenses/. # # --- START OF CONFIGURATION SECTION --- rt_tables[100]="intern" rt_tables[200]="prov_1" rt_tables[300]="prov_2" NetInfo () { # ------------------------------gateway---------metric--table---prio----fwd'ing case "$1" in 17.10.20.0/23) echo 17.10.20.1 100 prov_1 5000 1 ;; 84.70.129.64/28) echo 84.70.129.65 100 prov_2 4000 1 ;; 105.104.78.152/29) echo 105.104.78.153 100 prov_2 3000 1 ;; 105.104.72.16/28) echo 105.104.72.17 100 prov_2 2000 1 ;; 192.168.0.0/24) echo 192.168.0.1 100 intern 1100 0 ;; 10.201.50.0/24) echo 10.201.50.1 200 intern 1200 0 ;; esac } # --- END OF CONFIGURATION SECTION --- RouteNet () { # return if we don't have all 8 arguments [ "${#@}" != 8 ] && return # name the arguments to keep things clear dev="$1" ip="$2" net_mask="$3" gw="$4" metric="$5" table="$6" prio="$7" fwd="$8" echo sysctl -w net.ipv4.conf.$dev.forwarding=$fwd for action in del add do cat <<EOF ip route $action $net_mask dev $dev scope link src $ip table $table ip route $action default via $gw dev $dev metric $metric table $table ip rule $action from $net_mask table $table prio $prio ip rule $action to $net_mask table $table prio $prio EOF done } ShowTables () { echo -e "\n$1 RULES" ip rule list for rt in main "${rt_tables[@]}" do echo -e "\n$1 TABLE $rt" ip route show table $rt done echo "" } CheckDeps () { if [ -z "`which ip`" ] then echo "\"ip\" binary not found!" >/dev/stderr exit 1 elif [ -z "`which ipcalc`" ] then echo "\"ipcalc\" binary not found!" >/dev/stderr exit 1 elif [ -z "`ip rule list|grep 'from all lookup main'`" ] then echo "kernel does not appear to support policy routing" >/dev/stderr exit 1 fi } # parse the command line, skipping stuff we don't know, until there's nothing left while : do for arg in "$@" do case $arg in -t) no_change="1"; shift 1;; *) shift 1 ;; esac continue 2 done break done CheckDeps echo "" [ -z "$no_change" ] && \ cat /dev/null > /etc/iproute2/rt_tables for rt in "${!rt_tables[@]}" do echo "adding routing table: $rt ${rt_tables[$rt]}" [ -z "$no_change" ] && \ echo "$rt ${rt_tables[$rt]}" >> /etc/iproute2/rt_tables done interfaces="`ip addr show | sed -n 's/^[0-9][0-9]*: \([^:]*\): .*$/\1/p'`" # reset the command-line arguments with the interface names set -- $interfaces ShowTables "BEFORE" echo "EXEC RULES AND ROUTES" { for dev in "$@" do for ip_mask in `ip addr show $dev | sed -n 's/^\s*inet \([0-9\.\/]*\) .*$/\1/p'` do ip=${ip_mask%/*} mask=${ip_mask#*/} [ -z "$ip" -o -z "$mask" ] && echo "error reading \"ip addr show $dev\" info" >/dev/stderr [ "$mask" -eq "32" ] && net="$ip" || \ net="`ipcalc -n $ip/$mask|sed -n 's/^network[=:][[:space:]]*\([0-9\.]*\).*$/\1/ip'`" [ -z "$net" ] && echo "error reading \"ipcalc -n $ip/$mask\" output" >/dev/stderr RouteNet $dev $ip $net/$mask `NetInfo $net/$mask` done done # remove possible duplicate default routes etc from multiple ips on same network # sort reverse to exec the deletes first, then the adds } | sort -u -r | while read line do echo "$line" [ -z "$no_change" ] && \ eval $line >/dev/null 2>&1 done ShowTables "AFTER" ip route flush cache echo "FORWARDING" sysctl -a 2>/dev/null | grep '^net\.ipv4\..*\.forwarding' echo "" |