Simple dual upstream gateways in CentOS, revisited

Sun 05 May 2019

Tags: , ,

Recently had to setup a few servers that needed dual upstream gateways, and used an ancient blog post I wrote 11 years ago (!) to get it all working. This time around I hit a gotcha that I hadn't noted in that post, and used a simpler method to define the rules, so this is an updated version of that post.

Situation: you have two upstream gateways (gw1 and gw2) on separate interfaces and subnets on your linux server. Your default route is via gw1 (so all outward traffic, and most incoming traffic goes via that), but you want to be able to use gw2 as an alternative ingress pathway, so that packets that have come in on gw2 go back out that interface.

(Everything below done as root, so sudo -i first if you need to.)

1) First, define a few variables to make things easier to modify/understand:

# The device/interface on the `gw2` subnet
GW2_DEV=eth1
# The ip address of our `gw2` router
GW2_ROUTER_ADDR=172.16.2.254
# Our local ip address on the `gw2` subnet i.e. $GW2_DEV's address
GW2_LOCAL_ADDR=172.16.2.10

2) The gotcha I hit was that 'strict reverse-path filtering' in the kernel will drop all asymmetrically routed entirely, which will kill our response traffic. So the first thing to do is make sure that is either turned off or set to 'loose' instead of 'strict':

# Check the rp_filter setting for $GW2_DEV
# A value of '0' means rp_filtering is off, '1' means 'strict', and '2' means 'loose'
$ cat /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
1
# For our purposes values of either '0' or '2' will work. '2' is slightly
# more conservative, so we'll go with that.
echo 2 > /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
$ cat /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
2

3) Define an extra routing table called gw2 e.g.

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local tables
#
102     gw2
#

4) Add a default route via gw2 (here 172.16.2.254) to the gw2 routing table:

$ echo "default table gw2 via $GW2_ROUTER_ADDR" > /etc/sysconfig/network-scripts/route-${GW2_DEV}
$ cat /etc/sysconfig/network-scripts/route-${GW2_DEV}
default table gw2 via 172.16.2.254

5) Add an iproute 'rule' saying that packets that come in on our $GW2_LOCAL_ADDR should use routing table gw2:

$ echo "from $GW2_LOCAL_ADDR table gw2" > /etc/sysconfig/network-scripts/rule-${GW2_DEV}
$ cat /etc/sysconfig/network-scripts/rule-${GW2_DEV}
from 172.16.2.10 table gw2

6) Take $GW2_DEV down and back up again, and test:

$ ifdown $GW2_DEV
$ ifup $GW2_DEV

# Test that incoming traffic works as expected e.g. on an external server
$ ssh -v server-via-gw2

For more, see:

View Comments

Simple dual upstream gateways in CentOS

Fri 23 May 2008

Tags: , ,


Update 2019-05-05: see this revised post for a simpler implementation method and a gotcha to watch out for. HT to Jim MacLeod for suggested improvements in his comments below.



Had to setup some simple policy-based routing on CentOS again recently, and had forgotten the exact steps. So here's the simplest recipe for CentOS that seems to work. This assumes you have two upstream gateways (gw1 and gw2), and that your default route is gw1, so all you're trying to do is have packets that come in on gw2 go back out gw2.

1) Define an extra routing table e.g.

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local tables
#
102     gw2
#

2) Add a default route via gw2 (here 172.16.2.254) to table gw2 on the appropriate interface (here eth1) e.g.

$ cat /etc/sysconfig/network-scripts/route-eth1
default table gw2 via 172.16.2.254

3) Add an ifup-local script to add a rule to use table gw2 for eth1 packets e.g.

$ cat /etc/sysconfig/network-scripts/ifup-local
#!/bin/bash
#
# Script to add/delete routing rules for gw2 devices
#

GW2_DEVICE=eth1
GW2_LOCAL_ADDR=172.16.2.1

if [ $(basename $0) = ifdown-local ]; then
  OP=del
else
  OP=add
fi

if [ "$1" = "$GW2_DEVICE" ]; then
  ip rule $OP from $GW2_LOCAL_ADDR table gw2
fi

4) Use the ifup-local script also as ifdown-local, to remove that rule

$ cd /etc/sysconfig/network-scripts
$ ln -s ifup-local ifdown-local

5) Restart networking, and you're done!

# service network restart

For more, see:

View Comments