Simple dual upstream gateways in CentOS, revisited

Sun 05 May 2019

Tags: , ,

Recently had to setup a few servers that needed dual upstream gateways, and used an ancient blog post I wrote 11 years ago (!) to get it all working. This time around I hit a gotcha that I hadn't noted in that post, and used a simpler method to define the rules, so this is an updated version of that post.

Situation: you have two upstream gateways (gw1 and gw2) on separate interfaces and subnets on your linux server. Your default route is via gw1 (so all outward traffic, and most incoming traffic goes via that), but you want to be able to use gw2 as an alternative ingress pathway, so that packets that have come in on gw2 go back out that interface.

(Everything below done as root, so sudo -i first if you need to.)

1) First, define a few variables to make things easier to modify/understand:

# The device/interface on the `gw2` subnet
GW2_DEV=eth1
# The ip address of our `gw2` router
GW2_ROUTER_ADDR=172.16.2.254
# Our local ip address on the `gw2` subnet i.e. $GW2_DEV's address
GW2_LOCAL_ADDR=172.16.2.10

2) The gotcha I hit was that 'strict reverse-path filtering' in the kernel will drop all asymmetrically routed entirely, which will kill our response traffic. So the first thing to do is make sure that is either turned off or set to 'loose' instead of 'strict':

# Check the rp_filter setting for $GW2_DEV
# A value of '0' means rp_filtering is off, '1' means 'strict', and '2' means 'loose'
$ cat /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
1
# For our purposes values of either '0' or '2' will work. '2' is slightly
# more conservative, so we'll go with that.
echo 2 > /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
$ cat /proc/sys/net/ipv4/conf/$GW2_DEV/rp_filter
2

3) Define an extra routing table called gw2 e.g.

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local tables
#
102     gw2
#

4) Add a default route via gw2 (here 172.16.2.254) to the gw2 routing table:

$ echo "default table gw2 via $GW2_ROUTER_ADDR" > /etc/sysconfig/network-scripts/route-${GW2_DEV}
$ cat /etc/sysconfig/network-scripts/route-${GW2_DEV}
default table gw2 via 172.16.2.254

5) Add an iproute 'rule' saying that packets that come in on our $GW2_LOCAL_ADDR should use routing table gw2:

$ echo "from $GW2_LOCAL_ADDR table gw2" > /etc/sysconfig/network-scripts/rule-${GW2_DEV}
$ cat /etc/sysconfig/network-scripts/rule-${GW2_DEV}
from 172.16.2.10 table gw2

6) Take $GW2_DEV down and back up again, and test:

$ ifdown $GW2_DEV
$ ifup $GW2_DEV

# Test that incoming traffic works as expected e.g. on an external server
$ ssh -v server-via-gw2

For more, see:

View Comments

Setting up a file transfer host

Fri 09 Feb 2018

Tags: , , ,

Had to setup a new file transfer host recently, with the following requirements:

  • individual login accounts required (for customers, no anonymous access)
  • support for (secure) downloads, ideally via a browser (no special software required)
  • support for (secure) uploads, ideally via sftp (most of our customers are familiar with ftp)

Our target was RHEL/CentOS 7, but this should transfer to other linuxes pretty readily.

Here's the schema we ended up settling on, which seems to give us a good mix of security and flexibility.

  • use apache with HTTPS and PAM with local accounts, one per customer, and nologin shell accounts
  • users have their own groups (group=$USER), and also belong to the sftp group
  • we use the users group for internal company accounts, but NOT for customers
  • customer data directories live in /data
  • we use a 3-layer hierarchy for security: /data/chroot_$USER/$USER are created with a nologin shell
  • the /data/chroot_$USER directory must be owned by root:$USER, with permissions 750, and is used for an sftp chroot directory (not writeable by the user)
  • the next-level /data/chroot_$USER/$USER directory should be owned by $USER:users, with permissions 2770 (where users is our internal company user group, so both the customer and our internal users can write here)
  • we also add an ACL to /data/chroot_$USER to allow the company-internal users group read/search access (but not write)

We just use openssh internal-sftp to provide sftp access, with the following config:

Subsystem sftp internal-sftp
Match Group sftp
  ChrootDirectory /data/chroot_%u
  X11Forwarding no
  AllowTcpForwarding no
  ForceCommand internal-sftp -d /%u

So we chroot sftp connections to /data/chroot_$USER and then (via the ForceCommand) chdir to /data/chroot_$USER/$USER, so they start off in the writeable part of their tree. (If they bother to pwd, they see that they're in /$USER, and they can chdir up a level, but there's nothing else there except their $USER directory, and they can't write to the chroot.)

Here's a slightly simplified version of the newuser script we use:

die() {
  echo $*
  exit 1
}

test -n "$1" || die "usage: $(basename $0) <username>"

USERNAME=$1

# Create the user and home directories
mkdir -p /data/chroot_$USERNAME/$USERNAME
useradd --user-group -G sftp -d /data/chroot_$USERNAME/$USERNAME -s /sbin/nologin $USERNAME

# Set home directory permissions
chown root:$USERNAME /data/chroot_$USERNAME
chmod 750 /data/chroot_$USERNAME
setfacl -m group:users:rx /data/chroot_$USERNAME
chown $USERNAME:users /data/chroot_$USERNAME/$USERNAME
chmod 2770 /data/chroot_$USERNAME/$USERNAME

# Set user password manually
passwd $USERNAME

And we add an apache config file like the following to /etc/httpd/user.d:

Alias /CUSTOMER /data/chroot_CUSTOMER/CUSTOMER
<Directory /data/chroot_CUSTOMER/CUSTOMER>
Options +Indexes
Include "conf/auth.conf"
Require user CUSTOMER
</Directory>

(with CUSTOMER changed to the local username), and where conf/auth.conf has the authentication configuration against our local PAM users and allows internal company users access.

So far so good, but how do we restrict customers to their own /CUSTOMER tree?

That's pretty easy too - we just disallow customers from accessing our apache document root, and redirect them to a magic '/user' endpoint using an ErrorDocument 403 directive:

<Directory /var/www/html>
Options +Indexes +FollowSymLinks
Include "conf/auth.conf"
# Any user not in auth.conf, redirect to /user
ErrorDocument 403 "/user"
</Directory>

with /user defined as follows:

# Magic /user endpoint, redirecting to /$USERNAME
<Location /user>
Include "conf/auth.conf"
Require valid-user
RewriteEngine On
RewriteCond %{LA-U:REMOTE_USER} ^[a-z].*
RewriteRule ^\/(.*)$ /%{LA-U:REMOTE_USER}/ [R]
</Location>

The combination of these two says that any valid user NOT in auth.conf should be redirected to their own /CUSTOMER endpoint, so each customer user lands there, and can't get anywhere else.

Works well, no additional software is required over vanilla apache and openssh, and it still feels relatively simple, while meeting our security requirements.

View Comments

ThinkPad X250 on CentOS 7

Tue 06 Oct 2015

Tags: , , ,

Wow, almost a year since the last post. Definitely time to reboot the blog.

Got to replace my aging ThinkPad X201 with a lovely shiny new ThinkPad X250 over the weekend. Specs are:

  • CPU: Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
  • RAM: 16GB PC3-12800 DDR3L SDRAM 1600MHz SODIMM
  • Disk: 256GB SSD (swapped out for existing Samsung SSD)
  • Display: 12.5" 1920x1080 IPS display, 400nit, non-touch
  • Graphics: Intel Graphics 5500
  • Wireless: Intel 7265 AC/B/G/N Dual Band Wireless and Bluetooth 4.0
  • Batteries: 1 3-cell internal, 1 6-cell hot-swappable

A very nice piece of kit!

Just wanted to document what works and what doesn't (so far) on my standard OS, CentOS 7, with RH kernel 3.10.0-229.11.1. I had to install the following additional packages:

  • iwl7265-firmware (for wireless support)
  • acpid (for the media buttons)

Working so far:

  • media buttons (Fn + F1/mute, F2/softer, F3/louder - see below for configuration)
  • wifi button (Fn + F8 - worked out of the box)
  • keyboard backlight (Fn + space, out of the box)
  • sleep/resume (out of the box)
  • touchpad hard buttons (see below)
  • touchpad soft buttons (out of the box)

Not working / unconfigured so far:

  • brightness buttons (Fn + F5/F6)
  • fingerprint reader (supposedly works with fprintd)

Not working / no ACPI codes:

  • mute microphone button (Fn + F4)
  • application buttons (Fn + F9-F12)

Uncertain/not tested yet:

  • switch video mode (Fn + F7)

To get the touchpad working I needed to use the "evdev" driver rather than the "Synaptics" one - I added the following as /etc/X11/xorg.conf.d/90-evdev.conf:

Section "InputClass"
  Identifier "Touchpad/TrackPoint"
  MatchProduct "PS/2 Synaptics TouchPad"
  MatchDriver "evdev"
  Option "EmulateWheel" "1"
  Option "EmulateWheelButton" "2"
  Option "Emulate3Buttons" "0"
  Option "XAxisMapping" "6 7"
  Option "YAxisMapping" "4 5"
EndSection

This gives me 3 working hard buttons above the touchpad, including middle-mouse- button for paste.

To get fonts scaling properly I needed to add a monitor section as /etc/X11/xorg.conf.d/50-monitor.conf, specifically for the DisplaySize:

Section "Monitor"
  Identifier    "Monitor0"
  VendorName    "Lenovo ThinkPad"
  ModelName     "X250"
  DisplaySize   276 155
  Option        "DPMS"
EndSection

and also set the dpi properly in my ~/.Xdefaults:

*.dpi: 177

This fixes font size nicely in Firefox/Chrome and terminals for me.

I also found my mouse movement was too slow, which I fixed with:

xinput set-prop 11 "Device Accel Constant Deceleration" 0.7

(which I put in my old-school ~/.xsession file).

Finally, getting the media keys involved installing acpid and setting up the appropriate magic in 3 files in /etc/acpid/events:

# /etc/acpi/events/volumedown
event=button/volumedown
action=/etc/acpi/actions/volume.sh down

# /etc/acpi/events/volumeup
event=button/volumeup
action=/etc/acpi/actions/volume.sh up

# /etc/acpi/events/volumemute
event=button/mute
action=/etc/acpi/actions/volume.sh mute

Those files capture the ACPI events and handle them via a custom script in /etc/acpi/actions/volume.sh, which uses amixer from alsa-utils. Volume control worked just fine, but muting was a real pain to get working correctly due to what seems like a bug in amixer - amixer -c1 sset Master playback toggle doesn't toggle correctly - it mutes fine, but then doesn't unmute all the channels it mutes!

I worked around it by figuring out the specific channels that sset Master was muting, and then handling them individually, but it's definitely not as clean:

#!/bin/sh
#
# /etc/acpi/actions/volume.sh (must be executable)
#

PATH=/usr/bin

die() {
  echo $*
  exit 1
}
usage() {
  die "usage: $(basename $0) up|down|mute"
}

test -n "$1" || usage

ACTION=$1
shift

case $ACTION in
  up)
    amixer -q -c1 -M sset Master 5%+ unmute
    ;;
  down)
    amixer -q -c1 -M sset Master 5%- unmute
    ;;
  mute)
    # Ideally the next command should work, but it doesn't unmute correctly
#   amixer -q -c1 sset Master playback toggle

    # Manual version for ThinkPad X250 channels
    # If adapting for another machine, 'amixer -C$DEV contents' is your friend (NOT 'scontents'!)
    SOUND_IS_OFF=$(amixer -c1 cget iface=MIXER,name='Master Playback Switch' | grep 'values=off')
    if [ -n "$SOUND_IS_OFF" ]; then
      amixer -q -c1 cset iface=MIXER,name='Master Playback Switch'    on
      amixer -q -c1 cset iface=MIXER,name='Headphone Playback Switch' on
      amixer -q -c1 cset iface=MIXER,name='Speaker Playback Switch'   on
    else
      amixer -q -c1 cset iface=MIXER,name='Master Playback Switch'    off
      amixer -q -c1 cset iface=MIXER,name='Headphone Playback Switch' off
      amixer -q -c1 cset iface=MIXER,name='Speaker Playback Switch'   off
    fi
    ;;
  *)
    usage
    ;;
esac

So in short, really pleased with the X250 so far - the screen is lovely, battery life seems great, I'm enjoying the keyboard, and most things have Just Worked or have been pretty easily configurable with CentOS. Happy camper!

References:

View Comments

Fujitsu ScanSnap 1300i on RHEL/CentOS

Sun 21 Sep 2014

Tags: , , , , ,

Just picked up a shiny new Fujitsu ScanSnap 1300i ADF scanner to get more serious about less paper.

Fujitsu ScanSnap 1300i

I chose the 1300i on the basis of the nice small form factor, and that SANE reports it having 'good' support with current SANE backends. I'd also been able to find success stories of other linux users getting the similar S1300 working okay:

Here's my experience getting the S1300i up and running on CentOS 6.

I had the following packages already installed on my CentOS 6 workstation, so I didn't need to install any new software:

  • sane-backends
  • sane-backends-libs
  • sane-frontends
  • xsane
  • gscan2pdf (from rpmforge)
  • gocr (from rpmforge)
  • tesseract (from my repo)

I plugged the S1300i in (via the dual USB cables instead of the power supply - nice!), turned it on (by opening the top cover) and then ran sudo sane-find-scanner. All good:

found USB scanner (vendor=0x04c5 [FUJITSU], product=0x128d [ScanSnap S1300i]) at libusb:001:013
# Your USB scanner was (probably) detected. It may or may not be supported by
# SANE. Try scanimage -L and read the backend's manpage.

Ran sudo scanimage -L - no scanner found.

I downloaded the S1300 firmware Luuk had provided in his post and installed it into /usr/share/sane/epjitsu, and then updated /etc/sane.d/epjitsu.conf to reference it:

# Fujitsu S1300i
firmware /usr/share/sane/epjitsu/1300_0C26.nal
usb 0x04c5 0x128d

Ran sudo scanimage -L - still no scanner found. Hmmm.

Rebooted into windows, downloaded the Fujitsu ScanSnap Manager package and installed it. Grubbed around in C:/Windows and found the following 4 firmware packages:

Copied the firmware onto another box, and rebooted back into linux. Copied the 4 new firmware files into /usr/share/sane/epjitsu and updated /etc/sane.d/epjitsu.conf to try the 1300i firmware:

# Fujitsu S1300i
firmware /usr/share/sane/epjitsu/1300i_0D12.nal
usb 0x04c5 0x128d

Close and re-open the S1300i (i.e. restart, just to be sure), and retried sudo scanimage -L. And lo, this time the scanner whirrs briefly and ... victory!

$ sudo scanimage -L
device 'epjitsu:libusb:001:014' is a FUJITSU ScanSnap S1300i scanner

I start gscan2pdf to try some scanning goodness. Eeerk: "No devices found". Hmmm. How about sudo gscan2pdf? Ahah, success - "FUJITSU ScanSnap S1300i" shows up in the Device dropdown.

I exit, and google how to deal with the permissions problem. Looks like the usb device gets created by udev as root:root 0664, and I need 'rw' permissions for scanning:

$ ls -l /dev/bus/usb/001/014
crw-rw-r--. 1 root root 189, 13 Sep 20 20:50 /dev/bus/usb/001/014

The fix is to add a scanner group and local udev rule to use that group when creating the device path:

# Add a scanner group (analagous to the existing lp, cdrom, tape, dialout groups)
$ sudo groupadd -r scanner
# Add myself to the scanner group
$ sudo useradd -aG scanner gavin
# Add a udev local rule for the S1300i
$ sudo vim /etc/udev/rules.d/99-local.rules
# Added:
# Fujitsu ScanSnap S1300i
ATTRS{idVendor}=="04c5", ATTRS{idProduct}=="128d", MODE="0664", GROUP="scanner", ENV{libsane_matched}="yes"

Then logout and log back in to pickup the change in groups, and close and re-open the S1300i. If all is well, I'm now in the scanner group and can control the scanner sans sudo:

# Check I'm in the scanner group now
$ id
uid=900(gavin) gid=100(users) groups=100(users),10(wheel),487(scanner)
# Check I can scanimage without sudo
$ scanimage -L
device 'epjitsu:libusb:001:016' is a FUJITSU ScanSnap S1300i scanner
# Confirm the permissions on the udev path (adjusted to match the new libusb path)
$ ls -l /dev/bus/usb/001/016
crw-rw-r--. 1 root scanner 189, 15 Sep 20 21:30 /dev/bus/usb/001/016
# Success!

Try gscan2pdf again, and this time it works fine without sudo!

And so far gscan2pdf 1.2.5 seems to work pretty nicely. It handles both simplex and duplex scans, and both the cleanup phase (using unpaper) and the OCR phase (with either gocr or tesseract) work without problems. tesseract seems to perform markedly better than gocr so far, as seems pretty typical.

So thus far I'm a pretty happy purchaser. On to a paperless searchable future!

View Comments

Yum Error Performing Checksum

Tue 08 May 2012

Tags: , ,

Ok, this has bitten me enough times now that I'm going to blog it so I don't forget it again.

Symptom: you're doing a yum update on a centos5 or rhel5 box, using rpms from a repository on a centos6 or rhel6 server (or anywhere else with a more modern createrepo available), and you get errors like this:

http://example.com/repodata/filelists.sqlite.bz2: [Errno -3] Error performing checksum
http://example.com/repodata/primary.sqlite.bz2: [Errno -3] Error performing checksum

What this really means that yum is too stupid to calculate the sha256 checksum correctly (and also too stupid to give you a sensible error message like "Sorry, primary.sqlite.bz2 is using a sha256 checksum, but I don't know how to calculate that").

The fix is simple:

yum install python-hashlib

from either rpmforge or epel, which makes the necessary libraries available for yum to calculate the new checksums correctly. Sorted.

View Comments

AoE on RHEL/CentOS

Fri 13 Jan 2012

Tags: , , ,

Updated 2012-07-24: packages updated to aoe-80 and aoetools-34, respectively.


I'm a big fan of Coraid and their relatively low-cost storage units. I've been using them for 5+ years now, and they've always been pretty well engineered, reliable, and performant.

They talk ATA-over-Ethernet (AoE), which is a very simple non-routable protocol for transmitting ATA commands directly via Ethernet frames, without the overhead of higher level layers like IP and TCP. So they're a lighter protocol than something like iSCSI, and so theoretically higher performance.

One issue with them on linux is that the in-kernel 'aoe' driver is typically pretty old. Coraid's latest aoe driver is version 78, for instance, while the RHEL6 kernel (2.6.32) comes with aoe v47, and the RHEL5 kernel (2.6.18) comes with aoe v22. So updating to the latest version is highly recommended, but also a bit of a pain, because if you do it manually it has to be recompiled for each new kernel update.

The modern way to handle this is to use a kernel-ABI tracking kmod, which gives you a driver that will work across multiple kernel updates for a given EL generation, without having to recompile each time.

So I've created a kmod-aoe package that seems to work nicely here. It's downloadable below, or you can install it from my yum repository. The kmod depends on the 'aoetools' package, which supplies the command line utilities for managing your AoE devices.

kmod-aoe (v80):

aoetools (v34):

There's an init script in the aoetools package that loads the kernel module, activates any configured LVM volume groups, and mounts any filesystems. All configuration is done via /etc/sysconfig/aoe.

View Comments

OpenLDAP Tips and Tricks

Fri 28 Oct 2011

Tags: , , , , ,

Having spent too much of this week debugging problems around migrating ldap servers from RHEL5 to RHEL6, here are some miscellaneous notes to self:

  1. The service is named ldap on RHEL5, and slapd on RHEL6 e.g. you do service ldap start on RHEL5, but service slapd start on RHEL6

  2. On RHEL6, you want all of the following packages installed on your clients:

    yum install openldap-clients pam_ldap nss-pam-ldapd

  3. This seems to be the magic incantation that works for me (with real SSL certificates, though):

    authconfig --enableldap --enableldapauth \
  --ldapserver ldap.example.com \
  --ldapbasedn="dc=example,dc=com" \
  --update

  4. Be aware that there are multiple ldap configuration files involved now. All of the following end up with ldap config entries in them and need to be checked:

    • /etc/openldap/ldap.conf
    • /etc/pam_ldap.conf
    • /etc/nslcd.conf
    • /etc/sssd/sssd.conf

    Note too that /etc/openldap/ldap.conf uses uppercased directives (e.g. URI) that get lowercased in the other files (URI -> uri). Additionally, some directives are confusingly renamed as well - e.g. TLA_CACERT in /etc/openldap/ldap.conf becomes tla_cacertfile in most of the others. :-(

  5. If you want to do SSL or TLS, you should know that the default behaviour is for ldap clients to verify certificates, and give misleading bind errors if they can't validate them. This means:

    • if you're using self-signed certificates, add TLS_REQCERT allow to /etc/openldap/ldap.conf on your clients, which means allow certificates the clients can't validate

    • if you're using CA-signed certificates, and want to verify them, add your CA PEM certificate to a directory of your choice (e.g. /etc/openldap/certs, or /etc/pki/tls/certs, for instance), and point to it using TLA_CACERT in /etc/openldap/ldap.conf, and tla_cacertfile in /etc/ldap.conf.

  6. RHEL6 uses a new-fangled /etc/openldap/slapd.d directory for the old /etc/openldap/slapd.conf config data, and the RHEL6 Migration Guide tells you to how to convert from one to the other. But if you simply rename the default slapd.d directory, slapd will use the old-style slapd.conf file quite happily, which is much easier to read/modify/debug, at least while you're getting things working.

  7. If you run into problems on the server, there are lots of helpful utilities included with the openldap-servers package. Check out the manpages for slaptest(8), slapcat(8), slapacl(8), slapadd(8), etc.

Further reading:

View Comments

rpm-find-changes

Tue 18 Oct 2011

Tags: , , ,

rpm-find-changes is a little script I wrote a while ago for rpm-based systems (RedHat, CentOS, Mandriva, etc.). It finds files in a filesystem tree that are not owned by any rpm package (orphans), or are modified from the version distributed with their rpm. In other words, any file that has been introduced or changed from it's distributed version.

It's intended to help identify candidates for backup, or just for tracking interesting changes. I run it nightly on /etc on most of my machines, producing a list of files that I copy off the machine (using another tool, which I'll blog about later) and store in a git repository.

I've also used it for tracking changes to critical configuration trees across multiple machines, to make sure everything is kept in sync, and to be able to track changes over time.

Available on github: https://github.com/gavincarr/rpm-find-changes

View Comments

Google Hangout on CentOS 6

Mon 08 Aug 2011

Tags: , ,

Kudos to Google for providing linux plugins for their Google Plus+ Hangouts (a multi-way video chat system), for both debian-based and rpm-based systems. The library requirements don't seem to be documented anywhere though, so here's the magic incantation required for installation on CentOS6 x86_64:

yum install libstdc++.i686 gtk2.i686 \
  libXrandr.i686 libXcomposite.i686 libXfixes.i686 \
  pulseaudio-libs.i686 alsa-lib.i686
View Comments

Exploring Riak

Tue 16 Nov 2010

Tags: , , , ,

Been playing with Riak recently, which is one of the modern dynamo-derived nosql databases (the other main ones being Cassandra and Voldemort). We're evaluating it for use as a really large brackup datastore, the primary attraction being the near linear scalability available by adding (relatively cheap) new nodes to the cluster, and decent availability options in the face of node failures.

I've built riak packages for RHEL/CentOS 5, available at my repository, and added support for a riak 'target' to the latest version (1.10) of brackup (packages also available at my repo).

The first thing to figure out is the maximum number of nodes you expect your riak cluster to get to. This you use to size the ring_creation_size setting, which is the number of partitions the hash space is divided into. It must be a power of 2 (64, 128, 256, etc.), and the reason it's important is that it cannot be easily changed after the cluster has been created. The rule of thumb is that for performance you want at least 10 partitions per node/machine, so the default ring_creation_size of 64 is really only useful up to about 6 nodes. 128 scales to 10-12, 256 to 20-25, etc. For more info see the Riak Wiki.

Here's the script I use for configuring a new node on CentOS. The main things to tweak here are the ring_creation_size you want (here I'm using 512, for a biggish cluster), and the interface to use to get the default ip address (here eth0, or you could just hardcode 0.0.0.0 instead of $ip).

#!/bin/sh
# Riak configuration script for CentOS/RHEL

# Install riak (and IO::Interface, for next)
yum -y install riak perl-IO-Interface

# To set app.config:web_ip to use primary ip, do:
perl -MIO::Interface::Simple -i \
  -pe "BEGIN { \$ip = IO::Interface::Simple->new(q/eth0/)->address; }
      s/127\.0\.0\.1/\$ip/" /etc/riak/app.config

# To add a ring_creation_size clause to app.config, do:
perl -i \
  -pe 's/^((\s*)%% riak_web_ip)/$2%% ring_creation_size is the no. of partitions to divide the hash
$2%% space into (default: 64).
$2\{ring_creation_size, 512\},

$1/' /etc/riak/app.config

# To set riak vm_args:name to hostname do:
perl -MSys::Hostname -i -pe 's/127\.0\.0\.1/hostname/e' /etc/riak/vm.args

# display (bits of) config files for checking
echo
echo '********************'
echo /etc/riak/app.config
echo '********************'
head -n30 /etc/riak/app.config
echo
echo '********************'
echo /etc/riak/vm.args
echo '********************'
cat /etc/riak/vm.args

Save this to a file called e.g. riak_configure, and then to configure a couple of nodes you do the following (note that NODE is any old internal hostname you use to ssh to the host in question, but FIRST_NODE needs to use the actual -name parameter defined in /etc/riak/vm.args on your first node):

# First node
NODE=node1
cat riak_configure | ssh $NODE sh
ssh $NODE 'chkconfig riak on; service riak start'
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready

# All nodes after the first
FIRST_NODE=riak@node1.example.com
NODE=node2
cat riak_configure | ssh $NODE sh
ssh $NODE "chkconfig riak on; service riak start && riak-admin join $FIRST_NODE"
# Run the following until ringready reports TRUE
ssh $NODE riak-admin ringready

That's it. You should now have a working riak cluster accessible on port 8098 on your cluster nodes.

View Comments

Remote Rebuild, CentOS-style

Fri 08 Oct 2010

Tags: , ,

Problem: you've got a remote server that's significantly hosed, either through a screwup somewhere or a power outage that did nasty things to your root filesystem or something. You have no available remote hands, and/or no boot media anyway.

Preconditions: You have another server you can access on the same network segment, and remote access to the broken server, either through a DRAC or iLO type card, or through some kind of serial console server (like a Cyclades/Avocent box).

Solution: in extremis, you can do a remote rebuild. Here's the simplest recipe I've come up with. I'm rebuilding using centos5-x86_64 version 5.5; adjust as necessary.

Note: dnsmasq, mrepo and syslinux are not core CentOS packages, so you need to enable the rpmforge repository to follow this recipe. This just involves:

wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm
rpm -Uvh rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm

1. On your working box (which you're now going to press into service as a build server), install and configure dnsmasq to provide dhcp and tftp services:

# Install dnsmasq
yum install dnsmasq

# Add the following lines to the bottom of your /etc/dnsmasq.conf file
# Note that we don't use the following ip address, but the directive
# itself is required for dnsmasq to turn dhcp functionality on
dhcp-range=ignore,192.168.1.99,192.168.1.99
# Here use the broken server's mac addr, hostname, and ip address
dhcp-host=00:d0:68:09:19:80,broken.example.com,192.168.1.5,net:centos5x
# Point the centos5x tag at the tftpboot environment you're going to setup
dhcp-boot=net:centos5x,/centos5x-x86_64/pxelinux.0
# And enable tftp
enable-tftp
tftp-root = /tftpboot
#log-dhcp

# Then start up dnsmasq
service dnsmasq start

2. Install and configure mrepo to provide your CentOS build environment:

# Install mrepo and syslinux
yum install mrepo syslinux

# Setup a minimal /etc/mrepo.conf e.g.
cat > /etc/mrepo.conf
[main]
srcdir = /var/mrepo
wwwdir = /var/www/mrepo
confdir = /etc/mrepo.conf.d
arch = x86_64
mailto = root@example.com
smtp-server = localhost
pxelinux = /usr/lib/syslinux/pxelinux.0
tftpdir = /tftpboot

[centos5]
release = 5
arch = x86_64
metadata = repomd repoview
name = Centos-$release $arch
#iso = CentOS-$release.5-$arch-bin-DVD-?of2.iso
#iso = CentOS-$release.5-$arch-bin-?of8.iso
^D
# (uncomment one of the iso lines above, either the DVD or the CD one)

# Download the set of DVD or CD ISOs for the CentOS version you want
# There are fewer DVD ISOs, but you need to use bittorrent to download
mkdir -p /var/mrepo/iso
cd /var/mrepo/iso
elinks http://isoredirect.centos.org/centos/5.5/isos/x86_64/

# Once your ISOs are available in /var/mrepo/iso, and the 'iso' line
# in /etc/mrepo.conf updated appropriately, run mrepo itself
mrepo -gvv

3. Finally, finish setting up your tftp environment. mrepo should have copied appropriate pxelinux.0, initrd.img, and vmlinuz files into your /tftpboot/centos5-x86_64 directory, so all you need to supply is an appropriate grub boot config:

cd /tftpboot/centos5-x86_64
ls
mkdir -p pxelinux.cfg

# Setup a default grub config (adjust the serial/console and repo params as needed)
cat > pxelinux.cfg/default
default linux
serial 0,9600n8
label linux
  root (nd)
  kernel vmlinuz
  append initrd=initrd.img console=ttyS0,9600 repo=http://192.168.1.1/mrepo/centos5-x86_64
^D

Now get your server to do a PXE boot (via a boot option or the bios or whatever), and hopefully your broken server will find your dhcp/tftp environment and boot up in install mode, and away you go.

If you have problems with the boot, try checking your /var/log/messages file on the boot server for hints.

View Comments

Dell OMSA

Mon 22 Mar 2010

Tags: , , , , ,

Following on from my IPMI explorations, here's the next chapter in my getting-down-and-dirty-with-dell-hardware-on-linux adventures. This time I'm setting up Dell's OpenManage Server Administrator software, primarily in order to explore being able to configure bios settings from within the OS. As before, I'm running CentOS 5, but OMSA supports any of RHEL4, RHEL5, SLES9, and SLES10, and various versions of Fedora Core and OpenSUSE.

Here's what I did to get up and running:

# Configure the Dell OMSA repository
wget -O bootstrap.sh http://linux.dell.com/repo/hardware/latest/bootstrap.cgi
# Review the script to make sure you trust it, and then run it
sh bootstrap.sh
# OR, for CentOS5/RHEL5 x86_64 you can just install the following:
rpm -Uvh http://linux.dell.com/repo/hardware/latest/platform_independent/rh50_64/prereq/\
dell-omsa-repository-2-5.noarch.rpm

# Install base version of OMSA, without gui (install srvadmin-all for more)
yum install srvadmin-base

# One of daemons requires /usr/bin/lockfile, so make sure you've got procmail installed
yum install procmail

# If you're running an x86_64 OS, there are a couple of additional 32-bit
#   libraries you need that aren't dependencies in the RPMs
yum install compat-libstdc++-33-3.2.3-61.i386 pam.i386

# Start OMSA daemons
for i in instsvcdrv dataeng dsm_om_shrsvc; do service $i start; done

# Finally, you can update your path by doing logout/login, or just run:
. /etc/profile.d/srvadmin-path.sh

Now to check whether you're actually functional you can try a few of the following (as root):

omconfig about
omreport about
omreport system -?
omreport chassis -?

omreport is the OMSA CLI reporting/query tool, and omconfig is the equivalent update tool. The main documentation for the current version of OMSA is here. I found the CLI User's Guide the most useful.

Here's a sampler of interesting things to try:

# Report system overview
omreport chassis

# Report system summary info (OS, CPUs, memory, PCIe slots, DRAC cards, NICs)
omreport system summary

# Report bios settings
omreport chassis biossetup

# Fan info
omreport chassis fans

# Temperature info
omreport chassis temps

# CPU info
omreport chassis processors

# Memory and memory slot info
omreport chassis memory

# Power supply info
omreport chassis pwrsupplies

# Detailed PCIe slot info
omreport chassis slots

# DRAC card info
omreport chassis remoteaccess

omconfig allows setting object attributes using a key=value syntax, which can get reasonably complex. See the CLI User's Guide above for details, but here are some examples of messing with various bios settings:

# See available attributes and settings
omconfig chassis biossetup -?

# Turn the AC Power Recovery setting to On
omconfig chassis biossetup attribute=acpwrrecovery setting=on

# Change the serial communications setting (on with serial redirection via)
omconfig chassis biossetup attribute=serialcom setting=com1
omconfig chassis biossetup attribute=serialcom setting=com2

# Change the external serial connector
omconfig chassis biossetup attribute=extserial setting=com1
omconfig chassis biossetup attribute=extserial setting=rad

# Change the Console Redirect After Boot (crab) setting
omconfig chassis biossetup attribute=crab setting=enabled
omconfig chassis biossetup attribute=crab setting=disabled

# Change NIC settings (turn on PXE on NIC1)
omconfig chassis biossetup attribute=nic1 setting=enabledwithpxe

Finally, there are some interesting formatting options available to both omreport, for use in scripting e.g.

# Custom delimiter format (default semicolon)
omreport chassis -fmt cdv

# XML format
omreport chassis -fmt xml

# To change the default cdv delimiter
omconfig preferences cdvformat -?
omconfig preferences cdvformat delimiter=pipe
View Comments

IPMI on CentOS/RHEL

Thu 11 Mar 2010

Tags: , , , ,

Spent a few days deep in the bowels of a couple of datacentres last week, and realised I didn't know enough about Dell's DRAC base management controllers to use them properly. In particular, I didn't know how to mess with the drac settings from within the OS. So spent some of today researching that.

Turns out there are a couple of routes to do this. You can use the Dell native tools (e.g. racadm) included in Dell's OMSA product, or you can use vendor-neutral IPMI, which is well-supported by Dell DRACs. I went with the latter as it's more cross-platform, and the tools come native with CentOS, instead of having to setup Dell's OMSA repositories. The Dell-native tools may give you more functionality, but for what I wanted to do IPMI seems to work just fine.

So installation is just:

yum install OpenIPMI OpenIPMI-tools
chkconfig ipmi on
service ipmi start

and then from the local machine you can use ipmitool to access and manipulate all kinds of useful stuff:

# IPMI commands
ipmitool help
man ipmitool

# To check firmware version
ipmitool mc info
# To reset the management controller
ipmitool mc reset [ warm | cold ]

# Show field-replaceable-unit details
ipmitool fru print

# Show sensor output
ipmitool sdr list
ipmitool sdr type list
ipmitool sdr type Temperature
ipmitool sdr type Fan
ipmitool sdr type 'Power Supply'

# Chassis commands
ipmitool chassis status
ipmitool chassis identify [<interval>]   # turn on front panel identify light (default 15s)
ipmitool [chassis] power soft            # initiate a soft-shutdown via acpi
ipmitool [chassis] power cycle           # issue a hard power off, wait 1s, power on
ipmitool [chassis] power off             # issue a hard power off
ipmitool [chassis] power on              # issue a hard power on
ipmitool [chassis] power reset           # issue a hard reset

# Modify boot device for next reboot
ipmitool chassis bootdev pxe
ipmitool chassis bootdev cdrom
ipmitool chassis bootdev bios

# Logging
ipmitool sel info
ipmitool sel list
ipmitool sel elist                       # extended list (see manpage)
ipmitool sel clear

For remote access, you need to setup user and network settings, either at boot time on the DRAC card itself, or from the OS via ipmitool:

# Display/reset password for default root user (userid '2')
ipmitool user list 1
ipmitool user set password 2 <new_password>

# Display/configure lan settings
ipmitool lan print 1
ipmitool lan set 1 ipsrc [ static | dhcp ]
ipmitool lan set 1 ipaddr 192.168.1.101
ipmitool lan set 1 netmask 255.255.255.0
ipmitool lan set 1 defgw ipaddr 192.168.1.254

Once this is configured you should be able to connect using the 'lan' interface to ipmitool, like this:

ipmitool -I lan -U root -H 192.168.1.101 chassis status

which will prompt you for your ipmi root password, or you can do the following:

echo <new_password> > ~/.racpasswd
chmod 600 ~/.racpasswd

and then use that password file instead of manually entering it each time:

ipmitool -I lan -U root -f ~/.racpasswd -H 192.168.1.101 chassis status

I'm using an 'ipmi' alias that looks like this:

alias ipmi='ipmitool -I lan -U root -f ~/.racpasswd -H'

# which then allows you to do the much shorter:
ipmi 192.168.1.101 chassis status
# OR
ipmi <hostname> chassis status

Finally, if you configure serial console redirection in the bios as follows:

Serial Communication -> Serial Communication:       On with Console Redirection via COM2
Serial Communication -> External Serial Connector:  COM2
Serial Communication -> Redirection After Boot:     Disabled

then you can setup standard serial access in grub.conf and inittab on com2/ttyS1 and get serial console access via IPMI serial-over-lan using the 'lanplus' interface:

ipmitool -I lanplus -U root -f ~/.racpasswd -H 192.168.1.101 sol activate

which I typically use via a shell function:

# ipmi serial-over-lan function
isol() {
   if [ -n "$1" ]; then
       ipmitool -I lanplus -U root -f ~/.racpasswd -H $1 sol activate
   else
       echo "usage: sol <sol_ip>"
   fi
}

# used like:
isol 192.168.1.101
isol <hostname>

Further reading:

View Comments

Mocking RPMs on CentOS

Fri 26 Feb 2010

Tags: , ,

Mock is a Fedora project that allows you to build RPM packages within a chroot environment, allowing you to build packages for other systems than the one you're running on (e.g. building CentOS 4 32-bit RPMs on a CentOS 5 64-bit host), and ensuring that all the required build dependencies are specified correctly in the RPM spec file.

It's also pretty under-documented, so these are my notes on things I've figured out over the last week setting up a decent mock environment on CentOS 5.

First, I'm using mock 1.0.2 from the EPEL repository, rather than older 0.6.13 available from CentOS Extras. There are apparently backward-compatibility problems with versions of mock > 0.6, but as I'm mostly building C5 packages I decided to go with the newer version. So installation is just:

# Install mock and python-ctypes packages (the latter for better setarch support)
$ sudo yum --enablerepo=epel install mock python-ctypes

# Add yourself to the 'mock' group that will have now been created
$ sudo usermod -G mock gavin

The mock package creates an /etc/mock directory with configs for various OS versions (mostly Fedoras). The first thing you want to tweak there is the site-defaults.cfg file which sets up various defaults for all your builds. Mine now looks like this:

# /etc/mock/site-defaults.cfg

# Set this to true if you've installed python-ctypes
config_opts['internal_setarch'] = True

# Turn off ccache since it was causing errors I haven't bothered debugging
config_opts['plugin_conf']['ccache_enable'] = False

# (Optional) Fake the build hostname to report
config_opts['use_host_resolv'] = False
config_opts['files']['etc/hosts'] = """
127.0.0.1 buildbox.openfusion.com.au nox.openfusion.com.au localhost
"""
config_opts['files']['etc/resolv.conf'] = """
nameserver 127.0.0.1
"""

# Setup various rpm macros to use
config_opts['macros']['%packager'] = 'Gavin Carr <gavin@openfusion.com.au>'
config_opts['macros']['%debug_package'] = '%{nil}'

You can use the epel-5-{i386,x86_64}.cfg configs as-is if you like; I copied them to centos-5-{i386,x86_64}.cfg versions and removed the epel 'extras', 'testing', and 'local' repositories from the yum.conf section, since I typically want to build using only 'core' and 'update' packages.

You can then run a test by doing:

# e.g. initialise a centos-5-i386 chroot environment
$ CONFIG=centos-5-i386
$ mock -r $CONFIG --init

which will setup an initial chroot environment using the given config. If that seemed to work (you weren't inundated with error messages), you can try a build:

# Rebuild the given source RPM within the chroot environment
# usage: mock -r <mock_config> --rebuild /path/to/SRPM e.g.
$ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/clix-0.3.4-1.of.src.rpm

If the build succeeds, it drops your packages into the /var/lib/mock/$CONFIG/result directory:

$ ls -1 /var/lib/mock/$CONFIG/result
build.log
clix-0.3.4-1.of.noarch.rpm
clix-0.3.4-1.of.src.rpm
root.log
state.log

If it fails, you can check mock output, the *.log files above for more info, and/or rerun mock with the -v flag for more verbose messaging.

A couple of final notes:

  • the chroot environments are cached, but rebuilding them and checking for updates can be pretty network intensive, so you might want to consider setting up a local repository to pull from. mrepo (available from rpmforge) is pretty good for that.

  • there don't seem to be any hooks in mock to allow you to sign packages you've built, so if you do want signed packages you need to sign them afterwards via a rpm --resign $RPMS.

View Comments

Skype 2.1 on CentOS 5

Sat 19 Sep 2009

Tags: , , ,

The new skype 2.1 beta (woohoo - Linux users are now only 2.0 versions behind Windows, way to go Skype!) doesn't come with a CentOS rpm, unlike earlier versions. And the Fedora packages that are available are for FC9 and FC10, which are too recent to work on a stock RHEL/CentOS 5 system.

So here's how I got skype working nicely on CentOS 5.3, using the static binary tarball.

Note that while it appears skype has finally been ported to 64-bit architectures, the only current 64-bit builds are for Ubuntu 8.10+, so installing on a 64-bit CentOS box requires 32-bit libraries to be installed (sigh). Otherwise you get the error: skype: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory.

# the available generic skype binaries are 32-bit, so if you're running a 64-bit
# system you need to make sure you have various 32-bit libraries installed
yum install glib2.i386 qt4.i386 zlib.i386 alsa-lib.i386 libX11.i386 \
  libXv.i386 libXScrnSaver.i386

# installing to /opt (tweak to taste)
cd /tmp
wget http://www.skype.com/go/getskype-linux-beta-static
cd /opt
tar jxvf /tmp/skype_static-2.1.0.47.tar.bz2
ln -s skype_static-2.1.0.47 skype

# Setup some symlinks (the first is required for sounds to work, the second is optional)
ln -s /opt/skype /usr/share/skype
ln -s /opt/skype/skype /usr/bin/skype

You don't seem to need pulseaudio installed (at least with the static binary - I assume it's linked in statically already).

Tangentially, if you have any video problems with your webcam, you might want to check out the updated video drivers available in the kmod-video4linux package from the shiny new ELRepo.org. I'm using their updated uvcvideo module with a Logitech QuickCam Pro 9000 and Genius Slim 1322AF, and both are working well.

View Comments

Yum Download SRPMs

Sat 19 Sep 2009

Tags: , , ,

Found a nice post today on how to use yum to download source RPMs, rather than having to do a manual search on the relevant mirror.

View Comments

Open Fusion RPM Repository

Thu 05 Feb 2009

Tags: , ,

Updated 2014-09-26 for CentOS 7.

Over the last few years I've built up quite a collection of packages for CentOS, and distribute them via a yum repository. They're typically packages that aren't included in DAG/RPMForge when I need them, so I just build them myself. In case they're useful to other people, this post documents the repository locations, and how you can get setup to make use of it yourself.

Obligatory Warning: this is a personal repository, so it's primarily for packages I want to use myself on a particular platform i.e. coverage is uneven, and packages won't be as well tested as a large repository like RPMForge. Also, I routinely build packages that replace core packages, so you'll want the repo disabled by default if that concerns you. Use at your own risk, packages may nuke your system and cause global warming, etc. etc.

Locations:

To add the Open Fusion repository to your yum configuration, just install the following 'openfusion-release' package:

# CentOS 5:
sudo rpm -Uvh http://repo.openfusion.net/centos5-x86_64/openfusion-release-0.7-1.of.el5.noarch.rpm
# CentOS 6:
sudo rpm -Uvh http://repo.openfusion.net/centos6-x86_64/openfusion-release-0.7-1.of.el6.noarch.rpm
# CentOS 7:
sudo rpm -Uvh http://repo.openfusion.net/centos7-x86_64/openfusion-release-0.7-1.of.el7.noarch.rpm

And here are the openfusion-release packages as links:

Feedback and suggestions are welcome. Packaging requests are also welcome, particularly when they involve my wishlist. ;-)

Enjoy.

View Comments

Basic KVM on CentOS 5

Fri 24 Oct 2008

Tags: , , ,

I've been using kvm for my virtualisation needs lately, instead of xen, and finding it great. Disadvantages are that it requires hardware virtualisation support, and so only works on newer Intel/AMD CPUs. Advantages are that it's baked into recent linux kernels, and so more or less Just Works out of the box, no magic kernels required.

There are some pretty useful resources covering this stuff out on the web - the following sites are particularly useful:

There's not too much specific to CentOS though, so here's the recipe I've been using for CentOS 5:

# Confirm your CPU has virtualisation support
egrep 'vmx|svm' /proc/cpuinfo

# Install the kvm and qemu packages you need
# From the CentOS Extras repository (older):
yum install --enablerepo=extras kvm kmod-kvm qemu
# OR from my repository (for most recent kernels only):
ARCH=$(uname -i)
OF_MREPO=http://www.openfusion.com.au/mrepo/centos5-$ARCH/RPMS.of/
rpm -Uvh $OF_MREPO/openfusion-release-0.3-1.of.noarch.rpm
yum install kvm kmod-kvm qemu

# Install the appropriate kernel module - either:
modprobe kvm-intel
# OR:
modprobe kvm-amd
lsmod | grep kvm

# Check the kvm device exists
ls -l /dev/kvm

# I like to run my virtual machines as a 'kvm' user, not as root
chgrp kvm /dev/kvm
chmod 660 /dev/kvm
ls -l /dev/kvm
useradd -r -g kvm kvm

# Create a disk image to use
cd /data/images
IMAGE=centos5x.img
# Note that the specified size is a maximum - the image only uses what it needs
qemu-img create -f qcow2 $IMAGE 10G
chown kvm $IMAGE

# Boot an install ISO on your image and do the install
MEM=1024
ISO=/path/to/CentOS-5.2-x86_64-bin-DVD.iso
# ISO=/path/to/WinXP.iso
qemu-kvm -hda $IMAGE -m ${MEM:-512} -cdrom $ISO -boot d
# I usually just do a minimal install with std defaults and dhcp, and configure later

# After your install has completed restart without the -boot parameter
# This should have outgoing networking working, but pings don't work (!)
qemu-kvm -hda $IMAGE -m ${MEM:-512} &

That should be sufficient to get you up and running with basic outgoing networking (for instance as a test desktop instance). In qemu terms this is using 'user mode' networking which is easy but slow, so if you want better performance, or if you want to allow incoming connections (e.g. as a server) you need some extra magic, which I'll cover in a "subsequent post":kvm_bridging.

View Comments

Simple KVM Bridging

Fri 24 Oct 2008

Tags: , , ,

Following on from my post yesterday on "Basic KVM on CentOS 5", here's how to setup simple bridging to allow incoming network connections to your VM (and to get other standard network functionality like pings working). This is a simplified/tweaked version of Hadyn Solomon's bridging instructions.

Note this this is all done on your HOST machine, not your guest.

For CentOS:

# Install bridge-utils
yum install bridge-utils

# Add a bridge interface config file
vi /etc/sysconfig/network-scripts/ifcfg-br0
# DHCP version
ONBOOT=yes
TYPE=Bridge
DEVICE=br0
BOOTPROTO=dhcp
# OR, static version
ONBOOT=yes
TYPE=Bridge
DEVICE=br0
BOOTPROTO=static
IPADDR=xx.xx.xx.xx
NETMASK=255.255.255.0

# Make your primary interface part of this bridge e.g.
vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Add:
BRIDGE=br0
# Optional: comment out BOOTPROTO/IPADDR lines, since they're
# no longer being used (the br0 takes precedence)

# Add a script to connect your guest instance to the bridge on guest boot
vi /etc/qemu-ifup
#!/bin/bash
BRIDGE=$(/sbin/ip route list | awk '/^default / { print $NF }')
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif $BRIDGE $1
# END OF SCRIPT
# Silence a qemu warning by creating a noop qemu-ifdown script
vi /etc/qemu-ifdown
#!/bin/bash
# END OF SCRIPT
chmod +x /etc/qemu-if*

# Test - bridged networking uses a 'tap' networking device
NAME=c5-1
qemu-kvm -hda $NAME.img -name $NAME -m ${MEM:-512} -net nic -net tap &

Done. This should give you VMs that are full network members, able to be pinged and accessed just like a regular host. Bear in mind that this means you'll want to setup firewalls etc. if you're not in a controlled environment.

Notes:

  • If you want to run more than one VM on your LAN, you need to set the guest MAC address explicitly, since otherwise qemu uses a static default that will conflict with any other similar VM on the LAN. e.g. do something like:
# HOST_ID, identifying your host machine (2-digit hex)
HOST_ID=91
# INSTANCE, identifying the guest on this host (2-digit hex)
INSTANCE=01
# Startup, but with explicit macaddr
NAME=c5-1
qemu-kvm -hda $NAME.img -name $NAME -m ${MEM:-512} \
  -net nic,macaddr=00:16:3e:${HOST_ID}:${INSTANCE}:00 -net tap &
  • This doesn't use the paravirtual ('virtio') drivers that Hadyn mentions, as these aren't available until kernel 2.6.25, so they're not available to CentOS linux guests without a kernel upgrade.
View Comments

Simple dual upstream gateways in CentOS

Fri 23 May 2008

Tags: , ,


Update 2019-05-05: see this revised post for a simpler implementation method and a gotcha to watch out for. HT to Jim MacLeod for suggested improvements in his comments below.



Had to setup some simple policy-based routing on CentOS again recently, and had forgotten the exact steps. So here's the simplest recipe for CentOS that seems to work. This assumes you have two upstream gateways (gw1 and gw2), and that your default route is gw1, so all you're trying to do is have packets that come in on gw2 go back out gw2.

1) Define an extra routing table e.g.

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local tables
#
102     gw2
#

2) Add a default route via gw2 (here 172.16.2.254) to table gw2 on the appropriate interface (here eth1) e.g.

$ cat /etc/sysconfig/network-scripts/route-eth1
default table gw2 via 172.16.2.254

3) Add an ifup-local script to add a rule to use table gw2 for eth1 packets e.g.

$ cat /etc/sysconfig/network-scripts/ifup-local
#!/bin/bash
#
# Script to add/delete routing rules for gw2 devices
#

GW2_DEVICE=eth1
GW2_LOCAL_ADDR=172.16.2.1

if [ $(basename $0) = ifdown-local ]; then
  OP=del
else
  OP=add
fi

if [ "$1" = "$GW2_DEVICE" ]; then
  ip rule $OP from $GW2_LOCAL_ADDR table gw2
fi

4) Use the ifup-local script also as ifdown-local, to remove that rule

$ cd /etc/sysconfig/network-scripts
$ ln -s ifup-local ifdown-local

5) Restart networking, and you're done!

# service network restart

For more, see:

View Comments