Hackery

incron tips and traps

Fri 31 Aug 2018

(Updated April 2020: added new no. 7 after being newly bitten...)

incron is a useful little cron-like utility that lets you run arbitrary jobs (like cron), but instead of being triggered at certain times, your jobs are triggered by changes to files or directories.

It uses the linux kernel inotify facility (hence the name), and so it isn't cross-platform, but on linux it can be really useful for monitoring file changes or uploads, reporting or forwarding based on status files, simple synchronisation schemes, etc.

Again like cron, incron supports the notion of job 'tables' where commands are configured, and users can have manage their own tables using an incrontab command, while root can manage multiple system tables.

So it's a really useful linux utility, but it's also fairly old (the last release, v0.5.10, is from 2012), doesn't appear to be under active development any more, and it has a few frustrating quirks that can make using it unnecessarily difficult.

So this post is intended to highlight a few of the 'gotchas' I've experienced using incron:

You can't monitor recursively i.e. if you create a watch on a directory incron will only be triggered on events in that directory itself, not in any subdirectories below it. This isn't really an incron issue since it's a limitation of the underlying inotify mechanism, but it's definitely something you'll want to be aware of going in.
The incron interface is enough like cron (incrontab -l, incrontab -e, man 5 incrontab, etc.) that you might think that all your nice crontab features are available. Unfortunately that's not the case - most significantly, you can't have comments in incron tables (incron will try and parse your comment lines and fail), and you can't set environment variables to be available for your commands. (This includes PATH, so you might need to explicitly set a PATH inside your incron scripts if you need non-standard locations. The default PATH is documented as /usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin.)
That means that cron MAILTO support is not available, and in general there's no easy way of getting access to the stdout or stderr of your jobs. You can't even use shell redirects in your command to capture the output (e.g. echo $@/$# >> /tmp/incron.log doesn't work). If you're debugging, the best you can do is add a layer of indirection by using a wrapper script that does the redirection you need (e.g. echo $1 2&>1 >> /tmp/incron.log) and calling the wrapper script in your incrontab with the incron arguments (e.g. debug.sh $@/$#). This all makes debugging misbehaving commands pretty painful. The main place to check if your commands are running is the cron log (/var/log/cron) on RHEL/CentOS, and syslog (/var/log/syslog) on Ubuntu/Debian.
incron is also very picky about whitespace in your incrontab. If you put more than one space (or a tab) between the inotify masks and your command, you'll get an error in your cron log saying cannot exec process: No such file or directory, because incron will have included everything after the first space as part of your command e.g. (gavin) CMD ( echo /home/gavin/tmp/foo) (note the evil space before the echo).
It's often difficult (and non-intuitive) to figure out what inotify events you want to trigger on in your incrontab masks. For instance, does 'IN_CREATE' get fired when you replace an existing file with a new version? What events are fired when you do a mv or a cp? If you're wanting to trigger on an incoming remote file copy, should you use 'IN_CREATE' or 'IN_CLOSE_WRITE'? In general, you don't want to guess, you actually want to test and see what events actually get fired on the operations you're interested in. The easiest way to do this is use inotifywait from the inotify-tools package, and run it using inotifywait -m <dir>, which will report to you all the inotify events that get triggered on that directory (hit <Ctrl-C> to exit).
The "If you're wanting to trigger on an incoming remote file copy, should you use 'IN_CREATE' or 'IN_CLOSE_WRITE'?" above was a trick question - it turns out it depends how you're doing the copy! If you're just going a simple copy in-place (e.g. with scp), then (assuming you want the completed file) you're going to want to trigger on 'IN_CLOSE_WRITE', since that's signalling all writing is complete and the full file will be available. If you're using a vanilla rsync, though, that's not going to work, as rsync does a clever write-to-a-hidden-file trick, and then moves the hidden file to the destination name atomically. So in that case you're going to want to trigger on 'IN_MOVED_TO', which will give you the destination filename once the rsync is completed. So again, make sure you test thoroughly before you deploy.
Though cron works fine with symlinks to crontab files (in e.g. /etc/cron.d, incron doesn't support this in /etc/incron.d - symlinks just seem to be quietly ignored. (Maybe this is for security, but it's not documented, afaict.)

Have I missed any? Any other nasties bitten you using incron?