Blosxom4Nagios

Wed 19 Mar 2008

Tags: ,

A little whole ago at one of my client sites we decided that we wanted to monitor the bulk of our Nagios notifications via RSS rather than via email or jabber. The centralised river-of-news architecture of RSS is just a much better fit for high-volume notification flow than the individual silos and persistent messaging nature of email.

There are a few nice RSS solutions out there for Nagios, including the following:

We quite liked Altinity's RSS4NAGIOS because it was notifications (rather than alerts) based, and because it was pretty easy to just drop in and use. The only thing we didn't really like was that it was still relatively static - it provides nice per-user feeds, but you can't carve those feeds up or drill down to subsets of the data very easily. We wanted to be able to slice-and-dice things a bit more dynamically - be able to look at feeds for per-host, per-hostgroup, per-status, or date-filtered notifications, for example.

I'm also a blosxom developer, so I quickly realised that I could do everything I wanted pretty trivially using blosxom. All I needed was a script to capture and tag notifications as blosxom posts, and then I could have dynamic filtering, multi-dimensional tag-based feeds, etc., all pretty much for free.

So a couple of afternoons later, Blosxom4Nagios was up and running. It's basically a single-purpose blosxom install that you drop into a directory somewhere (/var/log/nagios/blosxom, by default), hook into apache for display, hook into nagios to accept notifications, and away you go.

Notifications are tagged by type (host/service), state (OK, WARNING, CRITICAL etc.), hostname, hostgroup, service name, service group, contact, and date, so you can filter notifications based on any of these, and produce feeds (both RSS2 and atom) on any of them as well e.g.

etc.

Screenshots:

Default View

Filtering by date

Filtering by host

Blosxom4Nagios is available here:

and is licensed under the same MIT Licence as blosxom itself.

Comments and feedback welcome.

View Comments

MySQL blocked hosts

Wed 31 Oct 2007

Tags: , ,

Here's an interesting one: one of my clients has been seeing mysql db connections from one of their app servers (and only one) being periodically locked out, with the following error message reported when attempting to connect:

Host _hostname_ is blocked because of many connection errors.
Unblock with 'mysqladmin flush-hosts'.

There's no indication in any of the database logs of anything untoward, or any connection errors at all, in fact. As a workaround, we've bumped up the max_connect_errors setting on the mysql instance, and haven't really had time to dig much further.

Till tonight, when I decided to figure out what was going on.

Turns out there's plenty of other people seeing this too, although MySQL seems to be in "it's not a bug, it's a feature" mode - see this bug report.

That thread helped clue me in, however. Turns out that mysql counts any connection to the database, even ones that don't attempt to make an actual database connection, as a connection error, but they only log ones that attempt to login. So there's a nice class of silent errors - and in fact, a nice DOS attack against MySQL - if you make standard TCP connections to mysql without logging in.

We, being clever and careful, were doing exactly that with nagios - making a simple TCP connection to port 3306 - in order to simply and cheaply check that mysql was listening on that port. Hmmmm.

Easy enough to remedy, of course, once you figure out what's going on. I even had a nice nagios plugin lying around to let me do more sophisticated database checks - check_db_query_rowcount - so just had to replace the simple check_tcp check with that, and all is right with the world.

But it's a plain and simple bug, and MySQL need to get it fixed. Personally I think a simple tcp connection should not count as a connection error at all without a login attempt (assuming it's not left half-open etc.). Alternatively, if you do want to count that as a connection error fine, but at least log some kind of error so the issue is discoverable and can be handled by someone.

Silent errors are deadly.

View Comments