A few years ago I ended up tasked with creating an in-depth monitoring solution for use by our clients. I was still doing my research on the topic when I was informed that not only had a client ordered this not-yet-created monitoring system (which I hadn't been informed of), but they had gone live with informing anyone at all, and were now upset that the monitoring system wasn't in place. So I now had a day to settle on the plan for a system and to implement it.
Luckily for me, I found out there were really only a few specific items this particular client was expecting to be monitored, so I decided to go with a central log server based system. A small program installed on each of their servers would run via cron, and inject warnings and notification into the outgoing syslog stream. As I had begun leaning towards a syslog based system, I had spent much of my time researching what to use to handle both the storage of a potentially large amount of logs, and the analysis required to make the whole thing useful. At the time, the only viable non-commercial offering seemed to be syslog-ng. It could supposedly handle a very large load, it could be used to handle a lot of the initial filtering, and could either pipe out to other programs for analysis, or easily write to many places for slightly delayed analysis. Ultimately, I configured it to store everything in one area of the disk, and to look for potentially interesting items and store those in another area, where a script running from cron would consume anything that came in, sending out email alerts and trouble tickets as required. It also use rrdtool to track and graph various stats like cpu usage, diskspace, and etc (this was because I was modifying code already being used for internally that happened to do this on a small scale for our own equipment).
The system has been hacked on over the past few years, and went through one hardware update. Now we are giving it a rather significant hardware upgrade, and want to make it more user friendly. Part of this user friendliness is to provide a web interface for viewing logs and interesting stats, and to allow configuration of monitors and etc by the user (right now configuration changes have to be requested and implemented by one us). While designing the new system, we decided to go with the possibility of logging at least some things to an sql database (everything would be nice, but I'm not convinced we have enough server power in place for that). The base system will be FreeBSD 7.2, and i spent most of a day trying to install syslog 3.0.4 with the ability to log to sql natively compiled. I tried using both the port (which was actually 3.0.3), and grabbing the source, but could never get it to compile with sql support enabled. It would appear to find the libnet libraries at first, but ultimately decided it really couldnt see them and would fail to compile. I'm far from being an expert on unix build tools, but usually if I fumble around long enough I find the correct invocations to get where I want to be. I enlisted a coworker who mentioned needing something to do, but he had no better luck.
syslog-ng does have the ability to dispatch logs to a fifo pipe, so I could do that and write or copy a script to handle the actual writing to the database ... but since I am already worried about performance I decided to look elsewhere again.
rsyslog was still very new when I first setup this system, so it wasn't much of a contender. However, it has really grown, so I'm going to give it a test run. After making sure the port system had a version I was happy with, i ran 'make install', and a few minutes later I had syslog server logging to mysql. I have another co-worker dealing with the web interface portion (I'll probably just have him hack up phpLogCon), I expect I'll need a day or so to rewrite and test the log analysis code I have in place now, and then I can begin slowly moving our servers to the new system. Hopefully by the end of next week all will have gone well, I'll be able to move our clients over, and move on to the next project.