Syslog Plugin for ClusterControl

Logging is a fundamental tool for the system administrator. It helps identify whether systems are running as configured, and track any unusual activity when trying to diagnose and isolate problems. Without even basic logging, when something unexpected happens, you could experience excessive downtime due to the lack of diagnostic data.

Syslog is a widely used standard for message logging. A centralized syslog server, where you can analyze logs of your infrastructure in one place, can greatly improve your visibility into what is going on. It is possible to have ClusterControl alarms sent to syslog via a syslog plugin. In this blog post, we will show you how to install this plugin. More plugins and integrations can be found on our Tools page.

What is syslog?

In a Linux operating system, the kernel and other internal components generate alerts and messages. These messages are typically stored in a file system or relayed to another device in the form of syslog messages. An internal daemon, called Syslogd, handles the syslog process. Syslog is widely used, and provides a central point for collecting and processing system logs. These system logs are useful later for troubleshooting and auditing.

The Syslog protocol is supported by a wide range of devices and can be used to log different types of events. Syslog is able to:

• accept inputs from a wide variety of sources
• transform them
• and output the results to diverse destinations

Syslog consists of three basic components:

1. Device: This is the equipment that generates log messages, this could be a server, router, firewall, or any other device that can generate syslog event data.
2. Collector: This equipment is configured to receive messages generated by a log device where it will subsequently store these externally generated messages local to itself.
3. Relay: Like a log collector it is configured to receive messages, these messages are not stored but forwarded to another device, usually a log collector but could be another relay.

Nowadays, most of Linux boxes come with syslog configured for local logging only and will not accept or process any messages from outside sources. In a local logging configuration the host performs both roles of device and collector, messages generated by the system are directed to the syslog process which routes these messages to a local destination on the host machine based on the rules set in a file /etc/syslog.conf.

ClusterControl Syslog Plugin

This plugin writes the new alarms instantly to the syslog, by default it would be /var/log/syslog on Debian-based systems or /var/log/messages on Redhat-based systems on the local host. If you have remote syslog configured for central log management, you can leverage this capability to capture ClusterControl alarms and process them. You can then use the information for things like a system health check, triggering an alert, intrusion detection or security audit purposes.

To install the plugin, simply do the following 4 steps on ClusterControl node:

1. Create the ClusterControl plugin directory. We will use the default location which is /var/cmon/plugins:

   $ mkdir -p /var/cmon/plugins

2. Grab the plugin from our s9s-admin Github repository:

   $ git clone https://github.com/severalnines/s9s-admin

3. Copy it over to the plugin directory:

   $ cp s9s-admin/plugins/syslog/syslog.py /var/cmon/plugins/

4. Restart CMON to load the plugin:

   $ service cmon restart

That’s it. Now you will get a copy of alarms triggered by ClusterControl in the local syslog. You can verify this with the following command on Debian:

$ tail -f /var/log/syslog
Jan 14 04:01:42 cc-server cmon-syslog-plugin: cluster-1, new alarm: MySQL server disconnected (id=573205676966301289)
Jan 14 04:20:31 cc-server cmon-syslog-plugin: cluster-1, new alarm: SSH failed (id=6970644932146548621)

For RHEL based systems, you can find them at /var/log/messages:

$ tail -f /var/log/messages
Jan 14 04:01:42 cc-server cmon-syslog-plugin: cluster-1, new alarm: MySQL server disconnected (id=573205676966301289)
Jan 14 04:20:31 cc-server cmon-syslog-plugin: cluster-1, new alarm: SSH failed (id=6970644932146548621)

To learn more on how ClusterControl plugin is executed, please refer to this blog post under ‘How does the plugin execute’ section. That’s all folks!