ClusterControl consists of a number of components and one of the most important ones is the ClusterControl UI. This is the web application through which the user interacts with other backend ClusterControl components like controller, notification, web-ssh module and cloud module. Each component is packaged independently with its own name, therefore easier for potential issues to be fixed and delivered to the end user. For more info on ClusterControl components, check out the documentation page.

In this blog post, we are going to look into ways to customize our ClusterControl installation, especially the ClusterControl UI which by default will be located under /var/www/html (default document root of Apache). Note that it's recommended to host ClusterControl on a dedicated server where it can use all the default paths which will simplify ClusterControl maintenance operations.

Installing ClusterControl

For a fresh installation, go to our Download page to get the installation link. Then, start installing ClusterControl using the installer script as root user:

$ whoami
root
$ wget https://severalnines.com/downloads/cmon/install-cc
$ chmod 755 install-cc
$ ./install-cc

Follow the installation wizard accordingly and the script will install all dependencies, configure ClusterControl components and start them up. The script will configure Apache 2.4, and use the package manager to install ClusterControl UI which by default located under /var/www/html.

Preparation

Once ClusterControl is installed into its default location, we can then move the UI directories located under /var/www/html/clustercontrol and /var/www/html/cmon into somewhere else. Let's prepare the new path first.

Suppose we want to move the UI components to a user directory under /home. Firstly, create the user. In this example, the user name is "cc":

$ useradd -m cc

The above command will automatically create a home directory for user "cc", under /home/cc. Then, create the necessary directories for Apache usage for this user. We are going to create a directory called "logs" for Apache logs, "public_html" for Apache document root of this user and the "www" as a symbolic link to the public_html:

$ cd /home/cc
$ mkdir logs
$ mkdir public_html
$ ln -sf public_html www

Make sure all of them are owned by Apache:

$ chown apache:apache logs public_html

To allow Apache process to access public_html under user cc, we have to allow global read to the home directory of user cc:

$ chmod 755 /home/cc

We are now good to move stuff.

Customizing the Path

Stop ClusterControl related services and Apache:

$ systemctl stop httpd # RHEL/CentOS
$ systemctl stop apache2 # Debian/Ubuntu
$ systemctl stop cmon cmon-events cmon-ssh cmon-cloud

We basically have two options in moving the directory into the user's directory:

1. Move everything from /var/www/html into /home/cc/public_html.
2. Create a symbolic link from /var/www/html/clustercontrol to /home/cc/public_html (recommended).

If you opt for option #1, simply move the ClusterControl UI directories into the new path,/home/cc/public_html:

$ mv /var/www/html/clustercontrol /home/cc/public_html/
$ mv /var/www/html/cmon /home/cc/public_html/

Make sure the ownership is correct:

$ chown -R apache:apache /home/cc/public_html # RHEL/CentOS
$ chown -R www-data:www-data /home/cc/public_html # Debian/Ubuntu

However, there is a drawback since ClusterControl UI package will always get extracted under /var/www/html. This means if you upgrade the ClusterControl UI via package manager, the new content will be available under /var/www/html. Refer to "Potential Issues" section further down for more details.

If you choose option #2, which is the recommended way, you just need to create a symlink (link reference to another file or directory) under the user's public_html directory for both directories. When an upgrade happens, the DEB/RPM postinst script will replace the existing installation with the updated version under /var/www/html. To do a symlink, simply:

$ ln -sf /var/www/html/clustercontrol /home/cc/public_html/clustercontrol
$ ln -sf /var/www/html/cmon /home/cc/public_html/cmon

Another step is required for option #2, where we have to allow Apache to follow symbolic links outside of the user's directory. Create a .htaccess file under /home/cc/public_html and add the following line:

# /home/cc/public_html/.htaccess
Options +FollowSymlinks -SymLinksIfOwnerMatch

Open Apache site configuration file at /etc/httpd/conf.d/s9s.conf (RHEL/CentOS) or /etc/apache2/sites-enabled/001-s9s.conf (Debian/Ubuntu) using your favourite text editor and modify it to be as below (pay attention on lines marked with ##):

<VirtualHost *:80>
    ServerName cc.domain.com  ## 

    ServerAdmin webmaster@cc.domain.com
    DocumentRoot /home/cc/public_html  ##

    ErrorLog /home/cc/logs/error.log  ##
    CustomLog /home/cc/logs/access.log combined  ##

    # ClusterControl SSH & events
    RewriteEngine On
    RewriteRule ^/clustercontrol/ssh/term$ /clustercontrol/ssh/term/ [R=301]
    RewriteRule ^/clustercontrol/ssh/term/ws/(.*)$ ws://127.0.0.1:9511/ws/$1 [P,L]
    RewriteRule ^/clustercontrol/ssh/term/(.*)$ http://127.0.0.1:9511/$1 [P]
    RewriteRule ^/clustercontrol/sse/events/(.*)$ http://127.0.0.1:9510/events/$1 [P,L]

    # Main Directories
    <Directory />
            Options +FollowSymLinks
            AllowOverride All
    </Directory>

    <Directory /home/cc/public_html>  ##
            Options +Indexes +FollowSymLinks +MultiViews
            AllowOverride All
            Require all granted
    </Directory>

</VirtualHost>

The similar modifications apply to the HTTPS configuration at /etc/httpd/conf.d/ssl.conf (RHEL/CentOS) or /etc/apache2/sites-enabled/001-s9s-ssl.conf (Debian/Ubuntu). Pay attention to lines marked with ##:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>

               ServerName cc.domain.com  ##
               ServerAdmin webmaster@cc.domain.com ##

               DocumentRoot /home/cc/public_html  ##

                # ClusterControl SSH & events

                RewriteEngine On
                RewriteRule ^/clustercontrol/ssh/term$ /clustercontrol/ssh/term/ [R=301]
                RewriteRule ^/clustercontrol/ssh/term/ws/(.*)$ ws://127.0.0.1:9511/ws/$1 [P,L]
                RewriteRule ^/clustercontrol/ssh/term/(.*)$ http://127.0.0.1:9511/$1 [P]
                RewriteRule ^/clustercontrol/sse/events/(.*)$ http://127.0.0.1:9510/events/$1 [P,L]

                <Directory />
                        Options +FollowSymLinks
                        AllowOverride All
                </Directory>

                <Directory /home/cc/public_html>  ##
                        Options +Indexes +FollowSymLinks +MultiViews
                        AllowOverride All
                        Require all granted
                </Directory>

                SSLEngine on
                SSLCertificateFile /etc/pki/tls/certs/s9server.crt # RHEL/CentOS
                SSLCertificateKeyFile /etc/pki/tls/private/s9server.key # RHEL/CentOS
                SSLCertificateKeyFile /etc/ssl/certs/s9server.crt # Debian/Ubuntu
                SSLCertificateKeyFile /etc/ssl/private/s9server.key # Debian/Ubuntu

                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>

                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>
                BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
        </VirtualHost>
</IfModule>

Restart everything:

$ systemctl restart httpd
$ systemctl restart cmon cmon-events cmon-ssh cmon-cloud

Consider the IP address of the ClusterControl is 192.168.1.202 and domain cc.domain.com resolves to 192.168.1.202, you can access ClusterControl UI via one of the following URLs:

• http://192.168.1.202/clustercontrol
• https://192.168.1.202/clustercontrol
• http://cc.domain.com/clustercontrol
• https://cc.domain.com/clustercontrol 

You should see the ClusterControl login page at this point. The customization is now complete.

Potential Issues

Since the package manager simply executes the post-installation script during package upgrade, the content of the new ClusterControl UI package (the actual package name is clustercontrol.x86_64) will be extracted into /var/www/html (it's hard coded inside post installation script). The following is what would happen:

$ ls -al /home/cc/public_html # our current installation
clustercontrol
cmon
$ ls -al /var/www/html # empty
$ yum upgrade clustercontrol -y
$ ls -al /var/www/html # new files are extracted here
clustercontrol
cmon

Therefore, if you use symlink method, you may skip the following additional steps.

To complete the upgrade process, one has to replace the existing installation under the custom path with the new installation manually. First, perform the upgrade operation:

$ yum upgrade clustercontrol -y # RHEL/CentOS
$ apt upgrade clustercontrol -y # Debian/Ubuntu

Move the existing installation to somewhere safe. We will need the old bootstrap.php file later on:

$ mv /home/cc/public_html/clustercontrol /home/cc/public_html/clustercontrol_old

Move the new installation from the default path /var/www/html into user's document root:

$ mv /var/www/html/clustercontrol /home/cc/public_html

Copy bootstrap.php from the old installation to the new one:

$ mv /home/cc/public_html/clustercontrol_old/bootstrap.php /home/cc/public_html/clustercontrol

Get the new version string from bootstrap.php.default:

$ grep CC_UI_VERSION /home/cc/public_html/clustercontrol/bootstrap.php.default
define('CC_UI_VERSION', '1.7.4.6537-#f427cb');

Update the new version string for CC_UI_VERSION value inside bootstrap.php using your favourite text editor:

$ vim /home/cc/public_html/clustercontrol/bootstrap.php

Save the file and the upgrade is now complete.

That's it, folks. Happy customizing!

ClusterControl is all about delivering robust, open source, database management for the IT management needs of our clients. This goal drives us every day, so much so that it lead us to receive two awards recently from CompareCamp: the Rising Star of 2019 Award and the Great User Experience Award.

CompareCamp is a B2B Review Platform that delivers credible SaaS reviews and updated technology news from industry experts. Thousands of users rely on CompareCamp reviews that detail the pros and cons of software from different industries. 

ClusterControl was given the Great User Experience Award because it effectively boosted users’ rate of productivity through highly secured tools for real-time monitoring, failure detection, load balancing, data migrating, and automated recovery. Our dedicated features for node recovery, SSL encryption, and performance reporting received raves from experts.

We also received the Rising Star of 2019 Award due to our initial review and being a highly recommended IT management software by CompareCamp. 

To read the full review, please visit CompareCamp.

A disaster usually causes an outage, which means system downtime and potential loss of data. Once we have detected the blackout, we trigger our DR plan to recover from it. But it would be a surprise, if there is no backup, or after long hours of recovery, you see it's not the one you need. 

While outages can be costly - there is often a financial impact  which can be harmful to the business and data loss may be a reason to close the company. 

To minimize data loss, we need to have multiple copies of data in various places. We can design our infrastructure in different layers and abstract each layer from the one below it. For instance, we build a layer for clusters of database instances to protect against hardware failure. We replicate databases across datacenters so we can defend ourselves against a data center failure. Every additional layer adds complexity, which can become a nightmare to manage. But still, in essence, a backup will take the central place in the disaster recovery.

That's why it's crucial to be sure it's something we can rely on. But how to achieve this? Well, one of the options is to verify if backups were executed based on the last few lines of backup script. 

A simple example:

#!/bin/sh

mysqldump -h 192.168.1.1 -u user -ppassword dbname > filename.sql



if [ "$?" -eq 0 ]; then

    echo "Success."

else

    echo "Error."

fi

But what if the backup script did  not start at all? Google offers quite a bit of search results for "Linux cron, not running." 

Unfortunately, open-source databases often do not offer backup repository. 

Another backup testing. You may have heard  about Schrödinger's cat.  A known Schrödinger's Backup theory is . "The condition of any backup is unknown until a restore is attempted." Sounds like a simple approach but such an attempt would mean you have to set up a test environment, copy files run restore ... after every backup. 

In this article, we will see how you can use ClusterControl to make sure your backup is executed  to achieve Enterprise-Grade databases with Open Source Databases.

Backup Reports

ClusterControl has been aimed at operational reports. Operational Reporting provides support to day-to-day enterprise activity monitoring and control. The backup report is one of many. You can find reports like:

• Daily System Report
• Package Upgrade Report
• Schema Change Report
• Availability 
• Backup

But why you would need this?

You may already have an excellent monitoring tool with all possible metrics/graphs and you probably have also set up alerts based on metrics and thresholds (some will even have automated advisors providing them recommendations or fixing things automatically.) That's good - having visibility into your system is important; nevertheless, you need to be able to process a lot of information.

How does this work? ClusterControl collects information on the backup process, the systems, platforms, and devices in the backup infrastructure when the backup job is triggered. All of that information is aggregated and stored in a CMON (internal database), so there is no need to query particular databases additionally. Additionally, when it discovers that you have a running cluster, but there was no backup, it will be reported too.

In the report details, you can track a  backup ID with detailed data about the location, size, time, and backup method. Templates work with data for different database types, so when you manage your mixed environment, you will get the same feel and look. It helps to manage different database backups better.

CLI Reports

For those who prefer the command-line interface, a good option to track backups ClusterControl Command Line Interface (CLI).

CLI lets you execute most of the functions available within ClusterControl using simple commands. Backup execution and backup reports are one of them. 

Used in conjunction with the powerful GUI, it gives ClusterControl users alternative ways to manage their open-source database environments using whatever engine they prefer.

$ s9s backup --list --cluster-id=1 --long --human-readable

ID CID STATE     OWNER HOSTNAME CREATED  SIZE FILENAME

 1   1 COMPLETED dba   10.0.0.5 07:21:39 252K mysqldump_2017-05-09_072135_mysqldb.sql.gz

 1   1 COMPLETED dba   10.0.0.5 07:21:43 1014 mysqldump_2017-05-09_072135_schema.sql.gz

 1   1 COMPLETED dba   10.0.0.5 07:22:03 109M mysqldump_2017-05-09_072135_data.sql.gz

 1   1 COMPLETED dba   10.0.0.5 07:22:07 679 mysqldump_2017-05-09_072135_triggerseventsroutines.sql.gz

 2   1 COMPLETED dba   10.0.0.5 07:30:20 252K mysqldump_2017-05-09_073016_mysqldb.sql.gz

 2   1 COMPLETED dba   10.0.0.5 07:30:24 1014 mysqldump_2017-05-09_073016_schema.sql.gz

 2   1 COMPLETED dba   10.0.0.5 07:30:44 109M mysqldump_2017-05-09_073016_data.sql.gz

 2   1 COMPLETED dba   10.0.0.5 07:30:49 679 mysqldump_2017-05-09_073016_triggerseventsroutines.sql.gz

Beginning from version 1.4.1, the installer script will automatically install this package on the ClusterControl node. CLI is part of s9s-tools package. You can also install it separately on a different machine to manage the database cluster remotely. Similar to ClusterControl it uses secure SSH communication. 

Automatic Backup Verification

A backup is not a backup if we are not able to retrieve the data. Verifying backups is something that is usually overlooked by many companies. Let’s see how ClusterControl can automate the verification of backups and help avoid any surprises.

In ClusterControl, select your cluster and go to the "Backup" section, then, select “Create Backup”.

The automatic verify backup feature is available for the scheduled backups so, let’s choose the “Schedule Backup” option.

When scheduling a backup, in addition to selecting the common options like method or storage, we also need to specify schedule/frequency. In this example, we are going to setup MySQL backup verification. However the same can be achieved for PostgreSQL and Timescale databases. 

When backup verification is checked another tab will appear.

Here we can set all the necessary steps to prepare the environment. When IP is provided we are good to go and schedule such backup. Whenever backup finishes it will be copied to a temporary backup verification environment (“restore backup on” option).  After successful refresh, you will see the status of verification in the backup repository tab. 

Failed Backup Executions and Integration Services

Another interesting option to get more clues about backup execution is to use ClusterControl Integration services. You can control the backup execution status with third-party services.

Third-party tools integration enables you to automate alerts with other popular systems. Currently, ClusterControl supports ServiceNow, PagerDuty, VictorOps, OpsGenie, Slack, Telegram, and Webhooks. 

Below we can see an example of Slack channel integration. Whenever a backup event occurs it will appear in the slack channel.  

Conclusion

Backups are mandatory in any environment. They help you protect your data and are in the center of any disaster recovery scenario. ClusterControl can help automate the backup process for your databases and, in case of failure, restore it with a few clicks. Also, you can be sure they are executed successfully and reliable so in case of disaster, you will not lose your data.

High Availability is a paramount nowadays and there’s no better way to introduce high availability than to build it on top of the quorum-based cluster. Such cluster is able to easily handle failures of individual nodes and ensure that all nodes, which have disconnected from the cluster, will not continue to operate. There are several protocols that allow you to solve consensus issues, examples being Paxos or RAFT. You can always introduce your own code. 

With this in mind, we would like to introduce you to CMON HA, a solution we created which allows to build highly available clusters of cmon daemons to achieve ClusterControl high availability. Please keep in mind this is a beta feature - it works but we are adding better debugging and more usability features. Having said that, let’s take a look at how it can be deployed, configured and accessed.

Prerequisites

CMON, the daemon that executes tasks in ClusterControl, works with a MySQL database to store some of the data - configuration settings, metrics, backup schedules and many others. In the typical setup this is a standalone MySQL instance. As we want to build highly available solution, we have to consider highly available database backend as well. One of the common solutions for that is MySQL Galera Cluster. As the installation scripts for ClusterControl sets up the standalone database, we have to deploy our Galera Cluster first, before we attempt to install highly available ClusterControl. What is the better way of deploying a Galera cluster than using ClusterControl? We will use temporary ClusterControl to deploy Galera on top of which we will deploy highly available version of ClusterControl.

Deploying a MySQL Galera Cluster

We won’t cover here the installation of the standalone ClusterControl. It’s as easy as downloading it for free and then following the steps you are provided with. Once it is ready, you can use the deployment wizard to deploy 3 nodes of Galera Cluster in couple of minutes.

Pick the deployment option, you will be then presented with a deployment wizard.

Define SSH connectivity details. You can use either root or password or passwordless sudo user. Make sure you correctly set SSH port and path to the SSH key.

Then you should pick a vendor, version and few of the configuration details including server port and root password. Finally, define the nodes you want to deploy your cluster on. Once this is done, ClusterControl will deploy a Galera cluster on the nodes you picked. From now on you can as well remove this ClusterControl instance, it won’t be needed anymore.

Deploying a Highly Available ClusterControl Installation

We are going to start with one node, configure it to start the cluster and then we will proceed with adding additional nodes.

Enabling Clustered Mode on the First Node

What we want to do is to deploy a normal ClusterControl instance therefore we are going to proceed with typical installation steps. We can download the installation script and then run it. The main difference, compared to the steps we took when we installed a temporary ClusterControl to deploy Galera Cluster, is that in this case there is already existing MySQL database. Thus the script will detect it, ask if we want to use it and if so, request password for the superuser. Other than that, installation is basically the same.

Next step would be to reconfigure cmon to listen not only on the localhost but also to bind to IP’s that can be accessed from outside. Communication between nodes in the cluster will happen on that IP on port (by default) 9501. We can accomplish this by editing file: /etc/default/cmon and adding IP to the RPC_BIND_ADDRESSES variable:

RPC_BIND_ADDRESSES="127.0.0.1,10.0.0.101"

Afterwards we have to restart cmon service:

service cmon restart

Following step will be to configure s9s CLI tools, which we will use to create and monitor cmon HA cluster. As per the documentation, those are the steps to take:

wget http://repo.severalnines.com/s9s-tools/install-s9s-tools.sh

chmod 755 install-s9s-tools.sh

./install-s9s-tools.sh

Once we have s9s tools installed, we can enable the clustered mode for cmon:

s9s controller --enable-cmon-ha

We can then verify the state of the cluster:

s9s controller --list --long

S VERSION    OWNER GROUP NAME            IP PORT COMMENT

l 1.7.4.3565 system admins 10.0.0.101      10.0.0.101 9501 Acting as leader.

Total: 1 controller(s)

As you can see, we have one node up and it is acting as a leader. Obviously, we need at least three nodes to be fault-tolerant therefore the next step will be to set up the remaining nodes.

Enabling Clustered Mode on Remaining Nodes

There are a couple of things we have to keep in mind while setting up additional nodes. First of all, ClusterControl creates tokens that “links” cmon daemon with clusters. That information is stored in several locations, including in the cmon database therefore we have to ensure every place contains the same token. Otherwise cmon nodes won’t be able to collect information about clusters and execute RPC calls. To do that we should copy existing configuration files from the first node to the other nodes. In this example we’ll use node with IP of 10.0.0.103 but you should do that for every node you plan to include in the cluster.

We’ll start by copying the cmon configuration files to new node:

scp -r /etc/cmon* 10.0.0.103:/etc/

We may need to edit /etc/cmon.cnf and set the proper hostname:

hostname=10.0.0.103

Then we’ll proceed with regular installation of the cmon, just like we did on the first node. There is one main difference though. Script will detect configuration files and ask if we want to install the controller:

=> An existing Controller installation detected!

=> A re-installation of the Controller will overwrite the /etc/cmon.cnf file

=> Install the Controller? (y/N):

We don’t want to do it for now. As on the first node we will be asked if we want to use existing MySQL database. We do want that. Then we’ll be asked to provide passwords:

=> Enter your MySQL root user's password:

=> Set a password for ClusterControl's MySQL user (cmon) [cmon]

=> Supported special characters: ~!@#$%^&*()_+{}<>?

=> Enter a CMON user password:

=> Enter the CMON user password again: => Creating the MySQL cmon user ...

Please make sure you use exactly the same password for cmon user as you did on the first node.

As the next step, we want to install s9s tools on new nodes:

wget http://repo.severalnines.com/s9s-tools/install-s9s-tools.sh

chmod 755 install-s9s-tools.sh

./install-s9s-tools.sh

We want to have them configured exactly as on the first node thus we’ll copy the config:

scp -r ~/.s9s/ 10.0.0.103:/root/

scp /etc/s9s.conf 10.0.0.103:/etc/

There’s one more place where we ClusterControl stores token: /var/www/clustercontrol/bootstrap.php. We want to copy that file as well:

scp /var/www/clustercontrol/bootstrap.php 10.0.0.103:/var/www/clustercontrol/

Finally, we want to install the controller (as we skipped this when we ran the installation script):

apt install clustercontrol-controller

Make sure you do not overwrite existing configuration files. Default options should be safe and leave correct configuration files in place.

There is one more piece of configuration you may want to copy: /etc/default/cmon. You want to copy it to other nodes:

scp /etc/default/cmon 10.0.0.103:/etc/default

Then you want to edit RPC_BIND_ADDRESSES to point to a correct IP of the node.

RPC_BIND_ADDRESSES="127.0.0.1,10.0.0.103"

Then we can start the cmon service on the nodes, one by one, and see if they managed to join the cluster. If everything went well, you should see something like this:

s9s controller --list --long

S VERSION    OWNER GROUP NAME            IP PORT COMMENT

l 1.7.4.3565 system admins 10.0.0.101      10.0.0.101 9501 Acting as leader.

f 1.7.4.3565 system admins 10.0.0.102      10.0.0.102 9501 Accepting heartbeats.

f 1.7.4.3565 system admins 10.0.0.103      10.0.0.103 9501 Accepting heartbeats.

Total: 3 controller(s)

In case of any issues, please check if all the cmon services are bound to the correct IP addresses. If not, kill them and start again, to re-read the proper configuration:

root      8016 0.4 2.2 658616 17124 ?        Ssl 09:16 0:00 /usr/sbin/cmon --rpc-port=9500 --bind-addr='127.0.0.1,10.0.0.103' --events-client='http://127.0.0.1:9510' --cloud-service='http://127.0.0.1:9518'

If you manage to see the output from ‘s9s controller --list --long’ as above, this means that, technically, we have a running cmon HA cluster of three nodes. We can end here but it’s not over yet. The main problem that remains is the UI access. Only leader node can execute jobs. Some of the s9s commands support this but as of now UI does not. This means that the UI will work only on the leader node, in our current situation it is the UI accessible via https://10.0.0.101/clustercontrol. 

In the second part we will show you one of the ways in which you could solve this problem.

In the first part, we ended up with a working cmon HA cluster:

root@vagrant:~# s9s controller --list --long

S VERSION    OWNER GROUP NAME            IP PORT COMMENT

l 1.7.4.3565 system admins 10.0.0.101      10.0.0.101 9501 Acting as leader.

f 1.7.4.3565 system admins 10.0.0.102      10.0.0.102 9501 Accepting heartbeats.

f 1.7.4.3565 system admins 10.0.0.103      10.0.0.103 9501 Accepting heartbeats.

Total: 3 controller(s)

We have three nodes up and running, one is acting as a leader and remaining are followers, which are accessible (they do receive heartbeats and reply to them). The remaining challenge is to configure UI access in a way that will allow us to always access the UI on the leader node. In this blog post we will present one of the possible solutions which will allow you to accomplish just that.

Setting up HAProxy

This problem is not new to us. With every replication cluster, MySQL or PostgreSQL, it doesn’t matter,  there’s a single node where we should send our writes to. One way of accomplishing that would be to use HAProxy and add some external checks that test the state of the node, and based on that, return proper values. This is basically what we are going to use to solve our problem. We will use HAProxy as a well-tested layer 4 proxy and we will combine it with layer 7 HTTP checks that we will write precisely for our use case. First things first, let’s install HAProxy. We will collocate it with ClusterControl, but it can as well be installed on a separate node (ideally, nodes - to remove HAProxy as the single point of failure).

apt install haproxy

This sets up HAProxy. Once it’s done, we have to introduce our configuration:

global

        pidfile /var/run/haproxy.pid

        daemon

        user haproxy

        group haproxy

        stats socket /var/run/haproxy.socket user haproxy group haproxy mode 600 level admin

        node haproxy_10.0.0.101

        description haproxy server



        #* Performance Tuning

        maxconn 8192

        spread-checks 3

        quiet

defaults

        #log    global

        mode    tcp

        option  dontlognull

        option tcp-smart-accept

        option tcp-smart-connect

        #option dontlog-normal

        retries 3

        option redispatch

        maxconn 8192

        timeout check   10s

        timeout queue   3500ms

        timeout connect 3500ms

        timeout client  10800s

        timeout server  10800s



userlist STATSUSERS

        group admin users admin

        user admin insecure-password admin

        user stats insecure-password admin



listen admin_page

        bind *:9600

        mode http

        stats enable

        stats refresh 60s

        stats uri /

        acl AuthOkay_ReadOnly http_auth(STATSUSERS)

        acl AuthOkay_Admin http_auth_group(STATSUSERS) admin

        stats http-request auth realm admin_page unless AuthOkay_ReadOnly

        #stats admin if AuthOkay_Admin



listen  haproxy_10.0.0.101_81

        bind *:81

        mode tcp

        tcp-check connect port 80

        timeout client  10800s

        timeout server  10800s

        balance leastconn

        option httpchk

#        option allbackups

        default-server port 9201 inter 20s downinter 30s rise 2 fall 2 slowstart 60s maxconn 64 maxqueue 128 weight 100

        server 10.0.0.101 10.0.0.101:443 check

        server 10.0.0.102 10.0.0.102:443 check

        server 10.0.0.103 10.0.0.103:443 check

You may want to change some of the things here like the node or backend names which include here the IP of our node. You will definitely want to change servers that you are going to have included in your HAProxy.

The most important bits are:

        bind *:81

HAProxy will listen on port 81.

        option httpchk

We have enabled layer 7 check on the backend nodes.

        default-server port 9201 inter 20s downinter 30s rise 2 fall 2 slowstart 60s maxconn 64 maxqueue 128 weight 100

The layer 7 check will be executed on port 9201.

Once this is done, start HAProxy.

Setting up xinetd and Check Script

We are going to use xinetd to execute the check and return correct responses to HAProxy. Steps described in this paragraph should be executed on all cmon HA cluster nodes.

First, install xinetd:

root@vagrant:~# apt install xinetd

Once this is done, we have to add the following line:

cmonhachk       9201/tcp

to /etc/services - this will allow xinetd to open a service that will listen on port 9201. Then we have to add the service file itself. It should be located in /etc/xinetd.d/cmonhachk:

# default: on

# description: cmonhachk

service cmonhachk

{

        flags           = REUSE

        socket_type     = stream

        port            = 9201

        wait            = no

        user            = root

        server          = /usr/local/sbin/cmonhachk.py

        log_on_failure  += USERID

        disable         = no

        #only_from       = 0.0.0.0/0

        only_from       = 0.0.0.0/0

        per_source      = UNLIMITED

}

Finally, we need the check script that’s called by the xinetd. As defined in the service file it is located in /usr/local/sbin/cmonhachk.py.

#!/usr/bin/python3.5



import subprocess

import re

import sys

from pathlib import Path

import os



def ret_leader():

    leader_str = """HTTP/1.1 200 OK\r\n

Content-Type: text/html\r\n

Content-Length: 48\r\n

\r\n

<html><body>This node is a leader.</body></html>\r\n

\r\n"""

    print(leader_str)



def ret_follower():

    follower_str = """

HTTP/1.1 503 Service Unavailable\r\n

Content-Type: text/html\r\n

Content-Length: 50\r\n

\r\n

<html><body>This node is a follower.</body></html>\r\n

\r\n"""

    print(follower_str)



def ret_unknown():

    unknown_str = """

HTTP/1.1 503 Service Unavailable\r\n

Content-Type: text/html\r\n

Content-Length: 59\r\n

\r\n

<html><body>This node is in an unknown state.</body></html>\r\n

\r\n"""

    print(unknown_str)



lockfile = "/tmp/cmonhachk_lockfile"



if os.path.exists(lockfile):

    print("Lock file {} exists, exiting...".format(lockfile))

    sys.exit(1)



Path(lockfile).touch()

try:

    with open("/etc/default/cmon", 'r') as f:

        lines  = f.readlines()



    pattern1 = "RPC_BIND_ADDRESSES"

    pattern2 = "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"

    m1 = re.compile(pattern1)

    m2 = re.compile(pattern2)



    for line in lines:

        res1 = m1.match(line)

        if res1 is not None:

            res2 = m2.findall(line)

            i = 0

            for r in res2:

                if r != "127.0.0.1" and i == 0:

                    i += 1

                    hostname = r



    command = "s9s controller --list --long | grep {}".format(hostname)

    output = subprocess.check_output(command.split())

    state = output.splitlines()[1].decode('UTF-8')[0]

    if state == "l":

        ret_leader()

    if state == "f":

        ret_follower()

    else:

        ret_unknown()

finally:

    os.remove(lockfile)

Once you create the file, make sure it is executable:

chmod u+x /usr/local/sbin/cmonhachk.py

The idea behind this script is that it tests the status of the nodes using “s9s controller --list --long” command and then it checks the output relevant to the IP that it can find on the local node. This allows the script to determine if the host on which it is executed is a leader or not. If the node is the leader, script returns “HTTP/1.1 200 OK” code, which HAProxy interprets as the node is available and routes the traffic to it.. Otherwise it returns “HTTP/1.1 503 Service Unavailable”, which is treated as a node, which is not healthy and the traffic will not be routed there. As a result, no matter which node will become a leader, HAProxy will detect it and mark it as available in the backend:

You may need to restart HAProxy and xinetd to apply configuration changes before all the parts will start working correctly.

Having more than one HAProxy ensures we have a way to access ClusterControl UI even if one of HAProxy nodes would fail but we still have two (or more) different hostnames or IP to connect to the ClusterControl UI. To make it more comfortable, we will deploy Keepalived on top of HAProxy. It will monitor the state of HAProxy services and assign Virtual IP to one of them. If that HAProxy would become unavailable, VIP will be moved to another available HAProxy. As a result, we’ll have a single point of entry (VIP or a hostname associated to it). The steps we’ll take here have to be executed on all of the nodes where HAProxy has been installed.

First, let’s install keepalived:

apt install keepalived

Then we have to configure it. We’ll use following config file:

vrrp_script chk_haproxy {

   script "killall -0 haproxy"   # verify the pid existance

   interval 2                    # check every 2 seconds

   weight 2                      # add 2 points of prio if OK

}

vrrp_instance VI_HAPROXY {

   interface eth1                # interface to monitor

   state MASTER

   virtual_router_id 51          # Assign one ID for this route

   priority 102                   

   unicast_src_ip 10.0.0.101

   unicast_peer {

      10.0.0.102

10.0.0.103



   }

   virtual_ipaddress {

       10.0.0.130                        # the virtual IP

   } 

   track_script {

       chk_haproxy

   }

#    notify /usr/local/bin/notify_keepalived.sh

}

You should modify this file on different nodes. IP addresses have to be configured properly and priority should be different on all of the nodes. Please also configure VIP that makes sense in your network. You may also want to change the interface - we used eth1, which is where the IP is assigned on virtual machines created by Vagrant.

Start the keepalived with this configuration file and you should be good to go. As long as VIP is up on one HAProxy node, you should be able to use it to connect to the proper ClusterControl UI:

This completes our two-part introduction to ClusterControl highly available clusters. As we stated at the beginning, this is still in beta state but we are looking forward for feedback from your tests.

ClusterControl uses the Apache HTTP Server to serve its web interface, but it is also possible to use nginx. nginx + PHP fastcgi is well-known for its capabilities to run on a small memory footprint compared to standard Apache + PHP DSO.

In this post, we will show you how to run ClusterControl 1.7.5 and later on nginx web server by swapping out the default Apache web server installed during the initial deployment. This blog post does not mean that we officially support nginx, it just an alternative way that a portion of our users have been interested in.

Apache Configuration

Before we jump into nginx configurations, let’s look at how ClusterControl web application is configured with Apache web server. ClusterControl consists of a number of components, and some of them require specific Apache module to run properly:

• ClusterControl UI - Requires Apache rewrite module + PHP 5.4 and later
• ClusterControl Controller
• ClusterControl Notifications - Requires Apache rewrite module
• ClusterControl SSH - Requires Apache 2.4 proxy module (wstunnel for web socket)
• ClusterControl Cloud

ClusterControl UI is located in the Apache’s document root which might vary depending on the operating system. For legacy OS distribution like Ubuntu 14.04 LTS and Debian 8, the Apache's document root is located at /var/www. For more recent OS distributions, most of them are now running with Apache 2.4 with /var/www/html as the default document root.

Step One

Make sure ClusterControl UI exists in the Apache document root. Document root for RedHat/CentOS and Ubuntu 14.04 LTS (Apache 2.4) is located at /var/www/html while Debian and Ubuntu 12.04 and lower is located at /var/www. ClusterControl UI will be installed under this document root directory and you should see something like this:

$ ls -al /var/www/html

total 16

drwxr-xr-x 4 root   root 4096 Aug 8 11:42 .

drwxr-xr-x 4 root   root 4096 Dec 19 03:32 ..

dr-xr-xr-x 6 apache apache 4096 Dec 19 03:38 clustercontrol

drwxrwx--- 3 apache apache 4096 Dec 19 03:29 cmon

Step Two

Apache must be able to read custom configuration file (.htaccess) under the document root directory. Thus the installer script will generate a configuration file and set the global AllowOverride option to All. Example in /etc/httpd/conf.d/s9s.conf:

<Directory />

            Options +FollowSymLinks

            AllowOverride All

    </Directory>

    <Directory /var/www/html>

            Options +Indexes +FollowSymLinks +MultiViews

            AllowOverride All

            Require all granted

    </Directory>

Step Three

ClusterControl also requires the following rewrite rules:

    RewriteEngine On

    RewriteRule ^/clustercontrol/ssh/term$ /clustercontrol/ssh/term/ [R=301]

    RewriteRule ^/clustercontrol/ssh/term/ws/(.*)$ ws://127.0.0.1:9511/ws/$1 [P,L]

    RewriteRule ^/clustercontrol/ssh/term/(.*)$ http://127.0.0.1:9511/$1 [P]

    RewriteRule ^/clustercontrol/sse/events/(.*)$ http://127.0.0.1:9510/events/$1 [P,L]

The first 3 URL rewrite rules indicate that ClusterControl SSH URL will be rewritten to use WebSocket tunneling on port 9511. This allows ClusterControl users to access the monitored nodes via SSH directly inside the ClusterControl UI.

You may also notice another line with "sse/events" where the URL will be rewritten to port 9510 for cmon-events integration. Application cmon-events is a binary comes within ClusterControl Notifications package for notification integration with 3rd-party software like Slack, Telegram, Pagerduty and web hooks.

Step Four

Thus, ClusterControl suite requires the following PHP/Apache modules to be installed and enabled:

• common
• mysql
• ldap
• gd
• curl
• mod_proxy (websocket)

The standard Apache installation via package manager will install PHP to run as dynamic shared object (DSO). Running on this mode will require you to restart Apache in case of PHP configuration changes.

The following command should install all required packages for ClusterControl:

$ yum install httpd php php-mysql php-ldap php-gd php-curl mod_ssl #RHEL/CentOS

$ apt-get install apache2 php5-common php5-mysql php5-ldap php5-gd libapache2-mod-php5 php5-json php5-curl #Debian/Ubuntu

Step Five

The ClusterControl web components must be owned by Apache web server user ("apache" for RHEL/CentOS and "www-data" for Debian/Ubuntu).

Switching from Apache to nginx

We would need to configure nginx to behave similarly to our Apache configuration, as most of the Severalnines tools assume that ClusterControl is running on Apache. 

Step One

Install ClusterControl via the installer script:

$ wget https://severalnines.com/downloads/cmon/install-cc

$ chmod 755 install-cc

$ ./install-cc

The above will install ClusterControl and its components on top of Apache web server.

Step Two

Enable nginx repository. Depending on your operating system, please refer to this installation guide for details.

Step Three

Install nginx and PHP FPM:

$ yum install nginx php-fpm -y #RHEL/CentOS

$ sudo apt-get install nginx php5-fpm -y #Debian/Ubuntu

Step Four

Take note that removing Apache2 directly might cause dependent PHP packages to be uninstalled as well. So we take a safer approach by just turning it off and disabling it to start on boot:

Systemd:

$ systemctl stop httpd

$ systemctl disable httpd

Sysvinit RHEL/CentOS:

$ chkconfig httpd off

$ service httpd stop

Sysvinit Debian/Ubuntu:

$ sudo update-rc.d -f apache2 remove

$ sudo service apache2 stop

Step Five

Open the nginx default virtual host configuration file (RHEL/CentOS: /etc/nginx/conf.d/default.conf, Debian/Ubuntu: /etc/nginx/sites-available/default) and make sure it contains the following lines:

server {

        listen       0.0.0.0:80;

        server_name  localhost;



        access_log /var/log/nginx/localhost-access.log;

        error_log /var/log/nginx/localhost-error.log;



        root /var/www/html;

        index index.php;



        location ~ \.htaccess {

                deny all;

        }



        location ~ \.php$ {

                fastcgi_pass 127.0.0.1:9000;

                fastcgi_index index.php;

                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

                include /etc/nginx/fastcgi_params;

        }



        # Handle requests to /clustercontrol

        location /clustercontrol {

                alias /var/www/html/clustercontrol/app/webroot;

                try_files $uri $uri/ /clustercontrol/app/webroot/index.php;

        }



        # Equivalent of $is_args but adds an & character

        set $is_args_amp "";

        if ($is_args != "") {

                set $is_args_amp "&";

        }



        # Handle requests to /clustercontrol/access

        location ~ "^/clustercontrol/access/(.*)$" {

                try_files $uri $uri/ /clustercontrol/app/webroot/access/index.php?url=$1$is_args_amp$args;

        }



        # Handle requests to /clustercontrol/access2

        location ~ "^/clustercontrol/access2/(.*)$" {

                try_files $uri $uri/ /clustercontrol/app/webroot/access2/index.php?url=$1$is_args_amp$args;

        }



        # Pass to cmon-events module

        location /clustercontrol/sse/events/ {

                proxy_pass http://127.0.0.1:9510/events/;

        }



        # Pass to cmon-ssh module

        location /clustercontrol/ssh/term/ {

                proxy_pass http://127.0.0.1:9511/;

        }



        # Pass cmon-ssh module via websocket

        location /clustercontrol/ssh/term/ws/ {

                proxy_set_header X-Forwarded-Host $host:$server_port;

                proxy_set_header X-Forwarded-Server $host;

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_http_version 1.1;

                proxy_set_header Upgrade $http_upgrade;

                proxy_set_header Connection "upgrade";

                proxy_pass http://127.0.0.1:9511/ws/;

        }



        # Handle requests to /clustercontrol/ssh

        location /clustercontrol/ssh/ {

                try_files $uri $uri/ /clustercontrol/app/webroot/index.php?url=$1$is_args_amp$args;

        }



        # Redirect /clustercontrol/ssh/term to /term/

        rewrite ^/clustercontrol/ssh/term$ /clustercontrol/ssh/term/$1 permanent;



}

The above configuration example is specifically written to run ClusterControl UI on nginx in RHEL/CentOS. For other OS distributions, replace any occurrences of /var/www/html to its respective document root.

Step Six

Create a new virtual host configuration for HTTPS (optional):

$ vim /etc/nginx/conf.d/s9s-ssl.conf #RHEL/CentOS

$ vim /etc/nginx/sites-available/s9s-ssl #Debian/Ubuntu

And make sure it contains the following lines:

server {

        listen       443 ssl;

        server_name  localhost;



        access_log /var/log/nginx/localhost-access.log;

        error_log /var/log/nginx/localhost-error.log;



        # SSL cert and key path

        ssl_certificate      /etc/pki/tls/certs/s9server.crt;

        ssl_certificate_key  /etc/pki/tls/private/s9server.key;



        ssl_session_cache shared:SSL:1m;

        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;

        ssl_prefer_server_ciphers   on;




        root /var/www/html;

        index index.php;



        location ~ \.htaccess {

                deny all;

        }



        location ~ \.php$ {

                fastcgi_pass 127.0.0.1:9000;

                fastcgi_index index.php;

                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;

                include /etc/nginx/fastcgi_params;

        }



        # Handle requests to /clustercontrol

        location /clustercontrol {

                alias /var/www/html/clustercontrol/app/webroot;

                try_files $uri $uri/ /clustercontrol/app/webroot/index.php;

        }



        # Equivalent of $is_args but adds an & character

        set $is_args_amp "";

        if ($is_args != "") {

                set $is_args_amp "&";

        }



        # Handle requests to /clustercontrol/access

        location ~ "^/clustercontrol/access/(.*)$" {

                try_files $uri $uri/ /clustercontrol/app/webroot/access/index.php?url=$1$is_args_amp$args;

        }



        # Handle requests to /clustercontrol/access2

        location ~ "^/clustercontrol/access2/(.*)$" {

                try_files $uri $uri/ /clustercontrol/app/webroot/access2/index.php?url=$1$is_args_amp$args;

        }



        # Pass to cmon-events module

        location /clustercontrol/sse/events/ {

                proxy_pass http://127.0.0.1:9510/events/;

        }



        # Pass to cmon-ssh module

        location /clustercontrol/ssh/term/ {

                proxy_pass http://127.0.0.1:9511/;

        }



        # Pass cmon-ssh module via websocket

        location /clustercontrol/ssh/term/ws/ {

                proxy_set_header X-Forwarded-Host $host:$server_port;

                proxy_set_header X-Forwarded-Server $host;

                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

                proxy_http_version 1.1;

                proxy_set_header Upgrade $http_upgrade;

                proxy_set_header Connection "upgrade";

                proxy_pass http://127.0.0.1:9511/ws/;

        }



        # Handle requests to /clustercontrol/ssh

        location /clustercontrol/ssh/ {

                try_files $uri $uri/ /clustercontrol/app/webroot/index.php?url=$1$is_args_amp$args;

        }



        # Redirect /clustercontrol/ssh/term to /term/

        rewrite ^/clustercontrol/ssh/term$ /clustercontrol/ssh/term/$1 permanent;



}

The above configuration example is specifically written to run ClusterControl UI on nginx in RHEL/CentOS. Replace any occurrences of the following:

• /var/www/html to its respective document root for other OS distribution
• /etc/pki/tls/certs/s9server.crt to /etc/ssl/certs/s9server.crt for Debian/Ubuntu
• /etc/pki/tls/private/s9server.key to /etc/ssl/private/s9server.key for Debian/Ubuntu

For Debian/Ubuntu, and extra step is needed to create a symlink for /etc/nginx/sites-enabled/default-ssl:
 

$ sudo ln -sf /etc/nginx/sites-available/default-ssl /etc/nginx/sites-enabled/default-ssl

Step Seven

Enable and start nginx and php-fpm:

Systemd:

$ systemctl enable php-fpm

$ systemctl enable nginx

$ systemctl restart php-fpm

$ systemctl restart nginx

Sysvinit RHEL/CentOS:

$ chkconfig php-fpm on

$ chkconfig nginx on

$ service php-fpm start

$ service nginx start

Sysvinit Debian/Ubuntu:

$ sudo update-rc.d -f php-fpm defaults

$ sudo update-rc.d -f nginx defaults

$ sudo service php-fpm start

$ sudo service nginx start

Installation is now complete. At this point, PHP should run under fastcgi mode and nginx has taken over the web server role from Apache to serve ClusterControl UI. We can verify that with any web server detector extension on your preferred web browser:

Caveats

• Severalnines’s s9s_error_reporter might not get a complete error report on ClusterControl UI since it doesn’t collect any nginx related log files.
• ClusterControl is built on a common Apache configuration. There might be some features that do not function well (although we have not encountered any malfunctions so far).
• If you want to install ClusterControl manually on nginx (without using ClusterControl installer script), we recommend users to follow the Manual Installation documentation and install ClusterControl on Apache first. Then, follow the steps under "Switching from Apache to nginx" section to run on nginx.

We’re excited to announce the 1.7.5 release of ClusterControl - the only database management system you’ll ever need to take control of your open source database infrastructure. 

This new version features support for the latest MongoDB& PostgreSQL general releases as well as new operating system support allowing you to install ClusterControl on Centos 8 and Debian 10.

ClusterControl 1.7.4 provided the ability to place a node into Maintenance Mode. 1.7.5 now allows you to place (or schedule) the entire database cluster in Maintenance Mode, giving you more control over your database operations.

In addition, we are excited to announce a brand new function in ClusterControl we call “Freeze Frame.” This new feature will take snapshots of your MySQL or MariaDB setups right before a detected failure, providing you with invaluable troubleshooting information about what caused the issue. 

Release Highlights

Database Cluster-Wide Maintenance

• Perform tasks in Maintenance-Mode across the entire database cluster.
• Enable/disable cluster-wide maintenance mode with a cron-based scheduler.
• Enable/disable recurring jobs such as cluster or node recovery with automatic maintenance mode.

MySQL Freeze Frame (BETA)

• Snapshot MySQL status before cluster failure.
• Snapshot MySQL process list before cluster failure (coming soon).
• Inspect cluster incidents in operational reports or from the s9s command line tool.

New Operating System & Database Support

• Centos 8 and Debian 10 support.
• PostgreSQL 12 support.
• MongoDB 4.2 and Percona MongoDB v4.0 support.

Additional Misc Improvements

• Synchronize time range selection between the Overview and Node pages.
• Improvements to the nodes status updates to be more accurate and with less delay.
• Enable/Disable Cluster and Node recovery are now regular CMON jobs.
• Topology view for Cluster-to-Cluster Replication.

View Release Details and Resources

• Download ClusterControl
• ChangeLog
• ClusterControl Administration Guide
• Upgrade Instructions

Release Details

Cluster-Wide Maintenance 

The ability to place a database node into Maintenance Mode was implemented in the last version of ClusterControl (1.7.4). In this release we now offer the ability to place your entire database cluster into Maintenance Mode to allow you to perform updates, patches, and more.

MySQL & MariaDB Freeze Frame

This new ClusterControl feature allows you to get a snapshot of your MySQL statuses and related processes immediately before a failure is detected. This allows you to better understand what happened when troubleshooting, and provide you with actionable information on how you can prevent this type of failure from happening in the future. 

This new feature is not part of the auto-recovery features in ClusterControl. Should your database cluster go down those functions will still perform to attempt to get you back online; it’s just that now you’ll have a better idea of what caused it. 

Support for PostgreSQL 12

Released in October 2019, PostgreSQL 12 featured major improvements to indexing, partitioning, new SQL & JSON functions, and improved security features, mainly around authentication. ClusterControl now allows you to deploy a preconfigured Postgres 12 database cluster with the ability to fully monitor and manage it.

Support for MongoDB 4.2

MongoDB 4.2 offers unique improvements such as new ACID transaction guarantees, new query and analytics functions including new charts for rich data visualizations. ClusterControl now allows you to deploy a preconfigured MongoDB 4.2 or Percona Server for MongoDB 4.2 ReplicaSet with the ability to fully monitor and manage it.

Database security requires careful planning, but it is important to remember that security is not a state, it is a process. Once the database is in place, monitoring, alerting and reporting on changes are an integral part of the ongoing management. Also, security efforts need to be aligned with business needs.

Database vendors regularly issue critical patch updates to address software bugs or known vulnerabilities, but for a variety of reasons, organizations are often unable to install them in a timely manner, if at all. Evidence suggests that companies are actually getting worse at patching databases, with an increased number violating compliance standards and governance policies. Patching that requires database downtime would be of extreme concern in a 24/7 environment, however, most cluster upgrades can be performed online. 

ClusterControl is able to perform a rolling upgrade of a distributed environment, upgrading and restarting one node at a time. The logical upgrade steps might slightly differ between the different cluster types. Load balancers would automatically blacklist unavailable nodes that are currently being upgraded, so that applications are not affected. 

Operational Reporting on Version Upgrades and Patches is an area that requires constant attention, especially with the proliferation of open source databases in many organizations and more database environments being distributed for high availability.

ClusterControl provides a solid operational reporting framework and can help answer simple questions like 

• What versions of the software are running across the environment?
• Which servers should be upgraded?
• Which servers are missing critical updates?

Automatic Database Patching

ClusterControl provides the ability for automatic rolling upgrades for MySQL& MariaDB to ensure that your databases always use the latest patches and fixes. 

Upgrades are online and are performed on one node at a time. The node will be stopped, then software will be updated, and then the node will be started again. If a node fails to upgrade, the upgrade process is aborted. 

Rolling MySQL Database Upgrades

ClusterControl provides the ability for automatic rolling upgrades for MySQL-based database clusters by automatically applying the upgrade one node at a time which results in zero downtime.

After successfully installing the selected version you must perform a rolling restart - the nodes restart one by one.

ClusterControl supports you in that step making sure nodes are responding properly during the node restart.

Database Upgrade Assistance

ClusterControl makes it easy to upgrade your MongoDB and PostgreSQL databases by, with a simple click, promoting a slave or replica to allow you to upgrade the Master and vice versa.

Database Package Summary Operational Report

ClusterControl provides the Package Summary Operational Report that shows you how many technology and security patches are available to upgrade.

You can generate it ad-hoc and view in the UI, send it via email or you can schedule such a report to be delivered to you for example once per week.

As you can see, the Upgrade Report contains information about different hosts in the cluster, which database has been installed on them and in which version. It also contains information about how many other packages installed are not up to date. You can see the total number, how many are related to database services, how many are providing security updates and the rest of them. 

The Upgrade Report lists all of the not-up-to-date packages on a per-host basis. In the screenshot above you can see that the node 10.0.3.10 has two MongoDB util packages not up to date (those are the 2 DB packages mentioned in the summary). Then there is a list of security packages and all other packages which are not up to date.

Conclusion

ClusterControl goes an extra mile to make sure you are covered regarding the security (and other) updates. As you have seen, it is very easy to know if your systems are up to date. ClusterControl can also assist in performing the upgrade of the database nodes.

Backups are the means of protecting from data loss - should something happen, you can easily restore it from the backup. You cannot predict what part of the data will have to be restored - it can be everything or just a subset. Typically you want to have a full backup to ensure you can handle a total data loss scenario but what would happen if only a single table had been dropped? Can we do a partial restore if we used Xtrabackup to create our safety data copy? Let’s explore this scenario in a short blog post.

Partial Restore Using Xtrabackup

The main thing you have to keep in mind before you perform a partial restore with Xtrabackup is that this will break the consistency of the node where you would restore the backup. This is extremely important in replication or Galera setups where the consistency of the cluster is paramount as otherwise replication (standard or Galera) may break. 

How to approach this problem? It all depends on your environment. One of the solutions could be to use a separate host to restore missing data and then proceed with regular logical backup, something that you can restore on the live cluster without introducing data inconsistency. 

Alternatively, if you can afford to stop the whole cluster, you can perform the restore on all of the nodes in the cluster - this as well will result in a consistent state of the data across the whole environment. We won’t go into details how to proceed because, as we stated, this may depend on your business requirements, ability to schedule a downtime and so on. 

For now let’s take a look how to restore a single table, not focusing where you would do that.

We are assuming that a full backup created by Xtrabackup is ready. We have a simple environment of asynchronous replication with one master and one slave. We use Percona Server 8.0 therefore we ensured we have percona-xtrabackup-80 installed.

As can be seen, the backup has been created:

root@vagrant:~# ls -alh /backup/

total 149M

drwxr-xr-x  6 root root 4.0K Mar 13 12:24 .

drwxr-xr-x 25 root root 4.0K Mar 13 12:23 ..

-rw-r-----  1 root root 479 Mar 13 12:24 backup-my.cnf

-rw-r-----  1 root root 195 Mar 13 12:24 binlog.000005

-rw-r-----  1 root root   16 Mar 13 12:24 binlog.index

-rw-r-----  1 root root 5.8K Mar 13 12:24 ib_buffer_pool

-rw-r-----  1 root root 100M Mar 13 12:24 ibdata1

drwxr-x---  2 root root 4.0K Mar 13 12:24 mysql

-rw-r-----  1 root root 24M Mar 13 12:24 mysql.ibd

drwxr-x---  2 root root 4.0K Mar 13 12:24 performance_schema

drwxr-x---  2 root root 4.0K Mar 13 12:24 sbtest

drwxr-x---  2 root root 4.0K Mar 13 12:24 sys

-rw-r-----  1 root root 12M Mar 13 12:24 undo_001

-rw-r-----  1 root root 12M Mar 13 12:24 undo_002

-rw-r-----  1 root root   63 Mar 13 12:24 xtrabackup_binlog_info

-rw-r-----  1 root root   99 Mar 13 12:24 xtrabackup_checkpoints

-rw-r-----  1 root root 540 Mar 13 12:24 xtrabackup_info

-rw-r-----  1 root root 8.5K Mar 13 12:24 xtrabackup_logfile

-rw-r-----  1 root root 248 Mar 13 12:24 xtrabackup_tablespaces

Now, if we want to restore it, we have to prepare the backup - it’s a standard process for Xtrabackup. There is one major difference though in a way we will prepare it. We will use --export flag:

root@vagrant:~# xtrabackup --prepare --export --target-dir=/backup/

Now we can restore a particular table following this process:

1. We have to create the table using exactly the same schema as it used to have when the backup has been taken.
2. We have to discard its tablespace
3. We will copy the tablespace from the backup along with its *.cfg file
4. We will import new tablespace

Let’s assume one of the tables has been accidentally truncated:

mysql> SELECT COUNT(*) FROM sbtest.sbtest11\G

*************************** 1. row ***************************

COUNT(*): 0

1 row in set (0.00 sec)

In this case we already have the table with a proper schema in place and we can proceed to step 2):

mysql> ALTER TABLE sbtest.sbtest11 DISCARD TABLESPACE;

Query OK, 0 rows affected (0.02 sec)

Now we have to copy the data from the backup:

root@vagrant:~# cp /backup/sbtest/sbtest11.* /var/lib/mysql/sbtest/

root@vagrant:~# chown mysql.mysql /var/lib/mysql/sbtest/sbtest11.*

Finally, we can import the restored tablespace:

mysql> ALTER TABLE sbtest.sbtest11 IMPORT TABLESPACE;

Query OK, 0 rows affected (0.48 sec)



mysql> SELECT COUNT(*) FROM sbtest.sbtest11\G

*************************** 1. row ***************************

COUNT(*): 100000

1 row in set (0.05 sec)

As you can see, the contents of the table have been restored. Now, based on how we approached the whole problem, we can either repeat this process on all of the nodes in the cluster or we can use mysqldump or SELECT … INTO OUTFILE to extract this data and then load it on the live cluster.

Please keep in mind that Xtrabackup allows as well to take a backup of a single database or single table. This is another feature, loosely tied to what we have just discussed - it is not required to create a backup of a single table to be able to restore it. What is required though is the schema - you may want to schedule backups of the schema (no data is required) using mysqldump that will go along with your xtrabackup backups. You may find them very handy if your schema changes often.

How to Restore a Single Table Using ClusterControl?

ClusterControl, as of now, does not come with an ability to restore a single table out of full backup. You can schedule partial backups with ClusterControl though. Then you can use those backups and restore them on a separate host and then extract the data and apply it on the live cluster.

As you can see on the screenshot, you can decide which database you want to backup and then list the tables (or decide that you want to include all of them) you would like to backup. You can setup a backup schedule where you would backup individual tables, one at a time. You could as well design the schedule on a schema-per-schema basis. Once you have a backup ready, you can restore it on a standalone host:

Then, we will have to decide what host it is. You also have to make sure this host can be reached from ClusterControl node using SSH.

We want ClusterControl to setup software, provision it with data and then keep the server running after the backup has been restored.

We should review the options we took and then confirm that the backup should be restored.

Job has been started and all we need to do is to wait for it to complete.

Once the job has completed, you can access the backup verification server, dump the missing data and restore it on the live cluster.

Using a Load Balancer is a good idea for any database technology, as you can redirect applications to the available or healthy database nodes and even distribute the traffic across multiple servers to improve performance. This is not only useful on-prem but also in a cloud environment. In this blog, we’ll see how to deploy and configure a new database cluster with HAProxy on the Google Cloud Platform from scratch.

Creating the VM on Google Cloud

For this example, we’ll assume that you have a Google Cloud account created.

You can deploy your virtual machines directly from ClusterControl. Go to the deploy section and select “Deploy in the Cloud”.

Specify vendor and version for your new cluster.

Add the number of nodes, cluster name, and database information.

Choose the cloud credentials, in this case, your Google Cloud account. If you don’t have your account added in ClusterControl, you can follow our documentation for this task.

Now you can specify the virtual machine configuration, like operating system, size, and region.

ClusterControl will create the virtual machines, install the software, and configure it, all in the same job and in an unattended way.

You can monitor the creation process in the ClusterControl activity section. When it finishes, you will see your new cluster in the ClusterControl main screen.

Deploying HAProxy in Google Cloud

Note: To deploy it, first, you need to create the VM in the Google Cloud Platform as the virtual machine creation is not implemented for the ClusterControl load balancer deployment yet (it will be available soon).

Now you have your new cluster up and running, go to ClusterControl -> Select Cluster -> Cluster Actions -> Add Load Balancer.

Here you must add the information that ClusterControl will use to install and configure your HAProxy load balancer.

The information that you need to introduce is:

Action: Deploy or Import.

Server Address: IP Address for your HAProxy server.

Listen Port (Read/Write): Port for read/write mode.

Listen Port (Read-Only): Port for read-only mode.

Policy: It can be:

• leastconn: The server with the lowest number of connections receives the connection.
• roundrobin: Each server is used in turns, according to their weights.
• source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request.

Install for read/write splitting: For master-slave replication.

Build from Source: You can choose Install from a package manager or build from source.

And you need to select which servers you want to add to the HAProxy configuration and some additional information like:

Role: It can be Active or Backup.

Include: Yes or No.

Connection address information.

Also, you can configure Advanced Settings like Admin User, Backend Name, Timeouts, and more.

When you finish the configuration and confirm the deployment, you can follow the progress in the Activity section on the ClusterControl UI.

And when this finishes, you can go to ClusterControl -> Nodes -> HAProxy node, and check the current status.

You can also monitor your HAProxy servers from ClusterControl checking the Dashboard section.

Conclusion

A Load Balancer can help you to handle your database traffic by balancing it between multiple servers. It is also useful to improve your high availability environment by performing failover tasks. ClusterControl can help you too with different features like auto-recovery, monitoring, deployment, and even more, and it can manage on-prem, cloud or mixed environments with different database technologies at the same time.

The day happened when your database administrator did not show up on the daily online standup. Shortly after you have learned that he’ll be unavailable for an unknown period of time and it is you, who will have to step in and replace him. In a series of posts we would like to explain what are the most important areas you may have to take care of while your colleague will be off. Let’s take a look at security, one of the most important pieces of puzzle in a database environment.

What’s good, most likely you are pretty much covered. Your colleague should have secured the environment and, without human intervention, the environment should stay secure. Here’s a list of things you would like to check.

Checking Passwords

It’s quite important to ensure that all of the users have a proper password defined. You do that in a couple of ways. First, you can just check in the database. For MySQL you are looking for a query like this:

mysql> SELECT host, user FROM mysql.user where authentication_string='';

+------+-----------+

| host | user      |

+------+-----------+

| %    | emptypass |

+------+-----------+

1 row in set (0.00 sec)

For PostgreSQL you may want to review your pg_hba.conf file and verify that all of the entries have an ‘md5’ authentication method.

It is quite important to ensure that the access to your database is secured.

Checking Access

All existing servers should have already been secured but it might be that you will be asked to provision new instances. If you can reuse existing scripts, playbooks or other automation tools, you would probably be good. If you’d have to provision a new server by hand, though, please keep in mind that the access to such a server should be secured. Depending on the policies in your company this may mean different things but here’s a list of steps you can take to protect your database:

1. Do not bind to a public IP. Your database should not be exposed to the open world. Always use a private IP to bind to.
2. Make sure you have a proper network security in place. Firewalls, ACL’s etc - block everything, allow access only to the ports you have to access.
3. Do not use default, “official” ports. Instead of running SSH on port 22, run it on 2345. Instead of running your MySQL database on port 3306 use 3333 or any other port that you have available. It is security through obscurity, sure, but you’ll be amazed how far it can get you.

Security Patches

In a production environment you should do your best to stay on top of the security updates. Please keep in mind that it is not only a matter of database security fixes but also other packages that are installed in the system. Any service that runs on an exposed port has to be kept up to date because, if exploited, it could become an entry point to your infrastructure. If you do not have an automated patching process in place, spend some time and try to stay up to date with any security updates your distribution of choice may be publishing.

How ClusterControl Can Help With the Security of Your Database?

ClusterControl comes with a couple of features that may help you ensure your database is secured. First of all, it comes with user management. You can create new or edit existing users.

This can be used to change insecure passwords or create new passwords if the user does not have a password defined. You can also quickly check which users do have extensive privileges and, if needed, reduce them to the minimum required to perform their tasks.

Additional feature is the list of inactive users.

ClusterControl can prepare a list of accounts that have not been used to access MySQL database since its restart. This may help you to understand which accounts are inactive and, as a result, remove them to reduce the potential access paths to your database.

Another very useful feature that will help you to stay on top of the security updates are the Operational Reports. It comes with, among others, a Package Upgrade Report.

Such reports can be scheduled and executed on a regular basis. It may be delivered to one or more recipients via email. 

It creates a list of packages installed on the system, checks if the package is in the latest version and if not, you will have a notification in the report. Executing such reports on a regular basis helps to stay on top of the upgrades and keep your environment secure.

In case you would notice that the database packages have available updates, you can use another feature of ClusterControl to perform a minor version upgrade:

With a couple of clicks you can easily perform an upgrade of your MySQL or MariaDB database to the latest version, ensuring that you are protected from known security vulnerabilities.

As you can see, keeping the system secure requires several actions to be taken and it does require regular checks to stay on top of the updates. With database management platforms like ClusterControl, it becomes much easier to accomplish.

The alarming nature of COVID-19 is being felt globally. Many companies have closed or filed bankruptcy because of the global pandemic. 

Most organizations did not see this coming. Large companies have a better chance of being well prepared for this type of scenario for which a Disaster Recovery Plan (DRP) has been laid-out. At Severalnines, we advocate the best practices consisting of conventional approaches when dealing with crisis, especially in securing your database technology. 

When things go south and nothing has been prepared (or worse, discussed but never implemented) the end result can be a hard blow to the company. These types of scenarios can’t be avoided, only planned for.

Today’s blog will walk you through some things to consider in your organization in regards to process planning and database backups for these types of scenarios.

Always Document The Work

Whenever your engineers start working on projects, it's best to always document the work such as the layout, project specification, requirements, and it's underlying procedures required to run the playbooks/runbooks. The documentation must also cover a proper escalation when things go wrong and what actions are required to be taken.

It can also be ideal to host these documents accessible via a centralized environment or intranet (accessed via connecting to VPN if done remotely), or over the internet as long as it is stored securely and can be accessed with multiple layers such as using 2FA (Two-Factor Authentication) or MFA (Multi-Factor Authentication). It's purpose is to have these documents accessible wherever the engineer is located. So when specific actions have to be done remotely, it can be done and not to physically access this via the office. Otherwise, it can lead to difficulties and inconvenience to the engineer's perspective. 

No Documentation, No Wiki - What Shall I Do?

This scenario is a pure pain and is most likely being experienced at companies around the world right now. When things go awry, it's best to start investigating the tools being used and lay out the procedures that have to be done. Worse case scenario, you might end up re-creating the whole thing or some part of the specific actions that are led to unresolved investigation will be reset. 

If this happens it may make sense to reach out for some temporary database support. This way you will have experts in the technology available to talk to.

Best case for this scenario is to leverage on specific technologies that offer a full-stack solution which can offer you backup solutions, cluster management, monitoring tools, and support. For example, here at Severalnines, we recommend that you try our software ClusterControl.

Try to Automate the Work

Writing scripts for automation is ideal. You can leverage automation tools to manage things for you, such as Chef, Puppet, Ansible, Salt, or Terraform. In our previous blogs, we shared some how-to's you can use for these automation tools. Take a look at How to Automate Daily DevOps Database Tasks with Chef or Database Automation with Puppet: Deploying MySQL & MariaDB Galera Cluster for more info. 

Automating the workload can relieve situations during a DBA. As stated earlier, it's best that procedures are written in a knowledge-based platform which is accessible anywhere.

Although creating this automation can be difficult for some organizations as a limited number of engineers can handle this type of work. Your best bet is to leverage tools or solutions that offer automation without high levels of technical complexity. 

For example, if your database is hosted in AWS or are hosted in a managed services platform, there are big advantages. Your team can deploy a backup schedule, set the backup policy, and set its retention; then just hit submit. Whenever your database crashes (for example in AWS Aurora) it will be handled in the background with autorecovery dispatched without the end users knowing that something happened. From a  business perspective, the end result is business-as-usual and no downtime occurring.

Although that's a great option to take, it might not be an appropriate choice for your technology stack. There are certain solutions that can offer you automation to handle backups for you. For example, you can take advantage of Backup Ninja, or even use ClusterControl to manage backup and restore. You can also set schedules for your backup with support for various open-source databases.

Keep Documentation Up-To-Date

In my experience, this is a problem that you will most likely encounter when things go south. When changes have been made, but go undocumented, then things can go wrong when performing vital tasks. This can get you into a dilemma, because the result comes out different than what was documented. 

This is scary in situations like when you are in a deadlock mode (where you're in the middle of performing a backup, causing your production to be locked up). Afterwards, you are then informed (after a day) that the process has been changed, but the lack of documentation by the DBA has left damage to your database nodes and caused problems.

When it comes to changes and documentation process, there's a huge advantage when it comes to availing third party solutions. These third-party companies know how to handle the business and always kept the process in-line so with changes to the software. 

For example, in ClusterControl (MySQL database systems) you can choose a backup method using mysqldump or using Percona Xtrabackup/Mariabackup (for MariaDB). Customers that are unsure of what to choose have the liberty to contact support to discuss. 

ClusterControl is also a good example here as it extends as your virtual DBA that mostly does things that your company’s DBA can do. There are certain actions that only DBA's can understand or oversee the result, but having an extended DBA that you are sure of 24/7 is available is a perfect route to avoid business from being impacted.

Using Third Party Applications

There are multiple options to choose from nowadays. These types of scenarios are not new, especially to companies who have invested in security and assurance for their business to continue to flourish and avoid huge downtimes due to lack of skills or engineers that can do the specific task. 

Relying on third party applications that offer managed databases (including backup solutions) or availing SaaS to handle backup automation and restoration are very advantageous in these types of scenarios. This comes up with a price, though, so you should really avail yourself on what offers the functions that your company requires.

Using a fully-managed database service such as Amazon RDS helps you easily do this with ease. Although the backups are stored as snapshots, yet if you need more flexibility such as restoring a particular table is not supported. It is doable, but requires knowledge and skills on what tools or particular commands need to be invoked.

In the case of ClusterControl, you have easy-to-use options when managing your backups. System administrators won't have to worry about the technology and commands to use as it's already managed by the software. For example, take a look at the screenshot below,

you can create backup, or schedule backup by through user clicks.

You can also view the scheduled backups with the list of backups that had proceed successfully,

These are all done with a couple of clicks with no scripting or extra engineering work that has to be done. Backups can also be verified automatically, restored with PITR support, uploaded to the cloud, compressed, and encrypted to comply with security and regulatory requirements. 

Not only do these features allow you to manage backups with ease, it also enables you to be notified real-time with various integrations supported, either through e-mail or by using third-party notifications such as Slack, PagerDuty, Servicenow, and a lot more.

Conclusion

These rough times we are experiencing shows that many businesses need to change how to approach the possibility of your technical resources not being available to do their jobs. This elevates how seriously to consider Disaster Recovery Planning and how well prepared your organization is when a crisis hits.

Database monitoring is a key function for Database Administrators (DBA). With a proper monitoring platform, you can know immediately if there is something wrong with the system. 

Should your DBA be unavailable and you may need to take over the platform to monitor and handle non-critical issues. In this blog are tips that you need to know if you are monitoring your database with tips for troubleshooting any issues.

Database Monitoring Documentation

It is always good to have proper documentation about Database Monitoring. This information should contain information like what tool is being used, what metrics database are being monitored, and what operating system metrics are being monitored. DBAs must prepare the documentation with restricted access to the document. The document must describe information how-to access the database and provide credentials to access the monitoring tools.

How ClusterControl Monitoring Tools Can Help

There are various monitoring tools for databases, providing metrics related to the operating system; database memory, threads, locking, session, slow query. ClusterControl provides a great monitoring tool for your database that is free to use. You can enable agent-based Monitoring in your ClusterControl as shown below 

You just need to click agent-based monitoring in the dashboard tab, it will install prometheus, and the exporter for database and the operating system. After installation, it will appear the monitoring tool in your dashboard as shown below

The System Overview will show you basic monitoring usage for operating systems such as CPU Usage, Disk Space Usage and Memory Usage. On the CPU Usage, you will see utilization for User, Idle, System, Iowait, while in the Memory Usage it is monitoring related to Total Memory, Available, and Used. Disk Space usage monitored the disk space utilization on each device. Beside the System Overview, there are also MySQL Server Overview which have information about database status and utilization as shown below.

There is a lot of information shown in MySQL Server Overview such as MySQL Uptime, current running Queries, Connection Used, and Bufferpool Hit Ratio. There is a Handler Stats metric to show your current handler for read first, read last, read next, delete, update and insert. DB Connection monitors your current connection, from aborted connections, aborted clients, max connections, thread connected, to threads connected. MySQL InnoDB Metrics show you current InnoDB status information as shown below :

You can see the InnoDB BufferPool Activity from Created, Read, and Written. InnoDB Bufferpool Hit Ratio, and InnoDB Bufferpool Usage from Dirty page, Read from Disk, Bufferpool Size. 

Worse Case Scenario 

The worst case scenario if DBA does not implemented the Database Monitoring Tool yet, as a SysAdmin, you can go through command line with some utility tools in the Operating System,  as shown below :

• CPU Utilization
  • You can use top or htop command to see the running process in the operating system that eats your CPU.
• Memory Utilization
  • vmstat can help you check your memory utilization. Also, you can check your current memory usage using free command,  it will show you the swap utilization (if it was used), total memory, used memory, and free memory.
• Disk Utilization
  • iostat command can show you the read and write rate to the disk per second. While iotop shows you the percentage of IO Utilization within the services.

If you have access to the database with superuser privileges, you can do the basic monitoring for database with below command :

• show [full] processlist, it will show you the thread with running operation in database, user connected, ip address, database, and Time.
• show engine innodb status \G, is a command to monitor your current InnoDB information from row operations, buffer pool and memory usage, IO thread usage, transaction and locks.

Conclusion

Having proper documentation for databases and monitoring tools is a must for a DBA. It will be useful and helpful for others, like the SysAdmins when they need to run an investigation while the DBA is unavailable. 

SysAdmins also needs to understand a basic troubleshooting process that enables them to troubleshoot some issues and elaborate the root cause of issue when escalating it to the related party, ie. DBA.

A bastion host is a gateway host between an inside network and an outside network. It is commonly used to access remote hosts sitting on a different network which has no direct connection between the two endpoints. Bastion host acts as a middle-man to connect both ends, thus making it a "jump" host to access to the other side. This is one of the popular ways to secure the server from being exposed to the outside world.

In this blog post, we are going to look at how simple it is to set up ClusterControl as a bastion host to restrict remote SSH access to our database and proxy servers. Our architecture for this setup is looking like this:

In this setup, we have two database servers with a load balancer server deployed and managed by ClusterControl and another host acts as a backup bastion host. The bastion hosts sit in between the end-user and the internal network hosting our production database tier.

Deploy or Import a Database Into ClusterControl

The very first step is to install ClusterControl on a host that is not part of your database server or cluster, in this case is cc.mysuperdomain.com. Then setup a passwordless SSH from ClusterControl server to all databases and load balancer hosts. In this example, we have an OS user called "vagrant" on all hosts which also has sudo privileges. Thus, on ClusterControl server:

$ whoami
vagrant
$ ssh-keygen -t rsa # press enter on all prompts
$ cat /home/vagrant/.ssh/id_rsa.pub

Copy the public key entry as shown in the last command above and paste it into /home/vagrant/.ssh/authorized_keys on all other hosts that we want to monitor. If the target hosts support password authentication, there is a simpler way by using the following command:

$ ssh-copy-id vagrant@192.168.111.20
$ ssh-copy-id vagrant@192.168.111.21
$ ssh-copy-id vagrant@192.168.111.22

Basically, we are authorizing the above hosts with that particular SSH key so the source host can access to the destination hosts without password, where the public key of the source host is in the allowed list in authrorized_keys file of the destination servers. Once done, open ClusterControl UI in a browser and go to "Deploy" to deploy a new database server/cluster or "Import" to import an existing server/cluster into ClusterControl. Under the "General & SSH Settings" section, specify the SSH user as "vagrant" with SSH Key Path "/home/vagrant/.ssh/id_rsa.pub", similar to the following screenshot:

Since ClusterControl requires a root or sudo user of the database hosts (as shown above), it surely can be used as a bastion host for SSH service to access the database and load balancer tiers from the external network. We can then close down all unnecessary communication from the outside world and make our production environment more secure.

Web-based SSH

Once the database server/cluster is deployed, we can use the ClusterControl SSH module to access the monitored host by going to ClusterControl -> Nodes -> pick a node -> Node Actions -> SSH Console. A new browser window will be popped up like below:

ClusterControl web-based SSH console is an extension module which works only on Apache 2.4 and later with proxy_module and WebSocket API. It comes as dependencies with the ClusterControl UI package and is enabled by default. ClusterControl uses the same SSH user, authenticated via passwordless SSH that is required when you first deploy or import the database server/cluster into ClusterControl.

The web SSH console mimics common terminal experience with popular emulator tools in the market. You can perform all Linux commands without character escaping, using the standard copy and paste methods (Ctrl+C/Ctrl+V) and the responses are in real time. You can open as many windows as you want and each of them will be established as a new SSH session that originates from the same IP address, which is the ClusterControl host address. The following output shows the active users for the current session if we have opened 3 web SSH windows:

Last login: Thu Apr  9 05:44:11 2020 from 192.168.0.19
[vagrant@proxy2 ~]$ w
 05:44:21 up  2:56, 3 users,  load average: 0.05, 0.05, 0.05
USER     TTY   FROM             LOGIN@ IDLE   JCPU  PCPU  WHAT
vagrant  pts/0 192.168.0.19     05:29  1:17   0.03s 0.03s -bash
vagrant  pts/1 192.168.0.19     05:44  17.00s 0.02s 0.02s -bash
vagrant  pts/2 192.168.0.19     05:44  2.00s  0.02s 0.01s w

To close the active SSH connection, type "exit" or "logout". Closing the web browser directly will NOT close the session which makes the connection stay idle. You may need to kill the connection manually from another session or wait until it reaches idle connection timeout.

Due to concern of security over this feature, this module can be disabled by setting the following constant inside /var/www/html/clustercontrol/bootstrap.php to "false", as shown below:

define('SSH_ENABLED', false);

Note that disabling the web SSH feature will not disable the current active SSH connection to the host until the window is closed. We encourage you to only use this feature if necessary, for example, in this case where ClusterControl is the bastion host to the database and load balancer tiers.

SSH Proxying via Bastion Host

Alternatively, we can use SSH ProxyCommand to relay the SSH communication to the end servers sitting behind the bastion host. Otherwise, we have to jump twice - one from client to the bastion host and another jump from the bastion host to the database host in the private network. 

To simplify SSH communication from our workstation to the end servers behind the bastion host, we can make use of SSH client configuration. Create a default SSH client configuration file at ~/.ssh/config and define them all like this:

Host cc-bastion
  Hostname cc.mysuperdomain.com
  User vagrant
  IdentityFile /Users/mypc/.ssh/mykey.pem

Host db1-prod
  Hostname 192.168.111.21
  User vagrant
  IdentityFile /Users/mypc/.ssh/bastion.pem
  ProxyCommand ssh -q -A cc-bastion -W %h:%p

Host db2-prod
  Hostname 192.168.111.22
  User vagrant
  IdentityFile /Users/mypc/.ssh/bastion.pem
  ProxyCommand ssh -q -A cc-bastion -W %h:%p

Host lb1-prod
  Hostname 192.168.111.20
  User vagrant
  IdentityFile /Users/mypc/.ssh/bastion.pem
  ProxyCommand ssh -q -A cc-bastion -W %h:%p

The SSH private key "bastion.pem" can be created by copying the content of /home/vagrant/.ssh/id_rsa on the ClusterControl server (since this key is already allowed in all database and load balancer servers). Login into ClusterControl server and run:

(bastion-host)$ cp /home/vagrant/.ssh/id_rsa bastion.pem

Remote copy bastion.pem from ClusterControl host back to your workstation:

(workstation)$ scp cc.mysuperdomain.com:~/bastion.pem /Users/mypc/.ssh/

For mykey.pem, you can create the private key manually inside your workstation:

(workstation)$ ssh-keygen -t rsa -f /Users/mypc/.ssh/mykey.pem
(workstation)$ cat /Users/mypc/.ssh/mykey.pem.pub

From the "cat" output above, add the entry into /home/vagrant/.ssh/authorized_keys on ClusterControl server (or use ssh-copy-id command as explained before). Note that on every section of the production servers, there is a ProxyCommand entry to relay our communication via bastion host to the servers in the private network. 

From our workstation, we can simply use the following command:

(workstation)$ ssh db1-prod
Last login: Thu Apr  9 09:30:40 2020 from 192.168.0.101
[vagrant@db1: ~]$ hostname
db1.production.local

At this point you are connected to db1-prod via bastion host, cc1.mysuperdomain.com.

Security Configurations

Now, we have verified our database servers can be accessed from ClusterControl UI. It's time to restrict the access to the bastion hosts, ClusterControl server (192.168.0.19), and also a backup host (192.168.0.20), in case the ClusterControl server becomes unreachable. We can achieve this by using two methods:

• TCP wrappers.
• Firewalls (pf, iptables, firewalld, ufw, csf).

TCP Wrappers (hosts.allow/hosts.deny)

TCP wrappers protect specific software using hosts.allow and hosts.deny files. For example you could use it to prevent people from connecting by telnet except from specific permitted addresses. TCP wrapper works as in the following order:

1. The /etc/hosts.allow file is read first - from top to bottom. 
2. If a daemon-client pair matches the first line in the file, access is granted. 
3. If the line is not a match, the next line is read and the same check is performed. 
4. If all lines are read and no match occurs, the /etc/hosts.deny file is read, starting at the top. 
5. If a daemon-client pair match is found in the deny file, access is denied. 
6. If no rules for the daemon-client pair are found in either file, or if neither file exists, access to the service is granted.

Thus if we want to allow only ClusterControl (192.168.0.19) and a backup bastion host (192.168.0.20) to the server, we would need to add the following lines into /etc/hosts.allow on every database host:

sshd : 192.168.0.19,192.168.0.20

And add the following lien inside /etc/hosts.deny to deny other hosts apart from the above:

sshd : ALL

When connecting from a host that is not allowed, one would see the following error:

[vagrant@host1 ~]$ hostname -I
192.168.0.211
[vagrant@host1 ~]$ ssh 192.168.0.23
ssh_exchange_identification: read: Connection reset by peer

iptables

Contrary to TCP wrappers, firewalls treat all software the same. There are basically two popular networking filters in the UNIX world called PF (Packet Filter, popular in BSD) and Netfilter (use iptables as the frontend, popular in Linux). Since ClusterControl only supports Linux-based operating systems, we will use iptables to configure the firewall. There are many configuration tools for iptables like firewalld, ufw (Uncomplicated Firewall) and csf (ConfigServer Firewall) to simplify the management of access list, firewall rules and policy chains.

The following iptables commands can be used to allow the SSH connections only from the bastion hosts:

$ iptables -A INPUT -p tcp -s 192.168.0.19 --dport 22 -m comment --comment 'Allow bastion host to SSH port' -j ACCEPT
$ iptables -A INPUT -p tcp -s 192.168.0.20 --dport 22 -m comment --comment 'Allow bastion host to SSH port' -j ACCEPT
$ iptables -A INPUT -p tcp -s 0.0.0.0/0 --dport 22 -m comment --comment 'Drop everything on SSH apart from the above' -j DROP

To save the current active rules, run:

$ iptables-save > ~/ssh.rules

If you want to load a saved rules, simply do:

$ iptables-restore < ~/ssh.rules

Since iptables can also be used for other purposes like packet forwarding, packet altering, NAT, PAT, rate limiting and much more, it could be a bit complicated to maintain depending on how complex your security policy is. 

Conclusion

Using a bastion host (or jump host) can reduce the security threat of SSH service, which is one of the most critical services to manage Linux machines remotely.

We’re excited to announce the 1.7.6 release of ClusterControl - the only database management system you’ll ever need to take control of your open source database infrastructure. 

This new edition expands our commitment to cloud integration by allowing a user to deploy a SQL database stack to the cloud provider of your choice with the HAProxy load balancer pre-configured. This makes it even simpler and faster to get a highly available deployment of the most popular open source databases into the cloud with just a couple of clicks.

In addition to this new function we also have improved our new MySQL Freeze Frame system by adding the ability to snapshot the process list before a cluster failure.

Release Highlights

Simple Cloud Deployment of HA Database Stack with Integrated HAProxy

• Improvements to the cloud deployment GUI to allow deployment and configuration of HAProxy along with the database stack to the cloud provider of your choosing. 

MySQL Freeze Frame (BETA)

• Now snapshots the MySQL process list before a cluster failure.

Additional Misc Improvements

• CMON Upgrade operations are logged in a log file.
• Many improvements and fixes to PostgreSQL Backup, Restore, and Verify Backup. 
• A number of legacy ExtJS pages have been migrated to AngularJS.

View Release Details and Resources

• Download ClusterControl
• ChangeLog
• ClusterControl Administration Guide
• Upgrade Instructions

Release Details

Cloud Deployment of HA Database Stack with Integrated HAProxy 

In ClusterControl 1.6 we introduced the ability to directly deploy a database cluster to the cloud provider of your choosing. This made the deployment of highly available database stacks simpler than it had ever been before. Now with the new release we are adding the ability to deploy an HAProxy Load Balancer right alongside the database in a complete, pre-configured full stack.

Load balancers are an essential part of traffic management and performance, and you can now deploy a pre-integrated database/load balancer stack using our easy-to-use wizard.

PostgreSQL Improvements

Over the course of the last few months, we have been releasing several patches which culminated in the release of ClusterControl 1.7.6. You can review the changelog to see all of them. Here are some of the highlights...

• Addition of Read/Write Splitting for HAProxy for PostgreSQL
• Improvements to the Backup Verification process
• Improvements to the Restore & Recovery functions
• Several fixes and improvements regarding Point-in-Time Recovery
• Bug fixes regarding the Log & Configuration files
• Bug fixes regarding process monitoring & dashboards

With the current COVID-19 situation ongoing, plenty of people have started to work from home. Among those are people whose job is to manage database systems. The lockdowns which have been announced all over the world mean that kids are also staying at home too. Homeschooling is now a thing, in many cases it comes with some sort of online learning activities. This creates pressure on the resources available at home. Who should be using laptops? Moms and Dads working from home or their kids, for their online classes. People often experience an “every laptop and tablet counts” situation. How can you do your job while having only an iPad available? Can you manage your database system with its help? Let’s take a look at this problem.

Connectivity

The main issue to solve would most likely be connectivity.

If you can use one of the supported VPN methods, good for you. If not, you can search the App Store for additional VPN clients like . Hopefully you’ll be able to find something suitable for you like, for example, OpenVPN Connect.

One way or the other, as soon as you can connect to your VPN, you can start working. There are a couple of ways to approach it. One might be a traditional way involving SSH access. Technically speaking, a 13'’ iPad with a Smart Keyboard can be quite a nice replacement for a laptop. Still, for those smaller, 10’’ screens, you have to accept some compromises.

For connecting over SSH we used Terminus. Here’s how it looks.

With on-screen keyboard work space is quite limited. On the other hand, you can achieve everything you could have achieved using your laptop. It’s just more time consuming and more annoying.

In full screen mode it’s slightly better but the font is really small. Sure, you can increase its size:

But then you end up scrolling through the kilometers of text. Doable but far from comfortable. You can clearly see that managing databases in such a way is quite hard, especially if we are talking about emergency situations where you have to act quickly.

Luckily, there’s another approach where you can rely on the database management platform to help you in your tasks. ClusterControl is an example of such a solution.

We are not going to lie, as every UI, ClusterControl will work better on larger screens, but it still works quite well:

It can help you to deal with the regular tasks like monitoring the replication status.

You can scroll through the metrics and see if there are any problems with your database environment.

With just a couple of clicks you can perform management tasks that otherwise would require executing numerous CLI commands.

You can manage your backups, edit the backup schedule, create new backups, restore, verify. All with just a couple of clicks.

As you can see, an iPad might be quite a powerful tool in dealing with database management tasks. Even with the low screen estate, through using proper tools like ClusterControl, you can achieve almost the same outcome.

If you are new to PostgreSQL the most common challenge you face is about how to tune up your database environment. 

When PostgreSQL is installed it automatically produces a basic postgresql.conf file. This configuration file is normally kept inside the data directory depending on the operating system you are using. For example, in Ubuntu PostgreSQL places the configurations (pg_hba.conf, postgresql.conf, pg_ident.conf) inside /etc/postgresql directory. Before you can tune your PostgreSQL database, you first have to locate the postgresql.conf files. 

But what are the right settings to use? and what are the values set to initially? Using external tools such as PGTune (and alternative tools like ClusterControl) will help you solve this specific problem. 

What is PGTune?

PGTune is a configuration wizard which was originally created by Greg Smith from 2ndQuadrant. It's based on a Python script which is, unfortunately, no longer supported. (It does not support newer versions of PostgreSQL.) It then transitioned into pgtune.leopard.in.ua (which is based on the original PGTune) and is now a configuration wizard you can use for your PG database configuration settings.

PGTune is used to calculate configuration parameters for PostgreSQL based on the maximum performance for a given hardware configuration. It isn't a silver bullet though, as many settings depend not only on the hardware configuration, but also on the size of the database, the number of clients and the complexity of queries. 

How to Use PGTune

The old version of PGTune was based on python script which you can invoked via shell command (using Ubuntu):

root@debnode4:~/pgtune-master# $PWD/pgtune -L -T Mixed -i /etc/postgresql/9.1/main/postgresql.conf | sed -e '/#.*/d' | sed '/^$/N;/^\n/D' 

stats_temp_directory = '/var/run/postgresql/9.1-main.pg_stat_tmp'

datestyle = 'iso, mdy'

default_text_search_config = 'pg_catalog.english'

default_statistics_target = 100

maintenance_work_mem = 120MB

checkpoint_completion_target = 0.9

effective_cache_size = 1408MB

work_mem = 9MB

wal_buffers = 16MB

checkpoint_segments = 32

shared_buffers = 480MB

But the new one is much more easier and way convenient since you can just access via browser. Just go to https://pgtune.leopard.in.ua/. A good example is like below:

All you need to do is specify the following fields below:

• DB version - the version of your PostgreSQL. It supports versions of PostgreSQL from 9.2, 9.3, 9.4, 9.5, 9.6, 10, 11, and 12.
• OS Type- the type of OS (Linux, OS X, Windows)
• DB Type - the database type which is mainly what kind of transactional processing your database will handle (Web Application, OLTP, Data Warehousing, Desktop Application, Mixed Type of Applications)
• Total Memory (RAM) - The total memory that your PG instance will handle. Need to specify it in GiB.
• Number of CPUs- Number of CPUs, which PostgreSQL can use CPUs = threads per core * cores per socket * sockets
• Number of Connections - Maximum number of PostgreSQL client connections
• Data Storage- Type of data storage device which you can choose from SSD, HDD, or SAN based storage.

Then hit the Generate button. Alternatively, you can also run ALTER SYSTEM statement which generates postgresql.auto.conf, but it won't take until you hit a PostgreSQL restart.

How Does It Sets The Values

The algorithm for this tool can be basically found here in configuration.js. It does share the same algorithm from the old PGTune starting here pgtune#L477. For example, versions of PostgreSQL < 9.5 supports checkpoint_segments, but PG >= 9.5 uses the min_wal_size and max_wal_size. 

Setting the checkpoint_segments or min_wal_size/max_wal_size depends on what type of PostgreSQL version and the DB type of database application transaction. See how in the snippet below:

if (dbVersion < 9.5) {

  return [

    {

      key: 'checkpoint_segments',

      value: ({

        [DB_TYPE_WEB]: 32,

        [DB_TYPE_OLTP]: 64,

        [DB_TYPE_DW]: 128,

        [DB_TYPE_DESKTOP]: 3,

        [DB_TYPE_MIXED]: 32

      }[dbType])

    }

  ]

} else {

  return [

    {

      key: 'min_wal_size',

      value: ({

        [DB_TYPE_WEB]: (1024 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_OLTP]: (2048 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_DW]: (4096 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_DESKTOP]: (100 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_MIXED]: (1024 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB'])

      }[dbType])

    },

    {

      key: 'max_wal_size',

      value: ({

        [DB_TYPE_WEB]: (4096 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_OLTP]: (8192 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_DW]: (16384 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_DESKTOP]: (2048 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB']),

        [DB_TYPE_MIXED]: (4096 * SIZE_UNIT_MAP['MB'] / SIZE_UNIT_MAP['KB'])

      }[dbType])

    }

  ]

}

Just to explain short, it detects if dbVersion < 9.5, then it determines the suggested values for variables checkpoint_segments or min_wal_size/max_wal_size based on the type of dbType value set during the web UI form.

Basically, you can learn more about the algorithm on how it decides to suggest the values by looking at this script configuration.js.

PostgreSQL Configuration Tuning with ClusterControl

If you are using ClusterControl to create, build, or import a cluster, it automatically does an initial tuning based on the given hardware specs. For example, creating a cluster with the following job specs below,

{

  "command": "create_cluster",

  "group_id": 1,

  "group_name": "admins",

  "job_data": {

    "api_id": 1,

    "cluster_name": "pg_11",

    "cluster_type": "postgresql_single",

    "company_id": "1",

    "datadir": "/var/lib/postgresql/11/",

    "db_password": "dbapgadmin",

    "db_user": "dbapgadmin",

    "disable_firewall": true,

    "disable_selinux": true,

    "generate_token": true,

    "install_software": true,

    "nodes": [

      {

        "hostname": "192.168.30.40",

        "hostname_data": "192.168.30.40",

        "hostname_internal": "",

        "port": "5432"

      },

      {

        "hostname": "192.168.30.50",

        "hostname_data": "192.168.30.50",

        "hostname_internal": "",

        "port": "5432",

        "synchronous": false

      }

    ],

    "port": "5432",

    "ssh_keyfile": "/home/vagrant/.ssh/id_rsa",

    "ssh_port": "22",

    "ssh_user": "vagrant",

    "sudo_password": "",

    "user_id": 1,

    "vendor": "default",

    "version": "11"

  },

  "user_id": 1,

  "user_name": "paul@severalnines.com"

}

Provides me the following tuning as shown below:

[root@ccnode ~]# s9s job --log  --job-id 84919 | sed -n '/stat_statements/,/Writing/p'

192.168.30.40:5432: Enabling stat_statements plugin.

192.168.30.40:5432: Setting wal options.

192.168.30.40:5432: Performance tuning.

192.168.30.40: Detected memory: 1999MB.

192.168.30.40:5432: Selected workload type: mixed

Using the following fine-tuning options:

  checkpoint_completion_target: 0.9

  effective_cache_size: 1535985kB

  maintenance_work_mem: 127998kB

  max_connections: 100

  shared_buffers: 511995kB

  wal_keep_segments: 32

  work_mem: 10239kB

Writing file '192.168.30.40:/etc/postgresql/11/main/postgresql.conf'.

192.168.30.50:5432: Enabling stat_statements plugin.

192.168.30.50:5432: Setting wal options.

192.168.30.50:5432: Performance tuning.

192.168.30.50: Detected memory: 1999MB.

192.168.30.50:5432: Selected workload type: mixed

Using the following fine-tuning options:

  checkpoint_completion_target: 0.9

  effective_cache_size: 1535985kB

  maintenance_work_mem: 127998kB

  max_connections: 100

  shared_buffers: 511995kB

  wal_keep_segments: 32

  work_mem: 10239kB

Writing file '192.168.30.50:/etc/postgresql/11/main/postgresql.conf'.

Additionally, it also tunes up your system or kernel parameters such as,

192.168.30.50:5432: Tuning OS parameters.

192.168.30.50:5432: Setting vm.swappiness = 1.

Conclusion

The ClusterControl tuning parameters are also based on the algorithm shared in pgtune#L477. It's not fancy, but you can change it to whatever values you would like. With these setting values, it allows you to have a raw start which is ready enough to handle a production load based on the initial given values.

MariaDB Enterprise Backup is a backup solution from MariaDB Corporation with a number of features such as non-blocking backups, full backup, incremental backup, partial backup and Point in Time Recovery.

We often get questions about the differences between MariaDB Backup and ClusterControl’s backup management features. So this is what this blog is about.

Creating backups vs managing them

MariaDB Backup is a fork of Percona XtraBackup, and is a tool to take physical backups of MariaDB server. It allows you to do things like full, incremental, partial backups. One can also perform point in time recovery with the help of binary logs. According to the documentation, the ‘Enterprise’ version of MariaDB backup provides “DDL statement tracking, which reduces lock-time during backups”.

ClusterControl supports MariaDB Backup as a backup method for MariaDB. It provides a graphical user interface to schedule full backups, incremental backups and partial backups, and perform recovery of backup files or also automates point in time recovery. In addition, ClusterControl provides features like encryption, compression, upload to the cloud storage (Azure, AWS, Google Cloud) and automatic verification of backups to ensure they are recoverable. 

Full Backup and Restore

To perform full backup using MariaDB Enterprise Backup, you can use mariabackup command utilities. There are 4 parameter inputs after the mariabackup command. The parameter are :

• Backup - this is used for backup the database using mariabackup utilities. 
• Prepare - to make a point-in-time consistent backup, you need to prepare the backup after the raw backup was executed.
• Copy-back - used to restore the extracted backup to the default data directory of mysql. It will copy the backup to the mysql directory, without removing the original backup.
• Move-back -  used to restore the extracted backup to the mysql data directory by moving all backup directories.

If you want to backup and restore, you just pass the mandatory parameter after mariabackup command. For a full backup command, below is a sample script using MariaDB Backup.

mariabackup --backup --target-dir=/backup/full/ --user=bkpuser --password=p4sswordb4ckup

There are some options you need to define, such as --target-dir, which is the target location for backup files, --user, used for credential users for backup, and --password for the credential backup password. 

To make the backup become point-int-time consistent, you must run the prepare after the full backup is finished. The data files are not consistent until you run the prepare, it is because when you run the backup, the data files were copied at different points in time during the backup.

To run prepare backup:

mariabackup --prepare --target-dir=/backup/full

After you run prepare, it will make the backup ready to be restored. You will see the message on the last line as below,  when the prepare was successful.

InnoDB: Shutdown completed; log sequence number 9553231

You can run the restore command using copy-back. Here is the sample script to restore the backup:

mariabackup --copy-back --target-dir=/backup/full

You can put the above script in a shell script command and give executable permission, configure it on the operating system scheduler.

Backup and Restore using ClusterControl Backup Management is very easy to use. ClusterControl supports logical backup and physical backup. For logical backup, ClusterControl uses mysqldump and for physical backup uses mariabackup full backup and incremental. 

There are two options on how you want to do the backup; you can create the backup directly or you can schedule the backup.

You can also enable some options like encryption, compression, parallel copy thread as shown below :

Restoring the backup is as easy as the backup was created. You just need to select the full backup file that you want to restore.

There are two options on how you want to restore the backup; you can restore the backup to the nodes where the backup was taken or you can restore the backup to a dedicated standalone host.

Incremental Backup and Restore

Taking a full backup of a very large database will be time consuming and resource intensive. Incremental backup is used to perform backup of the changes after the last full backup was taken. 

When incremental backup is running, MariaDB Enterprise Backup will compare previous full backup or incremental backup to find the last changes.

mariabackup --backup --incremental-basedir=/backup/full --target-dir=/backup/incr --user=bkpuser  --password=p4sswordb4ackup

Before you perform the incremental backup, you need to ensure that full backup has been prepared. After that, you can run the incremental backup, applying to the last full backup.

mariabackup --prepare  --target-dir=/backup/full --incremental-dir=/backup/incr

After the incremental backup has been applied to the full backup, the full backup directory will now have all the backup data prepared.

Restoring the prepared full backup with all the incremental changes can be done through :

mariabackup --copy-back --target-dir=/backup/full

To perform incremental backup in ClusterControl, you can choose the mariabackup incremental. You need to have the full prepared backup before doing the incremental backup.

ClusterControl will automatically find the nearest full backup when you run the incremental backup. And for restoring the backup, you can choose the full prepared backup and restore. It will prompt you how you want to restore the backup, either on the node or standalone host. It will restore the backup including incremental changes.

Partial Backup and Restore

Partial backup specifies which database or table you want to backup. You can either choose a list of databases and tables to back up, or you can exclude some of databases and tables from the backup. The options include : --databases, --databases-exclude, --tables, --tables-exclude

Below is a sample script to do the partial backup, for the card_data table.

mariabackup --backup --target-dir=/backup/partial --user=bkpuser --password=p4sswordb4ckup --tables=card_data

You still need to prepare the full partial backup to make the backup point-in-time consistent by running the below command : 

mariabackup --prepare --export --target-dir=/backup/partial

Performing partial restore is very different compared to restoring full backup and incremental backup. You need to prepare the tables and database in the running MariaDB Server, and then manually copy the data files into mysql data directory.

For example, you want to do a partial restore for the card_data table (non-partitioned table).

• Create the empty table of card_data with the same structure in the target database 
• Run the DISCARD tablespace on the table card_data.

  ALTER TABLE carddb.card_data DISCARD TABLESPACE;

• Copy the data files into mysql data directory 

  cp /backup/partial/carddb/card_data.* /var/lib/mysql/carddb

• Change the owner of files become mysql

  chown mysql:mysql /var/lib/mysql/carddb/card_data.*

• Last thing, import the tablespace: 

  ALTER TABLE carddb.card_data IMPORT TABLESPACE;

Partial Backup in ClusterControl is really straightforward, you just need to enable the Partial Backup option. It will give you the option to include or exclude database and tables as shown below :

The next part is similar to the full backup and incremental backup, you can choose settings like encryption and compression.

Restoring the partial backup is exactly the same as when we restore full backup. You just need to choose the partial backup, and the rest will be handled by ClusterControl.

Point in Time Recovery

Restoring the full backup or incremental backup does give you a backup from the time the backup was taken, but it does not give you any data that came after the backup was taken. These changes would be in the binary log. When you perform the prepared backup with binlog enabled, there will be a file called xtrabackup_binlog_info. The file contains a binary log file and position of the last sequence number.

You can perform the point in time recovery by extracting the changes to SQL, like after the restore has been done. You can run mysqlbinlog to extract the specific time in the source database node, and apply the SQL in the target/restored database node.

Point in Time Recovery (PITR) in ClusterControl can be enabled as shown below:

You need to define until what point to recover, there are two options supported which are time based or position based. For time based, you just need to fill the exact time when the data will be restored. For the position based, you need to fill the binlog name and position. The rest of the restore is similar.

Conclusion

That’s it for now. As we have seen above, MariaDB Backup is a nice tool with lots of options. ClusterControl provides an easy to use GUI to perform the backup procedures. It also adds a number of features like encryption, compression, scheduling, retention management and automatic backup verification.

Organizations around the world were unprepared for COVID-19's impact. Severalnines helps companies stay prepared when crisis strikes with their end-to-end disaster recovery solutions.

If the COVID-19 pandemic has shown us anything, it’s that it is a necessity for businesses to modernize, to innovate, and be prepared when crisis looms. For the businesses that are reeling from the economic effects of the pandemic, the value of disaster preparedness is more clear than ever. Severalnines offers businesses of all sizes the opportunity to stay prepared for disaster and secure their crucial, open source database infrastructure; the recent cluster-to-cluster replication feature completes the end-to-end disaster recovery features found in ClusterControl, reducing the complexity, expense, and management of multi-cloud Disaster Recovery.

Businesses are increasingly turning to hybrid, multi-cloud systems for their business data. However, management functions like backup, security, disaster recovery, and compliance are often a laborintensive and time-consuming process. This can result in long recovery times and even data loss in the event of a disaster. A solution that is able to orchestrate all the moving parts, in an automated fashion, can significantly improve recovery point (RPO) and recovery time (RTO) objectives.

Cluster-to-cluster replication gives businesses the ability to improve disaster tolerance for critical applications by providing automated, off-site, database replication (for instance to the cloud). This type of redundancy is a recommended practice for most use cases, but essential for critical services and applications. A local data center can suffer a complete outage, possibly from a hacking attack. A secondary site provides businesses with a way to recover operations and core business functions in a matter of minutes in some cases.

"Flexible multisite database replication with an industry-leading disaster recovery management toolset offers considerably lower RPOs than traditional offsite shipping of backups," said Vinay Joosery, CEO & Co-Founder of Severalnines. "The latest version of ClusterControl helps organizations automate the deployment and management of these sophisticated database setups, especially around switchover, failover and failback."

Large companies with the budget to develop their own disaster recovery plan often turn to the type of systems that utilize multiple failsafes, as the consequences of unplanned and unexpected downtime can be costly, even catastrophic. Severalnines advocates for conventional approaches when dealing with crises — often meaning preparedness and maintaining backups — as disaster scenarios can’t be avoided, only planned for.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability. Severalnines’ products are used by developers and administrators of all skill levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters.

Severalnines is often called the "anti-startup" as it is entirely self-funded by its founders. The company has enabled tens of thousands of deployments to date via its popular product ClusterControl for customers like BT, Orange, Cisco, HP, HP, Paytrail, Ping Identity, and Technicolor. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

To see who is using Severalnines today, visit https://www.severalnines.com/company.

The multi-cloud approach for databases, despite being harder to manage, is growing in popularity as it helps architect against failure and mitigates the risk of cloud lock-in.

STOCKHOLM, August 31, 2020 (Newswire.com) - Multi-cloud databases have been growing largely in popularity in recent years, and their adoption is becoming more widely accepted by more and more businesses. Severalnines is helping organizations take control of their database and efficiently deploy and manage highly available, open source database clusters. Not only does having multiple cloud vendors prevent organizations from being dependent on a single vendor, but the increased security and reliability of multiple, decentralized database deployments ensure that if one instance experiences an unexpected outage, business can quickly resume as usual.

Multiple database deployments also benefit data management processes including backup and restore, data recovery, and data migration; additional perks like added optimization and increased performance are among just a few of the examples of why a growing number of organizations prefer multiple databases in multiple locations across multiple platforms. 

Over the past few years, ClusterControl has added new functionality to allow for a complete multi-cloud approach. It not only allows users to deploy their databases to multiple clouds with just a couple of clicks but also provides a disaster recovery toolset to minimize operations downtime impacted by disasters or other unplanned incidents. ClusterControl supports enterprise-grade backup and restore management, various database replication options, and the deployment of databases to Amazon AWS, Google Cloud, or Microsoft Azure — in any combination.

“Adopting a multi-cloud strategy is key to mitigate cloud vendor lock-ins, creating applications, and services with a competitive edge by taking advantage of a cloud provider’s unique strengths and for business continuity,” said Alex Yu, VP of Products at Severalnines. “The ClusterControl platform empowers organizations to fully adopt stateful applications and services for use in multi and hybrid clouds without the fear of cloud vendor lock-in and provides an industry-leading database disaster recovery toolset to facilitate uninterrupted business operations during disasters.”

Severalnines is committed to bringing you advanced tools and systems to help you automate your open source database infrastructures. To learn more, visit https://severalnines.com/.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability. Severalnines' products are used by developers and administrators of all skill levels to provide the full "deploy, manage, monitor, scale" database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the anti-startup as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. It currently counts BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden, with offices in Singapore, Japan and the United States. To see who is using Severalnines today, visit https://www.severalnines.com/company.