No matter what kind of application/system you are running, you will always need to manage and monitor your databases. Even in a small project, you want to be sure everything is going fine or if you might need to change something, and there are also different tasks to perform if you want to ensure that your systems will run in a healthy way.

This COVID-19 pandemic increased the working and studying from home in an exponential way, so many platforms took an important role in this situation, and probably one of the most important was Moodle.

Moodle is a learning platform designed to provide educators, administrators, and learners with a single robust, secure, and integrated system to create personalized learning environments. It supports different database technologies (e.g. MariaDB, MySQL, PostgreSQL, etc).

In this blog, we will take a look at what you need to manage and monitor in a Moodle Database, and how you can do it easier using ClusterControl.

Moodle Database Configuration

First, let’s take a look at the Moodle Database configuration file: /var/www/html/moodle/config.php:

//=========================================================================

// 1. DATABASE SETUP

//=========================================================================

$CFG->dbtype    = 'mariadb';      // 'pgsql', 'mariadb', 'mysqli', 'sqlsrv' or 'oci'

$CFG->dblibrary = 'native';     // 'native' only at the moment

$CFG->dbhost    = '10.10.10.134';  // eg 'localhost' or 'db.isp.com' or IP

$CFG->dbname    = 'moodle';     // database name, eg moodle

$CFG->dbuser    = 'moodleuser';   // your database username

$CFG->dbpass    = 'moodlepass';   // your database password

$CFG->prefix    = 'mdl_';       // prefix to use for all table names

$CFG->dboptions = array(

    'dbpersist' => false, 

    'dbsocket'  => false, 

    'dbport'    => '',

    'dbhandlesoptions' => false,

    'dbcollation' => 'utf8mb4_unicode_ci', 

);

This is the basic database configuration, but there is also an optional configuration commented out in the same file:

'connecttimeout' => null,

    'readonly' => [

      'instance' => [ 

        [

          'dbhost' => '10.10.10.135',

          'dbport' => '',

          'dbuser' => '',

          'dbpass' => '',

        ],

…

So, it means you can configure slave read-only nodes to balance the read traffic and increase the performance. Another option is to use a Load Balancer, who will send the traffic to the available nodes and it could be able to help on failover tasks.

Now let’s see what you should monitor in this Moodle Database.

What to Monitor in the Moodle Database

When monitoring systems, there are two main things to take into account: the operating system and the database itself. You will need to define which metrics you are going to monitor from both sides and how you are going to do it. You need to monitor the metric always in the context of your system, and you should look for alterations on the behavior pattern.

Keep in mind that when one of your metrics is affected, it can also affect others, making troubleshooting of the issue more complex. Having a good monitoring and alerting system is important to make this task as simple as possible.

Operating System Monitoring

One important thing (which is common to all database engines and even to all systems) is to monitor the Operating System behavior. Here are some points to check.

CPU Usage

An excessive percentage of CPU usage could be a problem if it is not usual behavior. In this case, it is important to identify the process/processes that are generating this issue. If the problem is the database process, you will need to check what is happening inside the database.

RAM Memory or SWAP Usage

If you are seeing a high value for this metric and nothing has changed in your system, you probably need to check your database configuration. Depending on the database technology, there are different parameters to specify the amount of memory to be used for different database tasks.

Disk Usage

An abnormal increase in the use of disk space or an excessive disk access consumption are important things to monitor as you could have a high number of errors logged in the log file or a configuration that could generate an important disk access consumption instead of using memory to process the queries.

Load Average

It is related to the three points mentioned above. A high load average could be generated by an excessive CPU, RAM, or Disk Usage.

Network

A network issue can affect all the systems as the application can’t connect (or connect losing packages) to the database, so this is an important metric to monitor indeed. You can monitor latency or packet loss, and the main issue could be a network saturation, a hardware issue, or just a bad network configuration.

Database Monitoring

Monitoring your database is not only important to see if you are having an issue, but also to know if you need to change something to improve your database performance, that is probably one of the most important things to monitor in a database. Let’s see some metrics that are important for this.

Query Monitoring

In general, the databases are configured with compatibility and stability in mind by default, so you need to know your queries and his pattern, and configure your databases depending on the traffic that you have. Here, you can use the EXPLAIN command to check the query plan for a specific query, and you can also monitor the amount of SELECT, INSERT, UPDATE or DELETEs on each node. If you have a long query or a high number of queries running at the same time, that could be a problem for all the systems.

Active Sessions

You should also monitor the number of active sessions. If you are near the limit, you need to check if something is wrong or if you just need to increment the max connection value in the database configuration. The difference in the number can be an increase or decrease of connections. Bad usage of connection pooling, locking, or network issues are the most common problems related to the number of connections.

Database Locks

If you have a query waiting for another query, you need to check if that other query is a normal process or something new. In some cases, if somebody is making an update on a big table, for example, this action can be affecting the normal behavior of your database, generating a high number of locks.

Replication

The key metrics to monitor for replication are the lag and the replication state. The most common issues are networking issues, hardware resource issues, or under dimensioning issues. If you are facing a replication issue you will need to know this asap as you will need to fix it to ensure the high availability environment.

Backups

Avoiding data loss is one of the basic DBA tasks, so you don’t only need to take the backup, you should know if the backup was completed, and if it is usable. Usually, this last point is not taken into account, but it is probably the most important check in a backup process.

Database Logs

You should monitor your database log for errors, authentication issues, or even long-running queries. Most of the errors are written in the log file with detailed useful information to fix it.

Notifications & Alerting

Just monitoring a system is not enough if you don’t receive a notification about each issue. Without an alerting system, you should go to the monitoring tool to see if everything is fine, and it could be possible that you are having a big issue since many hours ago. This alerting job could be done by using email alerts, text alerts, or other tools like slack.

Moodle Database Management Tasks

Monitoring a database doesn’t make sense if you can’t manage it. If you detect an issue, you will need to fix it, or even if everything looks fine, you should be able to perform some basic management tasks that we will mention below.

Failover

This is an important task in a High Availability environment. It could be manual or automatic, but it is a must to be able to keep your database running in a DRP (Disaster Recovery Plan) or even in a maintenance planned task. You can also add a Load Balancer here, to avoid any change in your application after performing a failover task.

Scaling

Another important task is to scale-out your database topology, which means adding more database nodes. You can do it if you are expecting or having more traffic for any reason.

Backup Scheduling

Backups are a very important part of any DRP, to prevent data loss in the event of an accident or hardware failure. It might not always be enough to guarantee an acceptable Recovery Point Objective (RPO) but is a good first approach. Also, you should define a Recovery Time Objective (RTO) according to your company requirements. There are many ways to reach the RTO value, it depends on the company goals, and in any case, scheduling backups in a correct way is the key.

Backup verification plays also a very important role here. Rest assured, your backup must be a valid type of backup and is a reliable copy when a crisis takes place. Adding the mechanism to store your backup not only in your premises or datacenter but also store it elsewhere securely like in the cloud or to AWS S3 or Google Cloud Storage for example.

Database Security

Security is usually the major primary concern when it comes to managing your database cluster. Enable or disable remote access, who will be able to access it, how to add security restrictions, or how to manage the users and review the user's permission are pretty common topics in this section. 

Centralized Database Logs

Logs Centralization provides you a very convenient way to investigate and implement a security analysis tool to understand your database clusters and how they behave. In case you need to investigate a cluster-wide problem and see what has been going through your database clusters, proxies, or load balancers. It is very convenient to just look at one place. 

It's really difficult to find a tool to monitor all the necessary metrics for your Moodle Database and, at the same time, allow you to manage your database cluster. In general, you will need to use more than one tool and even some scripting will need to be made. One way to centralize the managing and monitoring tasks is by using ClusterControl, which provides you with features like backup management, monitoring and alerting, deployment and scaling, automatic recovery, and more important features to help you manage your databases. All these features on the same system.

Managing and Monitoring Your Moodle Database with ClusterControl

As we mentioned, using ClusterControl you can perform different management and monitoring tasks like add/remove load balancers, add/remove slave nodes, automatic failover and recovery, backups, create/modify advisors, create/modify dashboards, and more. Let’s see some of these features.

ClusterControl Monitoring

ClusterControl allows you to monitor your database servers in real-time. It has a predefined set of dashboards for you, to analyze some of the most common metrics mentioned above and even more. 

It also allows you to customize the graphs availables in the cluster, and you can enable agent-based monitoring to generate more detailed dashboards. 

Also, you have the query monitor and the performance sections, where you can find all your database information. With these features, you can see how your database is going.

ClusterControl Alerting

You can create alerts, which inform you of events in your cluster, or integrate it with different services such as PagerDuty or Slack.

ClusterControl Scaling

To scale-out your database you can (depending on the technology that you are using) add nodes or replication slaves. Go to the Cluster Actions section to see the available actions for your topology.

For a specific database node, you can also Reboot Host, Rebuild Replication Slave, Promote Slave, and perform more node actions with just one click.

ClusterControl Failover

Having the auto-recovery feature enabled, in case of failure, ClusterControl will promote the most advanced slave and, if there is a load balancer deployed, it will change the configuration to enable the new master and disable the old one. In this way,  you will have automatic failover in case of failure.

ClusterControl Backup Management

For backup management, ClusterControl centralizes it to protect, secure, and recover your data, and with the verification backup feature, you can confirm if the backup is good to go.

You can perform manual or scheduled backups using different tools like XtraBackup, Mariabackup, pg_basebackup, etc. The tool depends on the technology and you can specify this during the backup configuration process.

In ClusterControl, select your cluster and go to the "Backup" section to see your current backups. You also have PITR, Incremental/Differential Backups, Verify Backup, Restore, Create a Cluster from Backup, and more Backup Features.

ClusterControl Security

About security, you can manage users/roles for your database cluster. Go to ClusterControl -> Select Cluster -> Manage -> Schema and Users (or User Management if you are using PostgreSQL).

Here you can see all the accounts with the privileges assigned, and you can create a new one, or modify/edit an existing account.

In the security section, you can enable different security features like SSL or Audit Log.

ClusterControl Centralized Logs

You don’t need to access your database node to check the logs, you can find all your database logs centralized in the ClusterControl Log section.

As you can see, you can handle all the mentioned things from the same centralized system: ClusterControl.

Managing and Monitoring with the ClusterControl Command Line

For scripting and automating tasks, or even if you just prefer the command line, ClusterControl has the s9s tool. It is a command-line tool for managing your database cluster.

• Cluster List

• Node List

You can perform all the tasks that you are able to see in the ClusterControl UI (and even more), and you can integrate this feature with some external tools like slack, to manage it from there.

Conclusion

In this blog, we mentioned some important metrics to monitor in your Moodle Database, and some management tasks useful to keep your system running. Managing and Monitoring your Moodle Database is necessary but also a time-consuming task if you don’t have any tools to help with this.

For this, we also show how ClusterControl can make your life easier by using the main managing and monitoring features.

We’re excited to announce the 1.8 release of ClusterControl - the only database management system you’ll ever need to take control of your open source database infrastructure. 

ClusterControl has always allowed our users to manage multiple database clusters, using different technologies, in a polyglot environment; all from a single interface. But earlier this year a customer challenged us with the desire to manage hundreds of database nodes from a single controller.

This release is the culmination of the efforts to meet and exceed that need. Through countless system improvements, ClusterControl 1.8 now lets you deploy, manage, and monitor hundreds of nodes from one single controller.

Coupled with the scalability improvements, we also have added the ability to add keyword tags to your database clusters along with new search options, making it easy to quickly locate the cluster you wish to manage. This feature is currently only available as part of the CLI.

Release Highlights

Across the Board Scalability Improvements

• The backend components of ClusterControl have been fundamentally improved to allow for the ability to manage hundreds of database nodes from a single controller
• These improvements include improved load times while lowering CPU consumption

Cluster Tagging

• When managing a large number of database clusters you need to be able to find the one you are looking for quickly and easily. 
• Cluster Tagging allows you to add keyword tags to each cluster (e.g. Tag by department, project, application, server, cloud provider, etc). You can then look up clusters by the keyword tags.
• Cluster Tagging is currently only supported in the ClusterControl CLI

Integration with Vault for Maximum Data Protection

• Vault by HashiCorp is a service that allows you to keep your ultra sensitive data secure, even when deployed across multiple servers & clouds.
• ClusterControl can now integrate directly with the Vault APIs from the ClusterControl CLI, allowing you to shift from the standard methods of security to a more dynamic infrastructure as defined by ephemeral applications and servers, trusted sources of user and application identity, and software-based encryption.

View Release Details and Resources

• Download ClusterControl

• ChangeLog

• ClusterControl Administration Guide

• Upgrade Instructions

Release Details

Improved Scalability

Polyglot at Scale - this is what the new ClusterControl 1.8 offers you. Our customers now have the ability to manage hundreds of database nodes (including master/master, master/slave, analytics, etc) across an array of open source database technologies. 

ClusterControl has always allowed you to manage many different databases, but with the scalability improvements in this release, any enterprise can now manage all of their open source databases, using enterprise-grade features, at scale.

Cluster Tagging

Some DBA’s and SysAdmins are responsible for managing dozens, if not hundreds of databases. Without ClusterControl, these typically have to be managed individually, one-by-one.

ClusterControl allows you to deploy and manage many different technology stacks, all from a single controller. With cluster tagging, the user can now add keyword tags to each cluster and then filter by the keyword to show the clusters with that tag. This feature is currently only supported in the ClusterControl CLI.

Vault Integration

Not all data should be treated equally. Using a traditional infrastructure model, the system was protected by erecting a wall around the hardware, but in the days of multi-cloud, SaaS, and globalization this type of security model doesn’t really work anymore.

Vault is an API-centric approach that allows you to control the security of your data using dynamic, software-based methods instead of relying solely on what’s offered from the hosts and applications.

ClusterControl now has the ability to integrate with Vault via the ClusterControl CLI and ensure your most secret information remains protected.

 

The security is paramount nowadays, especially for the organizations that deal with the sensitive data - financial, health, personal data. In those cases you are obliged to comply with numerous security and data protection standards like HIPAA, PCI DSS or similar. With the most recent release, ClusterControl makes it even easier to meet the strict standards. ClusterControl 1.8.0 comes with the support for integration with HashiCorp Vault. Let’s take a look at the improvements it brings.

ClusterControl integration with HashiCorp Vault

ClusterControl is a management platform therefore it requires full access to the systems it manages - databases. It has to store passwords for management users. Currently it’s done in the cluster configuration files - all the required access details for the cluster are stored in one of the configuration files located in /etc/cmon.d/cmon_X.cnf. Here is the example for the PostgreSQL cluster:

postgresql_password=pass

postgresql_server_addresses=10.0.0.105:5432,10.0.0.106:5432

postgresql_user=myuser

repl_password=!ROos}Shf9Z,6fWdF2Iq83V0wXom!P03

repl_user=cmon_replication

rpc_key=yx9i2r8olpou9lgi

As you can see, whoever can access this file will be able to access the PostgreSQL nodes. It also stores rpc_key that can be used to manage the cluster using ClusterControl. Of course, those files are accessible only to the root user and one can argue that if someone has gained access to the root account, you are pretty much in a bad spot anyway, but ensuring that there are no passwords in the configuration file is still a good practice.

root@vagrant:~# ls -alh /etc/cmon.d/

total 28K

drwx------   2 root root 4.0K Sep 23 10:02 .

drwxr-xr-x 103 root root 4.0K Sep 23 13:34 ..

-rw-------   1 root root  668 Sep 23 13:34 cmon_1.cnf

-rw-------   1 root root  695 Sep 23 13:34 cmon_2.cnf

-rw-------   1 root root  600 Sep 23 13:34 cmon_3.cnf

-rw-------   1 root root  585 Sep 23 13:34 cmon_4.cnf

-rw-r--r--   1 root root   81 Sep  1  2016 README.txt

Here enters HashiCorp Vault - it is a product dedicated to storing sensitive data, just like the passwords we want to protect. It made sense to use it to integrate with ClusterControl. As of now, in ClusterControl 1.8.0, the only way to integrate is to use a pre-deployed Vault. Vault can be deployed in a number of ways, including clustered deployment. Covering this is out of the scope of this blog, we will still show you how to test the integration by deploying a standalone node of HashiCorp Vault and configure it with ClusterControl. Let’s take a look at the steps required to accomplish this.

Deploying HashiCorp Vault

The deployment process for HashiCorp Vault is pretty straightforward. At first, we have to create directory for the data:

mkdir -p /root/vault/data

cd /root/vault

Then we should download and decompress the Vault:

wget https://releases.hashicorp.com/vault/1.5.3/vault_1.5.3_linux_amd64.zip

unzip vault_1.5.3_linux_amd64.zip

Next, we should create SSL certificate:

# create an SSL certificate

if [ ! -f cert.crt ]; then

    openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout cert.key -out cert.crt

fi

Then we should create Valut’s configuration file:

# create a Vault configuration

cat - > vault.hcl <<CFGEND

ui = true



storage "file" {

  path = "/root/vault/data"

}



listener "tcp" {

  address     = "0.0.0.0:8200"

  tls_cert_file = "/root/vault/cert.crt"

  tls_key_file = "/root/vault/cert.key"

}



# replace with you computer's IP

api_addr = "https://10.0.0.100:8200"

CFGEND

Once this is done, you can start the Vault. It will have to be initialized and later unsealed before ClusterControl will be able to actually use it.

# Start the vault server:

$ ./vault server -config=vault.hcl

Starting from then the UI at https://IP:8200/ui/ is available and you can use it for management actions regarding the Vault. It is not necessary to use it though. We have to initialize the Vault now:

root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator init -key-shares=1 -key-threshold=1

Unseal Key 1: HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=



Initial Root Token: s.f1kl9XUBuFjdyV048QIgHoJh



Vault initialized with 1 key shares and a key threshold of 1. Please securely

distribute the key shares printed above. When the Vault is re-sealed,

restarted, or stopped, you must supply at least 1 of these keys to unseal it

before it can start servicing requests.



Vault does not store the generated master key. Without at least 1 key to

reconstruct the master key, Vault will remain permanently sealed!



It is possible to generate new unseal keys, provided you have a quorum of

existing unseal keys shares. See "vault operator rekey" for more information.

You can see here unseal key and root token. Unseal key is required in order to unseal the Vault, which we will do in the next step. Root token will be used by ClusterControl to connect to the Vault.

Let’s unseal the Vault:

root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator unseal HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=

Key             Value

---             -----

Seal Type       shamir

Initialized     true

Sealed          false

Total Shares    1

Threshold       1

Version         1.5.3

Cluster Name    vault-cluster-2bf2e433

Cluster ID      350fd80a-67c8-bda0-6d6a-e77354dacfa2

HA Enabled      false

Now we have to setup ClusterControl to use the Vault to store passwords. This can be accomplished by creating following entries in the /etc/cmon.cnf:

vault_token=s.f1kl9XUBuFjdyV048QIgHoJh

vault_auto_migrate=true

Vault_token should contain a root token for the Vault and vault_auto_migrate=true means that ClusterControl will automatically migrate data from configuration files into the Vault. This is a good moment to make a copy of your existing configuration files located in /etc/cmon.d/ as you will need them should you decide to revert this process and do not use Vault at some point in the future.

Once you have the configuration set up, you have to restart cmon process to migrate data into the Vault:

service cmon restart

This is it, ClusterControl, from now on, will use the data in the Vault to access the backend databases. We can easily check what exactly is stored there:

root@vagrant:~/vault# export VAULT_TOKEN=s.f1kl9XUBuFjdyV048QIgHoJh

root@vagrant:~/vault# ./vault secrets list

Path               Type         Accessor              Description

----               ----         --------              -----------

clustercontrol/    kv           kv_939a83de           Created by ClusterControl

cubbyhole/         cubbyhole    cubbyhole_42b40ca0    per-token private secret storage

identity/          identity     identity_594e8bfc     identity store

sys/               system       system_9509de58       system endpoints used for control, policy and debugging

Now, knowing the path, we can query the data:

root@vagrant:~/vault# ./vault kv list clustercontrol/

Keys

----

clustercontrol/



root@vagrant:~/vault# ./vault kv list clustercontrol/clustercontrol

Keys

----

cluster_0

cluster_1

cluster_2

cluster_3

cluster_4

Let’s retrieve data for cluster_4:

root@vagrant:~/vault# ./vault kv get clustercontrol/clustercontrol/cluster_4

====== Metadata ======

Key              Value

---              -----

created_time     2020-09-24T10:51:27.242621166Z

deletion_time    n/a

destroyed        false

version          5



============ Data ============

Key                     Value

---                     -----

backup_user_password    Y3rocu-7wWGSJ-56iMK7cd8@yK+70AzH

mysql_password          pass

postgresql_password     pass

repl_password           !ROos}Shf9Z,6fWdF2Iq83V0wXom!P03

rpc_key                 yx9i2r8olpou9lgi

As you can see, the passwords are stored in the Vault. You can access the data using the UI as well, after authenticating with root token:

Please keep in mind that ClusterControl, once it is configured to work with Vault, will require Vault to be available and unsealed before ClusterControl can work properly. If it won’t be available, ClusterControl won’t work. 

Should you want to revert this process, you should follow those steps:

1. Get all the keys and values from the Vault for all of the clusters.
2. Stop cmon process
3. Remove Vault configuration from /etc/cmon.cnf
4. Add the keys and values you extracted from the Vault into respective configuration files in /etc/cmon.d. You can use the backup copy of the configuration from before Vault has been used but please keep in mind that some additional passwords might have been added in the meantime - double check that your old configuration files contain all the data that is stored in the Vault.

We hope this short introduction to ClusterControl integration with HashiCorp Vault will be useful. If you decided to give it a try, let us know how it worked for you.

ClusterControl is an automation tool for managing open source databases. It supports various databases such as  MySQL, MariaDB, PostgreSQL, Timescale, and MongoDB. It is used to handle the full “deploy, monitor, and scale” cycle of your database, and supports a number of high availability clustered setups - from replication setups to synchronous multi-master clusters.

It plays an important role in managing database operations, for instance managing backups and verifying they are restorable, or managing a failover and orchestrating recovery. Therefore, it is important that the software is kept up to date. Patches are usually released on a weekly basis.

In this blog, we will discuss some tips when upgrading ClusterControl to the latest release.

ClusterControl Pre-Upgrade Tips

Before upgrading ClusterControl, you might want to check your current ClusterControl configuration and services, whether they are up and running. Also, check if there are any errors, misconfigurations, or under-optimized parameters for the CMON database, cmon cloud services, and cmon event services. You can dig into the ClusterControl logs. For the controller log, you can tail the logs on the path /var/log/cmon.log, cmon-event log can be found on /var/log/cmon-events.log, and the log related accessibility to the cloud can be checked at /var/log/cmon-cloud.log

Upgrading ClusterControl

There are two strategies for upgrading ClusterControl, you can upgrade the package directly on the ClusterControl host or you can install the latest version of ClusterControl on a new host, and after that migrate your managed databases to the new ClusterControl instance. 

Upgrade Process for an Existing ClusterControl Instance

Upgrading ClusterControl on the current host is really straightforward, the upgrade instructions will differ based on operating system distribution (CentOS or Debian/Ubuntu). Please take note, upgrading your ClusterControl instance won’t halt or interrupt your existing databases, but you might lose some functionality of ClusterControl eg: autorecovery cluster & node, during the time the upgrade is performed.

• For Redhat/CentOS distribution, clean the repository cache, and then install the clustercontrol packages as below: 

  $ yum clean all
  
  $ yum install clustercontrol clustercontrol-controller clustercontrol-ssh clustercontrol-notifications clustercontrol-cloud clustercontrol-clud s9s-tools

  It will automatically download the latest package and upgrade the existing package with the latest one. After that, restart the service to take effect.

  $ service cmon restart
  
  $ service cmon-ssh restart
  
  $ service cmon-events restart
  
  $ service cmon-cloud restart

• For Debian/Ubuntu distribution, you need to update the package repository, and then install the package through APT

  $ sudo apt-get update
  
  $ sudo apt-get install clustercontrol clustercontrol-controller clustercontrol-ssh clustercontrol-notifications clustercontrol-cloud clustercontrol-clud s9s-tools

  And then restart the ClusterControl services as below:

  $ service cmon restart
  
  $ service cmon-ssh restart
  
  $ service cmon-events restart
  
  $ service cmon-cloud restart

That’s it, you have now upgraded your existing ClusterControl instance. But if you would prefer to not touch your existing ClusterControl instance, for instance to make sure the new ClusterControl version works well in your environment, then you can set it up on a new host. 

Setup a New Host with the Latest ClusterControl Version

You can set up the latest ClusterControl on the new host, the easiest way to install ClusterControl is using the install-cc script as below:

$ wget http://www.severalnines.com/downloads/cmon/install-cc

$ chmod +x install-cc

$ sudo ./install-cc   # omit sudo if you run as root

Give execute permission to the file, and run as root user or other user that has sudo access. It will automatically install the latest version, you just need to follow the instructions. After the setup is finished, you can delete the managed databases and clusters from the old ClusterControl through Cluster Menu -> Delete Cluster.

It will prompt you with a dialogue box to confirm the removal. The database nodes won't be stopped or uninstalled, it just remove the nodes metadata from ClusterControl.

After that, you can import your database nodes into the new ClusterControl through Import Cluster.

And then choose the version, fill the ip address as below:

The last step is to press the Import button, it will automatically import your existing database into the new ClusterControl.

It would be great if you can test the upgrade on the staging/dev environment, before going to production. Also, if you are running ClusterControl HA on your environment, the best way to upgrade is through rolling upgrades. You can upgrade the ClusterControl on your standby node first before the primary upgrade.

Post-Upgrade

After upgrading finish, the easiest to verify if the version of ClusterControl is the latest version by running command below in controller node:

$ cmon -v

You sould also check the logs to see if there were any errors during the upgrade.

You may have heard about Percona Monitoring and Management (PMM), a solution from Percona that is focused on the monitoring of open source databases. It is a quite popular and free solution that provides access to the metrics that will give you a good insight on what is happening in the database.

ClusterControl, on the other hand, is a platform that is intended to both monitor and manage open source databases. You may be curious how those two solutions compare. This blog should give you some insight into what that looks like. Let’s focus on the monitoring first, and then we’ll talk about management.

Monitoring the Databases Using ClusterControl and PMM

Database Monitoring with ClusterControl

ClusterControl, by default, comes with basic monitoring capabilities where data is being collected over SSH and native database connections. There are several predefined dashboards that are intended to cover the most important aspects of the day-to-day operations.

There is also an option to deploy agent-based monitoring.

In the process ClusterControl will deploy a Prometheus which will be used to store metrics and it will be queried by ClusterControl to retrieve data and populate the graphs. On the backend databases ClusterControl will deploy exporters - agents that are intended to collect metrics from the database, operating system as well as with some of the supported load balancers. The data is pushed by exporters into Prometheus. The final outcome is a set of predefined dashboards that cover the most important aspects of the database operations:

The main advantage of ClusterControl is that it comes with predefined dashboards that are intended to show the most important bits of the metrics while still being able to graph every single metric that is collected:

This is quite important as people without in-depth knowledge of the given database technology may not be able to figure out on their own what data is the most important for them and what is that they should be looking at. On the other hand, power users can still access all the bits of the information collected from the database.

Database Monitoring with PMM

PMM is a standalone product that can be deployed in many ways, the most common being probably the Docker image that Percona made available. PMM is designed as an agent-based solution. The server, where the data is stored, has multiple services running, among others Prometheus to store time-series and Graphana, which provides an UI to access the data.

The data itself, by default, is accessible through numerous dashboards that cover pretty much every aspect of the database metrics.

This makes it possible to access all of the data quite easily but for the casual user it might be tricky to understand what is the most important piece of data between all of the metrics and dashboards made available by Percona Monitoring and Management.

Obviously, there are pros and cons to everything. While casual users will initially struggle to find meaningful data, someone experienced will be able to pretty much dissect the database performance and analyze it from multiple angles. PMM is definitely a powerful tool if you know how to use it.

Query Monitoring

Both products provide an option to monitor query performance. Both collect the data in the similar fashion - through Performance Schema or slow query log. ClusterControl presents only the most important bits of the data: total execution time, some metrics like an average execution time or standard deviation for a given query type:

On the other hand, PMM makes available a very detailed overview of the query performance

It allows to see the query performance change in time as well as detailed statistics about the particular query itself.

Management Features of ClusterControl and Percona Monitoring and Management

By default, PMM does not provide any kind of management features. The only optional feature is the automated failover handling and replication  topology management implemented through Orchestrator, which can be enabled manually when you start a PMM docker container. This is the only piece of management functionality that comes with PMM. Percona provides multiple management options through Kubernetes operators for Percona-flavoured databases, we have covered them in our other blog posts.

On the other hand ClusterControl has an extensive set of management features that starts with automated failover but goes through node management, scaling (adding nodes to the cluster), backups, automation of creating DR sites and clones of production clusters. ClusterControl also helps during the upgrade process and it comes with a user interface for load balancers like ProxySQL. The best way to get yourself familiar with all the features that ClusterControl provides would be to register for a free trial and experience all of them for yourself.

In these pandemic times, Learning Management System (LMS) platforms are getting more and more important to allow you to continue learning remotely when the traditional education system just isn’t available anymore.

Having an LMS platform without High Availability could be a problem in case of failure, as all your efforts to keep the system running won’t make sense without having the database available at all times.

In this blog, we will see a popular LMS application called Canvas LMS, and how to deploy it in a high availability way using PostgreSQL and ClusterControl.

What is Canvas?

Canvas is a web-based Learning Management System (LMS). It is used by learning institutions, educators, and students to access and manage online course learning materials and communicate about skill development and learning achievement.

Canvas includes a variety of customizable course creation and management tools, course and user analytics and statistics, and internal communication tools.

Canvas LMS PostgreSQL Database Deployment

First, let’s deploy a PostgreSQL Cluster, which will be used by the Canvas LMS application. For this, we will use ClusterControl to deploy 3 PostgreSQL nodes (1 Primary and 2 Standby nodes), and 2 HAProxy Load Balancers with Keepalived configured between them.

HAProxy is a load balancer that distributes traffic from one origin to one or more destinations and can define specific rules and/or protocols for this task. If any of the destinations stops responding, it is marked as offline, and the traffic is sent to the rest of the available destinations.

Keepalived is a service that allows you to configure a Virtual IP Address within an active/passive group of servers. This Virtual IP Address is assigned to an active server. If this server fails, the IP Address is automatically migrated to the “Secondary” passive server, allowing it to continue working with the same IP Address in a transparent way for the systems.

So, let's see how to implement the mentioned topology using ClusterControl.

Database Deployment

To perform a deployment from ClusterControl, simply select the option “Deploy” and follow the instructions that appear.

When selecting PostgreSQL, you must specify User, Key or Password, and Port to connect by SSH to your servers. You can also add a name for your new cluster and if you want ClusterControl to install the corresponding software and configurations for you.

After setting up the SSH access information, you need to define the database credentials, version, and datadir (optional). You can also specify which repository to use.

In the next step, you need to add your servers to the cluster that you are going to create using the IP Address or Hostname.

In the last step, you can choose if your replication will be Synchronous or Asynchronous, and then just press on “Deploy”

Once the task is finished, you can see your new PostgreSQL cluster in the main ClusterControl screen.

Now you have your cluster created, you can perform several tasks on it, like adding a load balancer (HAProxy) or a new replica.

Load Balancer Deployment

To perform a load balancer deployment, select the option “Add Load Balancer” in the cluster actions, and complete the asked information.

You only need to add IP or Hostname, Port, Policy, and the nodes you are going to use.

Keepalived Deployment

To perform a Keepalived deployment, select the option “Add Load Balancer” in the cluster actions and then, go to the Keepalived Tab.

Here, select the HAProxy nodes, and specify the Virtual IP Address that will be used to access the database.

At this moment, you should have the following topology:

Now, let’s connect this environment to Canvas LMS for High Availability.

How to Configure Canvas LMS

First, you will need to install it. There are different ways to do it, using an automated setup on Docker, manually, or even using different approaches like QuickStart or Production-Oriented installation. You can check the official documentation to choose the best method for you.

Once you have Canvas LMS installed, you can proceed to configure the database.yml file to use the PostgreSQL High Availability environment that you just deployed.

$ cat config/database.yml

production:

  adapter: postgresql

  encoding: utf8

  database: canvas_production

  host: VIRTUAL_IPADDRESS

  username: CANVAS_USER

  password: CANVAS_PASSWD

  timeout: 5000

Here, replace:

• VIRTUAL_IPADDRESS for Virtual IP Address configured in Keepalived

• CANVAS_USER for the Canvas database user

• CANVAS_PASSWD for the Canvas database password

Make sure you can access your database using your Virtual IP Address as the host and it is allowed in the pg_hba.conf PostgreSQL configuration file. You can test it by running the following command from your application server:

$ psql -h VIRTUAL_IPADDRESS -U CANVAS_USER canvas_production

ClusterControl Autorecovery Feature

So, the question is, apart from the deployment process, what is the role of ClusterControl here?

In case of failure, ClusterControl will promote the most advanced standby node to primary as well as notify you of the problem. It also fails over the rest of the standby node to replicate from the new primary server.

By default, HAProxy is configured with two different ports: read-write and read-only. In the read-write port, you have your primary node as online and the rest of the nodes as offline, and in the read-only port, you have both the primary and the standby nodes online.

When HAProxy detects that one of your nodes is not accessible, it automatically marks it as offline and does not take it into account for sending traffic to it. Detection is done by health check scripts that are configured by ClusterControl at the time of deployment. These check whether the instances are up, whether they are undergoing recovery, or are read-only.

When ClusterControl promotes a standby node, HAProxy marks the old primary as offline for both ports and puts the promoted node online in the read-write port.

If your active HAProxy, which is assigned a Virtual IP Address to which your systems connect, fails, Keepalived migrates this IP Address to your passive HAProxy automatically. This means that your systems are then able to continue to function normally.

Conclusion

In this blog, we talked about the importance of having a High Availability PostgreSQL environment to use with your Canvas LMS platform, and how ClusterControl can help you with both Deployment and Autorecovery tasks. We had also mentioned how to improve this environment by adding HAProxy and Keepalived for High Availability purposes.

Percona XtraDB Cluster is a very well known high availability solution in MySQL world. It is based on Galera Cluster and it does provide virtually synchronous replication across multiple nodes. As with every database, it is crucial to keep track of what is happening in the system, if performance is on the expected levels and, if not, what is the bottleneck. This is of utmost importance to be able to react properly in the situation, where performance is impacted. Of course, Percona XtraDB Cluster comes with multiple metrics and it is not always clear which of them are the most important ones to track the state of the database. In this blog we will discuss a couple of the key metrics you want to keep an eye on while working with PXC.

To make it clear, we will be focusing on the metrics unique to PXC and Galera, we will not be covering metrics for MySQL or InnoDB. Those metrics have been discussed in our previous blogs.

Let’s take a look at some of the most important information that PXC presents to us.

Flow Control

Flow control is pretty much the most important metric you can monitor in any Galera Cluster therefore let’s have a bit of background. Galera is a multi-master, virtually synchronous cluster. It is possible to execute writes on any of the database nodes that form it. Every write has to be sent to all of the nodes in the cluster to ensure that it can be applied - this process is called the certification. No transaction can be applied before all nodes agree it can be committed. If any of the nodes has performance problems that makes it unable to cope with the traffic, it will start issuing flow control messages which are intended to inform the rest of the cluster about the performance problems and ask them to reduce the workload and help the delayed node to catch up with the rest of the cluster.

You can track when nodes had to introduce artificial pause to let their lagging peers to catch up using flow control paused metric (wsrep_flow_control_paused):

You can also track if the node is sending or receiving the flow control messages (wsrep_flow_control_recv and wsrep_flow_control_sent).

This information will help you understand better which node is not performing on the same level as its peers. You can then focus on that node and try to understand what is the issue and how to remove the bottleneck.

Send and Receive Queues

Those metrics are sort of related to the flow control. As we have discussed, a node may be lagging behind other nodes in the cluster. It can be caused by a non-even workload split or by other reasons (some process running in the background, backup or some custom, heavy queries). Before the flow control kicks in, lagging nodes will attempt to store the incoming writesets in the receive queue (wsrep_local_recv_queue) hoping that the performance impact is transient and it will be able to catch up very soon. Only if the queue becomes too big (it is governed by gcs.fc_limit setting), flow control messages start to be sent across the cluster.

You can think of a receive queue as the early marker which shows that there are problems with the performance and the flow control may kick in.

On the other hand, send queue (wsrep_local_send_queue) will tell you that the node is not able to send the writesets to other members of the cluster which may indicate problems with the network connectivity (pushing the writesets to the network is not really resource-intensive).

Parallelization Metrics

Percona XtraDB cluster can be configured to use multiple threads to apply the incoming writesets - it allows it to better handle multiple threads connecting to the cluster and issuing writes at the same time. There are two main metrics that you may want to keep an eye on.

First, wsrep_cert_deps_distance, tells us what is the parallelization potential - how many writesets can, potentially, be applied at the same time. Based on this value you can configure the number of parallel slave threads (wsrep_slave_threads) that will work on applying incoming writesets. The rule of thumb is that there is no point in configuring more threads than the value of wsrep_cert_deps_distance.

Second metric, on the other hand, tells us how efficiently we were able to parallelize the process of applying writesets - wsrep_apply_oooe tells us how often applier started to apply writesets out of order (which points towards better parallelization).

Conclusion

As you can see, there are a couple of metrics worth looking at in Percona XtraDB Cluster. Of course, as we stated at the beginning of this blog, those are metrics strictly related to PXC and Galera Cluster in general.

You should also keep an eye on regular MySQL and InnoDB metrics to get a better understanding of the state of your database. And remember, you can monitor this technology for free using the ClusterControl Community Edition.

We’re excited to announce the 1.8.1 release of ClusterControl - the only database management system you’ll ever need to take control of your open source database infrastructure. 

In addition to several new security enhancements, ClusterControl now supports the deployment, monitoring and management of Percona XtraDB Cluster 8 and MariaDB Cluster 10.5; both of which are built on the new Galera Cluster 4 technology which offers streaming replication and new synchronization functions.

This release also features the ability to cluster “fully-aware” ProxySQL nodes offering significant scalability of the popular database load balancing technology.

We are also rolling out a brand new backup and restore methodology for MongoDB which offers several advantages when backing up and restoring replica sets and sharded clusters.

Release Highlights

ProxySQL Clustering

• ProxySQL Clustering lets you have multiple ProxySQL instances which are natively clustered and aware of one other.

MongoDB Backup Management Improvements

• Added Percona Backup for MongoDB which allows you to backup and restore Replica sets and Sharded Clusters

Security Improvements

• Improved prevention for Clickjacking
• Updated JQuery to version 3.5.0 which features several new security fixes
• Implemented CGI Generic Cross-Site Request Forgery Detection. 

New Database Version Support

• Support for Percona XtraDB Cluster 8.0
• Support for MariaDB Cluster 10.5

View Release Details and Resources

• Download ClusterControl
• ChangeLog
• ClusterControl Administration Guide
• Upgrade Instructions

Release Details

ProxySQL Clustering

In April 2020, ProxySQL released version 1.4.2 which introduced the ability to natively cluster ProxySQL nodes. This feature has now been added to ClusterControl. It allows multiple ProxySQL instances to be aware of one other and let you create users and query rules that are instantly propagated to all the other nodes in the cluster.

MongoDB Backup Management

With the implementation of Percona Backup for MongoDB ClusterControl now has the ability to backup and restore both non-sharded replica sets and sharded clusters. 

Security Improvements

Cross-Site Request Forgery (CSRF) is an attack that forces an end user to execute unwanted actions on a web application in which they’re currently authenticated. We have implemented the ability to detect these types of attacks improving database security. 

Clickjacking is the malicious practice of manipulating a website user's activity by concealing hyperlinks beneath legitimate clickable content, thereby causing the user to perform actions of which they are unaware. We have added new protections from this type of attack in this 

JQuery implemented a security fix in version 3.5.0 after an issue was reported that demonstrated the regex could introduce a cross-site scripting (XSS) vulnerability. ClusterControl has been improved to take advantage of this new version.

New Database Support

Galera Cluster 4, which is the basis of the technology behind both Percona XtraDB Cluster 8 and MariaDB Cluster 10.5, includes several game-changing features as part of its release. The addition of Streaming Replication allows the database to better handle large transactions by fragmenting the transitions and operations and replicating them to the slave nodes while the process is still going on.  This prevents crashes when a transaction or large query becomes too large for the normal process to complete. The release also contains a series of new synchronization functions as well as the addition of new system tables to improve the database monitoring ability. 

ClusterControl now allows you to deploy this new technology with it’s point-and-click interface, reducing the complexity of deploying and configuring the, traditional challenging, database stack.

ProxySQL is a well known load balancer in MySQL world - it comes with a great set of features that allow you to take control over your traffic and shape it however you see it fit. It can be deployed in many different ways - dedicated nodes, collocated with application hosts, silo approach - all depends on the exact environment and business requirements. The common challenge is that you, most of the cases, want your ProxySQL nodes to contain the same configuration. If you scale out your cluster and add a new server to ProxySQL, you want that server to be visible on all ProxySQL instances, not just on the active one. This leads to the question - how to make sure you keep the configuration in sync across all ProxySQL nodes?

You can try to update all nodes by hand, which is definitely not efficient. You can also use some sort of infrastructure orchestration tools like Ansible or Chef to keep the configuration across the nodes in a known state, making the modifications not on ProxySQL directly but through the tool you use to organize your environment.

If you happen to use ClusterControl, it comes with a set of features that allow you to synchronize the configuration between ProxySQL instances but this solution has its cons - it is a manual action, you have to remember to execute it after a configuration change. If you forget to do that, you may be up to a nasty surprise if, for example, keepalived will move Virtual IP to the non-updated ProxySQL instance.

None of those methods is simple or 100% reliable and the situation is when the ProxySQL nodes have different configurations and might be potentially dangerous.

Luckily, ProxySQL comes with a solution for this problem - ProxySQL Cluster. The idea is fairly simple - you can define a list of ProxySQL instances that will talk to each other and inform others about the version of the configuration that each of them contains. Configuration is versioned therefore any modification of any setting on any node will result in the configuration version being increased - this triggers the configuration synchronization and the new version of the configuration is distributed and applied across all nodes that form the ProxySQL cluster.

The recent version of ClusterControl allows you to set up ProxySQL clusters effortlessly. When deploying ProxySQL you should tick the “Use Native Clustering” option for all of the nodes you want to be part of the cluster.

Once you do that, you are pretty much done - the rest happens under the hood.

MySQL [(none)]> select * from proxysql_servers;

+------------+------+--------+----------------+

| hostname   | port | weight | comment        |

+------------+------+--------+----------------+

| 10.0.0.131 | 6032 | 0      | proxysql_group |

| 10.0.0.132 | 6032 | 0      | proxysql_group |

+------------+------+--------+----------------+

2 rows in set (0.001 sec)

On both of the servers the proxysql_servers table was set properly with the hostnames of the nodes that form the cluster. We can also verify that the configuration changes are properly propagated across the cluster:

We have increased the Max Connections setting on one of the ProxySQL nodes (10.0.0.131) and we can verify that the other node (10.0.0.132) will see the same configuration:

In case of a need to debug the process, we can always look to the ProxySQL log (typically located in /var/lib/proxysql/proxysql.log) where we will see information like this:

2020-11-26 13:40:47 [INFO] Cluster: detected a new checksum for mysql_servers from peer 10.0.0.131:6032, version 11, epoch 1606398059, checksum 0x441378E48BB01C61 . Not syncing yet ...

2020-11-26 13:40:49 [INFO] Cluster: detected a peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060, diff_check 3. Own version: 9, epoch: 1606398022. Proceeding with remote sync

2020-11-26 13:40:50 [INFO] Cluster: detected a peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060, diff_check 4. Own version: 9, epoch: 1606398022. Proceeding with remote sync

2020-11-26 13:40:50 [INFO] Cluster: detected peer 10.0.0.131:6032 with mysql_servers version 12, epoch 1606398060

2020-11-26 13:40:50 [INFO] Cluster: Fetching MySQL Servers from peer 10.0.0.131:6032 started. Expected checksum 0x441378E48BB01C61

2020-11-26 13:40:50 [INFO] Cluster: Fetching MySQL Servers from peer 10.0.0.131:6032 completed

2020-11-26 13:40:50 [INFO] Cluster: Fetching checksum for MySQL Servers from peer 10.0.0.131:6032 before proceessing

2020-11-26 13:40:50 [INFO] Cluster: Fetching checksum for MySQL Servers from peer 10.0.0.131:6032 successful. Checksum: 0x441378E48BB01C61

2020-11-26 13:40:50 [INFO] Cluster: Writing mysql_servers table

2020-11-26 13:40:50 [INFO] Cluster: Writing mysql_replication_hostgroups table

2020-11-26 13:40:50 [INFO] Cluster: Loading to runtime MySQL Servers from peer 10.0.0.131:6032

This is the log from 10.0.0.132 where we can clearly see that a configuration change for table mysql_servers was detected on 10.0.0.131 and then it was synced and applied on 10.0.0.132, making it in sync with the other node in the cluster.

As you can see, clustering ProxySQL is an easy yet efficient way to ensure its configuration stays in sync and helps significantly to use larger ProxySQL deployments. Let us know down in the comments what your experience with ProxySQL clustering is.

ClusterControl is a platform for monitoring and managing open source databases. It will provide you with a  single pane of glass to understand what is happening in the system, it will let you know via alerts that something is wrong and it will give you tools to ensure that you can bring the state of the cluster back to normal. There are still some functionalities that are not available directly in ClusterControl, one of them is managing the on call rotation and working with alerts. That’s why ClusterControl gives you an option to integrate with external tools and extend what you can achieve using those dedicated tools. One of the integrations that are directly supported in ClusterControl is OpsGenie - an oncall management tool. Let’s take a look at how we can easily integrate ClusterControl with it.

There are a couple of prerequisites on the OpsGenie part that we will not be going through. Basically, you have to have teams defined with members. You may also want to have an on call rotation in place - in general, you want to have OpsGenie configured to your liking. Once it’s done, the next step will be to setup the integration in ClusterControl.

You can do it in the “Integrations” section in the left hand side menu - if you don’t have any existing integration, you will see the screen exactly as on the screenshot above. Just click on “Add your first service” and we can proceed further.

You want to pick OpsGenie from the list of the integrations.

Then we fill the data - integration name, region in which our OpsGenie setup is working as well as the name of the teams that should get the notifications. We should also fill in the API key for the team that we can create in OpsGenie - instructions that you can see on the screenshot above are quite clear and should be enough to get this done.

In short, in the “Teams” menu we picked our team and then used “Integrations” to add new API integration. Then we can get the API key to use with ClusterControl.

Once we set up everything in ClusterControl we can test this by clicking on “Test”. If everything is ok, you will see a notification. Please keep in mind that you have to fill all the forms here, we removed the API key only for the purpose of getting this screenshot.

As a next step we have to decide which alerts will be sent to OpsGenie. We can pick all or only some of the clusters defined in ClusterControl.

We can also pick the events that we want to be sent to OpsGenie based on their severity and category.

Once it is done, we can see our integration added into ClusterControl.

Now, let’s see if it actually works for real. For that we are going to simulate a master failure on our MariaDB 10.5 replication cluster. We will kill mariadbd process:

root@vagrant:~# killall -9 mariadbd

root@vagrant:~# killall -9 mariadbd

mariadbd: no process found

Next, we’ll wait a bit and see what has been sent to OpsGenie:

As you can see on the screenshot above, multiple alerts have been opened. Some of them have been cleared but some are still open. Those are related to the failed master. We can check the state of the cluster in ClusterControl.

As you can see, we have a failed master running as a node with read_only=ON. It is not a part of the replication topology. What we can do here is to bring it back into the replication chain as a slave. This step can happen automatically if you want but the default behavior of ClusterControl allows you to investigate the failed master before it is rebuilt. In this case we will trigger the rebuild process manually.

You can trigger it from Nodes Actions menu.

We picked the master node to get the data from and clicked “Proceed”.

After a bit of time, which depends on the size of the data as well as disk and network speed, the job should complete and we will see a new, clean replication topology.

In the meantime, in OpsGenie, we see that all of the alerts are now cleared and closed as the situation is back to normal.

Conclusion

As you can see, the integration with OpsGenie is really easy to set up and it will allow your team to develop and manage the on call rotation while using ClusterControl to manage your databases.

Everybody has a secret, and ClusterControl is not the exception… ClusterControl has a powerful, easy-to-use, and friendly UI, but using this UI is not the only way to manage your clusters. ClusterControl has a CLI called s9s, which is even more powerful than the UI, having features that are not available there yet. In this blog, we will show you how to use the ClusterControl CLI and what are the secret things that you can do using it.

ClusterControl CLI

Also known as s9s, is a command-line tool introduced in ClusterControl version 1.4.1 to interact, control, and manage database clusters using the ClusterControl system. ClusterControl CLI opens a new door for cluster automation where you can easily integrate it with existing deployment automation tools like Ansible, Puppet, Chef, etc. The command-line tool is invoked by executing a binary called s9s added by default in the ClusterControl installation.

You can find more information in the official documentation or even by running the s9s command with the help parameter:

$ s9s --help

Usage:

  s9s COMMAND [OPTION...]

Where COMMAND is:

    account - to manage accounts on clusters.

      alarm - to manage alarms.

     backup - to view, create and restore database backups.

    cluster - to list and manipulate clusters.

 controller - to manage Cmon controllers.

        job - to view jobs.

maintenance - to view and manipulate maintenance periods.

   metatype - to print metatype information.

       node - to handle nodes.

    process - to view processes running on nodes.

replication - to monitor and control data replication.

     report - to manage reports.

     script - to manage and execute scripts.

     server - to manage hardware resources.

      sheet - to manage spreadsheets.

       user - to manage users.

…

Also, the s9s tool has a man page for each command to get more detailed information.

$ man s9s node

Now you know what s9s is, let’s what kind of things are available here that are not possible in the ClusterControl UI.

Cluster Tagging

When managing a large number of database clusters you need to be able to find the one you are looking for quickly and easily. Cluster Tagging allows you to add keyword tags to each cluster (e.g. Tag by department, project, application, server, cloud provider, etc). You can then look up clusters by the keyword tags.

A cluster can be created with tags using the --with-tags option:

$ s9s cluster --create --cluster-name="TAGGED_CLUSTER" --cluster-type=galera --provider-version="10.4" --vendor=mariadb  --nodes="mysql://10.10.10.10:3306" --os-user=centos --with-tags="CLUSTER1;PRODUCTION1" --log

Or you can tag an existing cluster:

$ s9s tree --add-tag --tag="PRODUCTION1" /TAGGED_CLUSTER

Then, you can filter by using the created tags:

$ s9s cluster --list --long --with-tags="PRODUCTION1"

Vault Integration

Vault by HashiCorp is a service that allows you to keep your sensitive data secure, even when deployed across multiple servers and clouds. ClusterControl can integrate directly with the Vault APIs from the ClusterControl CLI, allowing you to shift from the standard methods of security to a more dynamic infrastructure as defined by ephemeral applications and servers, trusted sources of user and application identity, and software-based encryption. 

You can find more details about this feature in the corresponding blog post.

Enabling ClusterControl HA

ClusterControl HA, also known as CMON HA, is not enabled by default for compatibility reasons. To implement this, you will need to run an s9s controller command with the “--enable-cmon-ha” parameter.

When the CMON HA is enabled, CmonController class objects will be created and used to implement the high availability features. So, if the controller has at least one CmonController object, the CMON HA is enabled.

$ s9s controller --enable-cmon-ha

The CMON HA is now enabled, but there is no redundancy, only one controller is running. The one existing controller in this stage should be a leader although there are no followers.

Now, you will need to install additional ClusterControl servers one by one. The next controllers should use the same CMON Database and should have the same configuration files. When the additional controllers are started, they will find the leader in the CMON Database and will ask the leader to let them join. When the join is successful, one more CmonController will be created for every joining controller.

You can find more information about this feature in the two-part blog series here: Part 1 and Part 2.

Abort/Restart Jobs

You can use the ClusterControl CLI to abort a running job. This command will send a signal to a running ClusterControl Job in order to abort the execution. The job only will be aborted if and when the job supports aborting what it is doing.

First, you will need to get the job id that is necessary to kill the job.

$ s9s job −−list

When you have it, just run the following command:

$ s9s job --kill --job-id=1172

When it is killed, you will see the ABORTED status in the job list.

Another option about the jobs is to create a copy of an existing job to re-run it. The cloned job will have all the properties that the original job had and will be executed in the same way.

$ s9s job --clone --job-id=1172 --log

Enabling SSL for MongoDB

In case you have a MongoDB cluster in place, and you want to enable SSL, it can be easily done using the ClusterControl CLI. You will need the cluster id number, and run the following command:

$ s9s cluster --cluster-id=22 --enable-ssl --log 

When it finishes, you will receive the message “SSL setup done”.

Note: You can get the cluster id by running the following command:

$ s9s cluster --list --long

Limitations

Currently, the s9s command-line tool has a user management module that is not yet fully integrated with the ClusterControl UI and the ClusterControl Controller. For example, there is no Role-Based Access Control for a user. This means that any user created to be used with the s9s command-line tool has full access to all clusters managed by the ClusterControl server.

Users created by the command line client will be shown in Job Messages, but it is not possible to use this user to authenticate and login from the UI. Also, the users created from the command line client are all super admins.

Conclusion

ClusterControl has a powerful, easy-to-use, and friendly UI, but that is not all the power of ClusterControl. It could be possible you need to use the command line to integrate ClusterControl with an existing tool or even for automation purposes. 

The ClusterControl CLI (s9s) allows you to use the same feature available in the UI, and even more, and it can be integrated with external tools like Ansible, Puppet, etc.

In this blog, we showed you how to use the ClusterControl CLI and what kind of things you can do using it that are not available in the UI yet. Most of the things that we mentioned in this blog will be integrated with the ClusterControl UI soon in a future release.

Monitoring the performance of MySQL databases is a daily task of almost every MySQL DBA. In this blog post we will try to demonstrate how to monitor the performance of your MySQL databases - with ClusterControl.

What Do You Need to Monitor?

In general, if you’re monitoring MySQL performance, you should probably consider monitoring the following things:

• Query Management - monitoring the performance of your queries can prove to be one of the most efficient ways to improve MySQL performance.
• Query Outlier Detection - historical data can tell us a lot: it can even be used to calculate expected query performance based on historical data, then compare the current query performance against the expected performance to detect query outliers.
• Database Schema Performance - ClusterControl can help you optimize the performance of your MySQL instances by providing schema insight to ensure your schemas are optimized for your workload.
• Load Balancing - load balancers like ProxySQL, HAProxy or MaxScale can make your database perform at the best of its ability. Those load balancers can easily be deployed using ClusterControl.

Database Load Balancing and Query Management Using ClusterControl

ClusterControl supports the following load balancers:

• HAProxy
• MariaDB MaxScale
• ProxySQL
• Keepalived (virtual IP address only)

Do note that ClusterControl must reside on an independent node, separate from your database cluster.

In regards to query management, ClusterControl allows you to absorb the following (you can see these things once you have switched to the Query Monitor tab):

• You can view top slow and long-running queries.
• You can view a list of currently running queries.
• You can view a list of query outliers.

This tab provides you with an aggregated list of all your top queries running on all of the nodes of your database cluster. ClusterControl gets the information in one of these two ways:

1. Queries are either retrieved from PERFORMANCE_SCHEMA or
2. If PERFORMANCE_SCHEMA is disabled or unavailable, ClusterControl will parse the content of the Slow Query Log.

If the Slow Query Log is in use, only queries that exceed the Long Query Time will be listed here.

Long Query Time collects queries that take longer than Long Query Time seconds to execute (for example, if the Long Query Time is 0.1 only queries that take more than 0.1 seconds to execute will be logged), “log queries not using indexes?” configures the behavior of ClusterControl in regards to queries without indexes etc.

The top queries table is automatically refreshed every 30 seconds (the refresh rate can be changed). The top queries table columns contain a list of sampled queries, database names, the total number of query occurrences, the number of rows involved in operations, the number of temporary tables created for a certain query etc. - the list of the aggregated top queries can also be ordered by Occurence or Execution Time to show either the most common or the slowest queries.

ClusterControl also allows you to view a list of currently running queries. This page is also automatically refreshed every 30 seconds and it displays:

• The MySQL server from which the process is retrieved.
• The connection ID.
• The database name.
• The MySQL user who issued the statement.
• The MySQL thread execution time in seconds.
• The hostname of the client issuing the statement.
• The type of command the thread is executing.
• The state of the thread (as explained in the MySQL documentation)

The query outliers page shows queries that are regarded as “outliers”. In other words, this page shows all queries that take more time to execute than a normal query of that type, longer being defined as a latency deviation of 2 sigmas + average_query_time. This page displays:

• The exact time the query was captured.
• The query itself.
• The time that the SQL query took to execute.
• The average execution time of the SQL query.
• The standard deviation execution time (microseconds).
• The query’s maximum execution time (microseconds).
• The query’s lock time (microseconds).

The query monitor tab provides a summary of query processing across all nodes in the cluster.

Summary

When it comes to monitoring the performance of your MySQL databases, ClusterControl can do wonders. ClusterControl can make the deployment of load balancers simple, it can help you easily manage your queries and monitor their performance, ClusterControl can also calculate expected query performance based on historical data.

SSL encryption is clearly one of the must-haves for database deployments. It is required by all security standards and even if your organization does not have to comply with PCI DSS, HIPPA or other similar standards, you probably still want to have your traffic secured properly.

ClusterControl provides you with options to manage SSL on the clusters it manages. 

On the screen above we can see that SSL encryption is enabled for both frontend connections and intra-cluster connections in the MariaDB Galera Cluster. Nowadays, due to the SSL being the de facto standard, modern versions of the databases come with self-signed, ready certificates and SSL enabled. This is great if you would like to ensure that your traffic is secured. On the other hand, if you would like to add another layer of security, you may want to use your own certificates, ensuring that only nodes that you set up can attempt to connect to the database. ClusterControl has you covered and in this blog post we will take a look at the SSL certificate management options that are available.

First of all, if you want to rotate SSL certificates, you can easily do it from ClusterControl. Just click on the “Change Certificate” and you will be presented with two options:

You can create a new, self-signed certificate or you can use existing certificates that you have already imported or created. If you’ll go with the creation of a new certificate, you will be asked couple of questions:

You should decide on the certificate expiration date and answer if you want the new certificate to be applied immediately or not. Applying it requires a rolling restart of the Galera Cluster.

Alternatively, you can go to the Key Management section of ClusterControl, where you can manage SSL certificates.

As you can see, there is a list of certificates in use by the Galera Cluster. You can also generate or import new certificates.

It can be done in a couple of ways. Let’s start with creating a CA and key. This step is not required, you can as well create a self-signed certificate.

After CA is created, we can create the server and client certificates.
Once that is completed, you will see the certificates appear on the list on the left. Such certificates can be used later.

If you have existing certificates, self-signed or not, you can as well import them into ClusterControl and use them with the SSL management that is available.

Let’s say we have a new certificate, either created or imported, and we want to use it for SSL encryption of the traffic between Galera nodes. All you need to do is to click on the “Change Certificate”.

As a next step, you should pick that you will be using an existing certificate.

Finally, you can pick the certificate from the list. That’s pretty much it. Click on the Save Changes button and ClusterControl will start the process of changing the certificate. Please keep in mind that all changes in the SSL certificate settings  will require nodes to be restarted.

After some time the job is completed and we can verify that the certificate is indeed in use:

As you can see, SSL certificate management in ClusterControl is quite easy and straightforward. It allows you to use your own certificates as well as generating new ones. All of that makes it easy to ensure that your data is properly encrypted in transit.

If you are dealing with a bunch of virtual machines, using Vagrant is probably the best tool to build, automate and manage the virtual machines to increase efficiency and productivity, especially in a development environment. ClusterControl can also be configured to run on Vagrant and can be automated easily using the Vagrantfile together with an automation script. 

Vagrant supports a number of virtualization platforms; e.g., VirtualBox, KVM, Hyper-V, Docker containers, VMware and AWS. In this example, we are going to use VirtualBox as the virtual machine provider and use Vagrant to provision the virtual machine host automatically.

Download VirtualBox from here, and download Vagrant from here. Install them into your workstation and you are good to go.

Deploying ClusterControl on Vagrant

To deploy a ClusterControl virtual machine, simply create the following files:

• Vagrantfile - Virtual machine definition and specification
• deploy-cc.sh - A bash script to install ClusterControl

Create a directory to store the Vagrant definition file:

$ mkdir ~/vagrant/clustercontrol

$ cd ~/vagrant/clustercontrol

Create the Vagrantfile definition file:

$ vim Vagrantfile

And add the following lines:

Vagrant.configure("2") do |config|

  config.vm.box = "centos/7"

  config.vm.network "forwarded_port", guest: 80, host: 8080

  config.vm.network "forwarded_port", guest: 443, host: 8443

  config.vm.network :private_network, ip: "192.168.11.11"

  config.vm.hostname = "cc.local"

  config.vm.provision "shell" do |s|

    s.inline = "yum install -y vim wget"

  end

  config.vm.provision "shell", path: "deploy-cc.sh"



  config.vm.provider :virtualbox do |vb|

    vb.customize [

      "modifyvm", :id,

      "--memory", "4096",

    ]

  end

end

Create the deployment script, deploy-cc.sh in the same directory. This script will be executed automatically every time the virtual machine starts up:

$ vim deploy-cc.sh

And add the following lines:

CC_IP=192.168.11.11 # IP address of the ClusterControl VM

CMON_PASS='cmonP455&' # CMON user password

ROOT_PASS='S3cr3t&' # MySQL root password

MYSQL_PORT=3306



# only install ClusterControl if cmon does not exist

if [ ! -e /usr/sbin/cmon ]; then

# download installer script

sudo wget severalnines.com/downloads/cmon/install-cc

# allow executable

sudo chmod 755 install-cc

# execute the installation

sudo S9S_CMON_PASSWORD=${CMON_PASS} S9S_ROOT_PASSWORD=${ROOT_PASS} S9S_DB_PORT=$MYSQL_PORT HOST=${CC_IP} ./install-cc

# generate an SSH key for user vagrant

ssh-keygen -t rsa -N '' -C vagrant@cc -f /home/vagrant/.ssh/id_rsa

# assign correct ownership to the private and public keys

sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa

sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa.pub

# optional - copy the public key to Apache docroot so others can download it for authorization

cat /home/vagrant/.ssh/id_rsa.pub | sudo tee /var/www/html/cc.pub

# optional - assign correct ownership to the public key

sudo chown -f apache:apache /var/www/html/cc.pub

fi

We are now ready to start our first instance. We can check the VM status by using the following command:

$ vagrant status

To create and start the VM, run the following command:

$ vagrant up

You should see a lot of lines coming up on the screen indicating the VM is being prepared by Vagrant. Wait until the process completes, and you should be able to access the ClusterControl UI at http://127.0.0.1:8080/ or https://127.0.0.1:8443/ of your local machine. Port 8080 is for plain HTTP connection while 8443 is for HTTPS, as defined inside the Vagrantfile under "forwarded_port" lines.

Deploying Database Clusters on Vagrant

Now we have the ClusterControl provisioned and ready to manage and monitor our development database cluster environment. Creating multiple virtual machines using Vagrant is easy using the Vagrantfile. In this example, we would like to deploy a 3-node Galera Cluster running on Percona XtraDB Cluster 8.0 as illustrated in the following diagram:

The cluster consists of 3 nodes of database nodes and 2 nodes of ProxySQL nodes for MySQL load balancing. This whole set can be considered as 1 group of nodes for this cluster. Therefore, we are going to define the VMs specification in one single Vagrantfile. Firstly, create a directory for our database cluster:

$ mkdir ~/vagrant/clustercontrol/percona-xtradb-cluster-80/

$ cd ~/vagrant/clustercontrol/percona-xtradb-cluster-80/

Then, create a Vagrantfile definition using your preferred text editor for our 3-node database cluster and 2-node ProxySQL cluster:

$ vim ~/vagrant/clustercontrol/percona-xtradb-cluster-80/Vagrantfile

And add the following lines:

nodes = [

  { :hostname => 'db1', :ip => '192.168.11.21', :ram => 1024, :guestport => 3306, :hostport => 13306 },

  { :hostname => 'db2', :ip => '192.168.11.22', :ram => 1024, :guestport => 3306, :hostport => 23306 },

  { :hostname => 'db3', :ip => '192.168.11.23', :ram => 1024, :guestport => 3306, :hostport => 33306 },

  { :hostname => 'proxy1', :ip => '192.168.11.26', :ram => 512, :guestport => 6033, :hostport => 16033 },

  { :hostname => 'proxy2', :ip => '192.168.11.27', :ram => 512, :guestport => 6033, :hostport => 26033 }

]



Vagrant.configure("2") do |config|

  nodes.each do |node|

    config.vm.define node[:hostname] do |nodeconfig|

      nodeconfig.vm.box = 'centos/7'

      nodeconfig.vm.hostname = node[:hostname] + ".local"

      nodeconfig.vm.network :private_network, ip: node[:ip]

      nodeconfig.vm.network "forwarded_port", guest: node[:guestport], host: node[:hostport]



      memory = node[:ram] ? node[:ram] : 512;

      nodeconfig.vm.provider :virtualbox do |vb|

        vb.customize [ "modifyvm", :id, "--memory", memory.to_s ]

        vb.customize [ "modifyvm", :id, "--audio", "none" ]

      end



      nodeconfig.vm.provision "shell" do |s|

        s.inline = "yum install -y wget vim"

      end

    end

  end



  (1..3).each do |i|

    config.vm.define "db#{i}" do |dbnode|

      dbnode.vm.provision "shell", path: "authorize.sh"

    end

  end



  (1..2).each do |i|

    config.vm.define "proxy#{i}" do |dbnode|

      dbnode.vm.provision "shell", path: "authorize.sh"

    end

  end



end

In the same directory, create a shell script called authorize.sh and add the following lines:

CC_IP=192.168.11.11

curl -s http://${CC_IP}/cc.pub >> /home/vagrant/.ssh/authorized_keys

This script will pull the public key from the ClusterControl server (as configured inside deploy-cc.sh when we provision our ClusterControl VM), and store it inside the authorized key list for the OS user "vagrant". This is basically to automatically configure the passwordless SSH which is mandatory for the nodes to be managed by ClusterControl.

To create and start the VMs for our cluster, run the following command:

$ vagrant up

Wait for a moment until all VMs are up and running. You can verify with the vagrant status command:

$ vagrant status

Once done, it's time to deploy our Percona XtraDB Cluster using ClusterControl. Open ClusterControl at http://127.0.0.1:8080/clustercontrol and go to Deploy -> MySQL Galera and enter the following details:

We specified vagrant as the SSH user because we have auto-authorized the public key in every nodes' authorized_keys list when provisioning the VMs using Vagrant using the authorize.sh script. Note that every Vagrant address VM for Virtualbox will have at least two network interfaces. With VirtualBox, Vagrant requires the first network device attached to the virtual machine to be a NAT device. The NAT device is used for port forwarding, which is how Vagrant gets SSH access to the virtual machine. Our IP address as defined in the Vagrantfile will be added as in interface eth1, eth2 and so on, which is crucial when defining them under the "Define MySQL Servers" section as shown below:

When specifying the IP address in the second step above, use the IP address of what we have defined inside Vagrantfile (which is equivalent to eth1's IP address of the created VM). The green tick next to the IP address indicates the node is reachable via passwordless SSH from the ClusterControl node. Finally, click "Deploy" and wait until the deployment completes.

You should see the following once the deployment job is complete:

Deploying Load Balancers on Vagrant

Similarly, for load balancers, we have already prepared the virtual machine using Vagrant with the proper IP address and passwordless SSH. To deploy a ProxySQL, go to ClusterControl -> Manage -> Load Balancers -> ProxySQL -> Deploy ProxySQL and specify the first load balancer, 192.168.11.26 as below:

Do not forget to toggle on the "Use Native Clustering" since we are going to have more than one ProxySQL host running. Click on the "Deploy ProxySQL" button to start the load balancer deployment. You can monitor the job progress under ClusterControl -> Activity -> Jobs. Repeat the above steps for the second load balancer, 192.168.11.27. Finally, we can tie these load balancers together with a virtual IP address (a floating IP address for a single endpoint) using Keepalived by going to ClusterControl -> Manage -> Load Balancers -> Keepalived -> Deploy Keepalived and choose both of the load balancers with a virtual IP address (192.168.11.20) on the interface eth1, as shown below:

Our database cluster deployment with a high-availability setup similar to our production cluster is now complete.

To shut down all the database nodes, we can just simply run the following command in the active directory:

$ cd ~/vagrant/clustercontrol/percona-xtradb-cluster-80

$ vagrant halt

To shut down the ClusterControl node, navigate back to the ClusterControl directory and run the same command:

$ cd ~/vagrant/clustercontrol

$ vagrant halt

Substitute the "vagrant halt" command with the "vagrant destroy" command if you want to destroy the virtual machines (usually for housekeeping purposes) and you can always re-create it back later using the "vagrant up" command.

A hybrid cloud database is a mix of active hosts residing on both on-premises and cloud infrastructures, working as a single operational system. It helps implement a high availability environment, and since both database and applications need to be portable in order to work on on-prem and cloud, it also reduces dependency on a particular cloud vendor (cloud vendor lock-in). Deploying this kind of environment is not that straightforward though.

In this blog post, we are going to look into using ClusterControl to Deploy a Hybrid Cloud MySQL Database, between on-premises and cloud infrastructure running on AWS. This setup allows us to bring the database closer to the clients and applications residing in a cloud environment, and we will be keeping a replica of the database on the on-premises for disaster recovery and live backup purposes.

Architecture 

Our deployment consists of four nodes - One node for ClusterControl and one node for master server other two nodes are in slave servers, where 2 of the database nodes are located in the AWS data center as the primary datacenter (as well as ClusterControl), while the third node is located in the on-premises datacenter. The following diagram illustrates our final architecture.

Preparing Your Cloud Environment 

First, you need to create your environment with your main Cloud Provider. In this case, we will use AWS with three EC2 instances.

Then, go to the on-premises environment and create one virtual machine or host that will be the standby node.

ClusterControl - 52.14.222.75

Master - 3.17.190.53

Slave 1 - 3.14.146.131

Slave 2 - 192.168.33.15

We will assume you have a ClusterControl installation running and have already created Cloud Provider accounts. Note that we are using cloud instances with public IPs for the purpose of the blog. For a production-grade setup, you have to secure your environment, for instance cloud VMs should be within a VPC with just private IP addresses. You would have a secure connection between on-prem and the VPC. 

Deploying MySQL using ClusterControl 

Go to your ClusterControl GUI, and select the option “Deploy”. If you already have a MySQL instance running, then you need to select the “Import Existing Server/Database” instead.

Here we are going to do a new deployment so we are selecting the “Deploy” option.When selecting MySQL, you must specify User, Key or Password, and port to connect by SSH to your MySQL nodes. 

For more details about the SSH connectivity please check the ClusterControl user requirements.

After setting up the SSH access, you must define the database user and version. Datadir is optional. You can also specify which repository to use. Here we are choosing MySQL 8.0 version.

Make sure you get a green tick icon indicating ClusterControl is able to connect to the database nodes to perform the deployment. Click on the "Deploy" button to start the deployment and you can monitor the deployment progress under Activity -> Jobs -> Create Cluster. You can monitor the creation status in the ClusterControl activity monitor.

Once the task is finished, you can see your new MySQL setup in the main ClusterControl screen.

And check your final topology in the Topology View Section.

Conclusion 

ClusterControl features will allow you to quickly set up replication in a Hybrid Cloud environment, between different Cloud Providers, or even between a cloud provider and an On-prem environment. About the communication between the cloud providers, or between private and public cloud, for security reasons, you must restrict the traffic only from known sources, to reduce the risk of unauthorized access to your network.

Disaster recovery has many means and can be implemented in many ways. The one thing to keep in mind is that when you are designing a production environment, you should always consider how you are going to recover from a serious incident. What options are available - that mostly depends on the business requirements. If you have to maintain availability through all scenarios, your options are limited and are more expensive. On the other hand, if you can afford a bit of downtime, the range of solutions you can apply significantly increases. In this blog post, we would like to discuss a bit how hybrid cloud setups change your disaster recovery plans.

Two Versus One

Let’s assume you have an on-prem setup. You are owning a datacenter or you are renting servers in an external DC. Whenever you introduce a public cloud in the mix and connect your own network to the secured network in your public cloud, the main, obvious change is that you now have an environment spanning across two locations. This is a giant leap forward in terms of disaster recovery handling. With one location the best you can hope for is to ship your backups somewhere outside of the original datacenter: an off-site backup server, uploading the data to another hosting provider or some sort of a file store, just like S3 in AWS. This also means that the recovery time will be quite long - if your main datacenter would become unavailable, the only course of action for you would be to use the backup you have taken earlier and attempt to rebuild your whole environment on some other hosting provider.

Including public cloud in your environment ensures that you have two locations that you can use for storing the nodes. Having nodes in more than one location means that even if the on-prem side of the network is impacted, you still have working databases in the public cloud part of the network. Those databases may be used to provide service continuity - you can promote one of them to become writable, you can use the data they contain to scale out the public cloud environment and move your operations, temporarily at least, there.

Velocity

Traditional, on-prem, setups, it is not so easy to scale out. You are limited by how fast you can have new hardware ordered and delivered. Depending on the situation this may vary. It could be days if you are ordering the hardware on your own, it can be hours if you put an order with the hosting provider who has the hardware in stock and the only thing that remains is to put it together and install it in the RAC.

Public cloud is years ahead compared to what you can accomplish with “standard” hosting providers. Of course, there might be edge cases but typically you can spin up new infrastructure in a matter of minutes. This makes it a great choice for the disaster recovery scenarios - you do not have to pre-allocate the hardware but you can rely on the availability of the resources in the cloud to quickly build up your infrastructure to production-grade size even if you would start with a skeleton environment.

You can resize instances, you can add more of them.

All of it is typically possible in a matter of minutes. The speed at which you can build up the environment leads us to another section.

Flexibility

With a public cloud integrated into your environment, you have plenty of options how to use it. What’s great, given the velocity in which you can deploy the infrastructure, you can experiment with different setups and you don’t have to commit to long contracts, what is typical for “traditional” providers. Do you predict your workload will increase temporarily? You can scale out vertically or horizontally using the resources in the cloud. Do you want to test different topologies or different deployment patterns for your disaster recovery processes? More small instances or few larger ones? No problem, just do that. Of course, if you can commit to a given set of resources, you can benefit from reduced expenses by contracting them for a longer period of time - this is also possible in the public cloud. Still, if you do not want to do it, you are free to experiment with different options before deciding on whatever works best for you.

As you can see, adding public cloud into the mix and creating a Hybrid Cloud, significantly increases your options and allows you to find the best setup faster and with lower costs. If you have experience working in such a setup and you’d like to share it with others, feel free to add a comment below.

Having a Hybrid Cloud Environment is a common topology for a Disaster Recovery Plan (DRP), or even to have redundancy on your systems, but there are some security considerations to take into account before implementing this kind of environment that could be a limitation in some cases.

In this blog, we will see what a Hybrid Cloud Environment is and how to keep it as secure as possible.

What is Hybrid Cloud?

It is a topology that uses a mix of private, and public cloud, and even on-premises services. It sounds similar to a Multi-Cloud environment, but the main difference is that this concept refers to the combinations of public and private specifically, which could include on-prem too.

Securing Communications

For security reasons, the communication between the Cloud Providers, the Cloud Provider and On-Prem, or between Private and Public Cloud must be encrypted, and you must restrict the traffic only from known sources to reduce the risk of unauthorized access to your network.

The usage of VPN, SSH, or Firewall Rules, or even a combination of them, is a must for this point.

Controlling Database Access

You should always restrict both physical and remote access.

• Physical access (On-prem): Restrict unauthorized physical access to the database server.

• Remote access: Limit the remote access to only the necessary people, and from the less amount of source possibles. Using a VPN to access it is definitely useful here, but there are also other options like SSH Tunneling.

Managing Database User Accounts

Depending on the technology, there are many ways to improve security for your user accounts.

• Remove inactive users.

• Grant only the necessary privileges.

• Restrict the source for each user connection.

• Define a secure password policy (or, depending on the technology, enable a plugin for this if there is one available).

Secure Installations and Configurations

There are some changes to do to secure your database installation.

• Install only the necessary packages and services on the server.

• Change the default admin user password and restrict the usage from only the localhost.

• Change the default port and specify the interface to listen in.

• Enable password security policy plugin.

• Configure SSL certificates to encrypt data-in-transit.

• Encrypt data-at-rest.

• Configure the local firewall to allow access to the database port only from the local network, or from the corresponding source.

Implement a WAF (Web Application Firewall)

SQL Injections or DoS attacks (Denial of Service) are the most common attacks to a database, and the most secure way to avoid them is by using a WAF to catch this kind of SQL queries or a SQL Proxy to analyze the traffic.

Keep Your OS and Database Up-to-Date

There are several fixes and improvements that the database vendor or the operating system release in order to fix or avoid vulnerabilities. It is important to keep your system as up-to-date as possible by applying patches and security upgrades.

Check CVE (Common Vulnerabilities and Exposures) Frequently

Every day, new vulnerabilities are detected for your database server. You should check it frequently to know if you need to apply a patch or change something in your configuration. One way to know it is by reviewing the CVE website, where you can find a list of vulnerabilities with a description, and you can look for your database version and vendor, to confirm if there is something critical to fix ASAP.

Conclusion

Your Hybrid Cloud Environment will be safer by checking the points mentioned above, but unfortunately, there is always a risk of being hacked as there is not a 100% secured system.

The key here is to minimize this risk, and for that, you should run periodically security scan tools like Nessus, looking for vulnerabilities, and to have a good monitoring system like ClusterControl, that not only allow you to have your system monitored, but also auto recover your systems in case of failure, or even quickly set up replication in a Hybrid Cloud environment, between different Cloud Providers, or even between a Cloud Provider and an On-prem environment and manage the setup in an easy and friendly way.

MySQL 5.6 has reached its EOL in February 2021, and we recommend that you start to plan migrating to MySQL 8.0. The latest version of MySQL is supported until April 2026 and keeps your database features up to date with continuously receiving updates and fixes, especially the security patches.

The next release of ClusterControl, will no longer provide a deployment option for MySQL5.6. However, we will continue to support and provide support for MySQL5.6 users on bugs and fixes related to ClusterControl for a stipulated time. 

Here are some related MySQL 8.0 blogs:

• Monitoring & Ops Management of MySQL 8.0 with ClusterControl

• What to Monitor in MySQL 8.0

• What’s New With MySQL Replication in MySQL 8.0

• MySQL in 2018: What’s in 8.0 and Other Observations

• Tips for Upgrading to from MySQL 5.7 to MySQL 8

• Moving from MySQL 5.7 to MySQL 8.0 - What You Should Know

With the EOL of MySQL5.6, there will be no more updates on bug fixes or security patches. Please refer to MySQL for requirements and technical compatibility before decommissioning your application from MySQL5.6.  Stay tuned with us for more updates on our latest release.

Hybrid cloud can be a great way of adding flexibility to your existing on-prem deployments. As we discussed in several blogs, public cloud can be a great addition to your own datacenter, ensuring you can easily scale out to handle the load, reduce your capex and be used to implement disaster recovery procedures. Security is another aspect you have to think through when you plan to build such systems. In this blog post, we will talk about some of the security considerations for hybrid cloud MariaDB deployments.

Connectivity

VPN

The major part of every hybrid infrastructure is the network. After all, we are talking about two environments, local, on-premises and a public cloud, that have to be connected and form a single entity. The connection has to be encrypted. How to approach it, there are numerous ways to do so.

One of them would be to use a solution that is made available by the cloud provider - most of them have some sort of a connectivity option available. It can be AWS Direct Connect if you happen to integrate with Amazon Web Services. If you plan to use Google Cloud, solutions are discussed on the following web site: https://cloud.google.com/hybrid-connectivity. In short, there are a significant number of different options that vary from hardware VPN integration to set up BGP peering.

On the other side of the spectrum, we have software VPN solutions. OpenVPN or similar kind of software can be used to set up a secure, encrypted network connection between your own datacenter and the public cloud. In such a case you would require a separate instance running in the public cloud that will be used for the VPN server. Utilizing software VPNs allows you to pick the solution that best suits your requirements and fits the best in your environment.

Firewall

Databases should never be accessible from external networks. It is paramount to build your environment in a way that the database tier will be reachable only from the limited set of hosts. Exactly what is required and how to do that, it is up to you to decide. A typical setup would consist of a secured database tier that can be accessed only from proxy tier and, if needed, some sort of a jump host should be implemented if required for automation and administration tasks. 

Application servers should not have direct access to the database - they do not need to. All the application should be required to do is to connect to the load balancer. Load balancers should be able to connect to the database. A load balancer like ProxySQL is perfectly capable of performing the read/write split and sending the reads and writes to the correct database nodes. The application should be able to connect to the ProxySQL and the rest will be handled by the proxy - database authentication, traffic shaping, distributing the traffic across numerous replicas that you may have. All unnecessary access should be restricted. Security groups, firewalls - these are the tools you want to use to secure your environment. 

In short, access to the database hosts should be allowed only on the required ports. For MariaDB it will be, obviously, a port used for the database but also other ports if needed - you may have some sort of exporters or agents installed. For Galera, you would need to open ports for intra-cluster communication. You may also want to have an open port for SSH connections. Ideally, limit the access on a per-host basis; only a limited set of hosts can access a given port. For example, the database port might be accessible from the other database nodes, localhost and proxy layer. There is no need to keep it open for other nodes. Your database nodes may even be located on a separate subnet, ensuring that the security is even tighter.

As for ports, best practices would be to change them from the default settings to something else. Ideally, something random. Changing SSH port from 22 to 2222 or MariaDB port from 3306 to 33306 may help to avoid some of the automated attacks but it can still be figured out if someone is actively looking to get into your network. If you want better security, you may just go ahead with some random values. Set SSH to 5762 and MariaDB to 24359. It is quite likely that no one will be able to guess those. Set your TCP timeouts so that the port scans would be very lengthy and expensive and this will surely increase your chances.

SSL

In addition to VPN and firewall, you should ensure that your database traffic is encrypted using SSL.

Ideally, you will protect both frontend connections (from the load balancers) and the communication between your database nodes (be it replication or the intra-cluster transfer in Galera clusters). ClusterControl can help you to enable those options with just a few clicks.

All you need to do is to let ClusterControl create new certificates or use one of the existing ones - you can import your own certificate if you want. Having SSL enabled ensures that the database traffic won’t be readable even by someone who gained access to your network.

Database Security

Of course, the network is not the only important aspect of security. Yes, it is critical, especially in the hybrid cloud environment, but there are also other very important aspects. One of them is the access control embedded into MariaDB.

Role-Based Access Control for MariaDB

MariaDB comes with a set of instruments to ensure that the database access is properly managed and restricted wherever it’s required. The first line of authentication is users. Everyone and everything that is allowed to access MariaDB should use an assigned user to connect to the database. Such users will have a proper password - you can have the password validation enabled in MariaDB to ensure that the passwords are strong enough. Ideally, you would limit the user’s access host only to hostnames or IP’s of load balancers - this should always be the way users connect to the database. For some administrative users, you may want to keep the localhost access if required. On top of enforcing the proper password strength you can configure the password to expire within some period of time or enforce password rotation on the users. As you can imagine, a proper password rotation policy is something you will want to have implemented.

Every user in MariaDB can have multiple privileges assigned. Privileges can be assigned on several levels - global level, database level, table level or even column level. The best practice is to grant a limited set of privileges to the users as possible. If the user requires only access to a particular table, just grant him that. There is no need for that user to access other tables not to mention other schemas. You can define quite detailed access rights using a large set of privileges that you can grant to users. It ranges from rights to read, update or delete data through database management privileges up to the “super” privilege that allows user to perform actions like managing the replication threads and bypass read_only setting.

On top of that, MariaDB comes with roles - to make user management easier it is possible to define roles with a given set of granted privileges and then assign those roles to the users. Such users will inherit grants related to the role it has been assigned to, making it way easier to manage grants on the large scale: instead of changing the grants for multiple users you can assign them to one specific role and then manage all their privileges by altering the privileges granted to the role they have been assigned to.

You should also ensure that you do not have any pre-existing users without an assigned password or with a too large set of privileges. Such security audit should be performed from time to time, ensuring that you are aware of potential security risks and you can plan to act on them.

Audit log

If your database comes with audit log, just like MariaDB does, you should consider using it to track the actions that are happening in the database. The audit log will help you to accomplish that. With it enabled you will be able to track even the details like which user executed what query. If you happen to use ClusterControl you can enable the audit log just with a couple of clicks:

To sum up this blog, there are a couple of things you should consider when designing a MariaDB deployment in the hybrid cloud environment. Some of them are strictly related to the way the environment is designed, some are pretty much related to the database type that you use and the fact that you use hybrid cloud doesn’t really change much. What is really important is to ensure that your database is properly protected - that’s the ultimate goal, no matter what the environment is.

The COVID-19 pandemic has had its fair share of influence on world media but of interest is the sheer amount of cyber attacks that have escalated since it began. Interestingly enough, is news of a 667 percent rise in Coronavirus-Related attacks since the beginning of March 2020. Downplaying the possibility of being attacked is far from the wisdom gained from corporations who have spent millions in setting up security controls, yet they still succumb to data breaches. This goes right down to medium and small businesses that have had their databases wiped off by ransomware. A point to note is that having an 'Assume Breach' mentality is ideal in all situations as you are devising your security strategies. At the point that your data gets stolen, you need to have been ready. 

Backing up your database stands out as one of the most fundamental steps you need to get ready. On top of that is the question of whether your backup can stand the test of time and security. And in that case, the exception rather than the norm is to ensure that you backup regularly, that your backups are secure and available.

Confidentiality, Integrity, Availability of Your Backups

Of most importance to consider for backup security is your business, security, and/or your compliance goal. With backups, as it is with your general security, it's always a matter of striking a balance between confidentiality, integrity, availability. Not forgetting, other security considerations like identification, authentication, authorization, and auditing; essentially aspects of access control.

For example, when it comes to considering the privacy of Personally Identifiable Information (PII) in your backups, all the above aspects (confidentiality, availability, and integrity) are equally important to consider since privacy breaches may be attributed to the lack of any of the aspects thereof. 

Backups Without ClusterControl

Open-source databases offer numerous tools that can help with doing backups. Below is a list:

You can decide to use either of the tools above depending on your needs; that is, according to the kind of database/cluster you are using. The drawback of using these native backup tools is the cost of maintenance that you may have to endure. ClusterControl offers numerous advantages over using these native tools as we shall see. To get more information about how you can utilize some of these tools please take a peek in our DevOps Guide to Database Backups for MySQL and MariaDB whitepaper.

Backup Management with ClusterControl

It is essential to have backups as a key operational aspect of your database management. Reasons being, incidents such as ransomware attacks, system crashes, hardware failures, power failures, or human errors (both accidental or intentional). ClusterControl offers numerous possibilities for your database security which is inclusive of being able to manage your database backups. With it, you have all the open-source backup tools wrapped in to make it as easy and convenient to plan and manage your backups for the open-source database/cluster of your choice. You can : 

1. Get an overview of all your Backups

2. Create/Schedule a Backup

3. Encrypt, Compress and Archive Offsite

4. Define Retention policies

5. Receive Backup Notifications

6. Verify/Restore Backups

Overview of all your Backups

You can get a glimpse of all your scheduled backups by navigating to Backups Tab. While on that tab, you can immediately be able to view the status of all your backups. You can further select the Create Backup, Restore Backups, Scheduled Backups or Settings Tabs as will be described in the next sections.

Create/Schedule a Backup

If you need to create/schedule a backup on ClusterControl, it is as easy as navigating from Backups -> Create Backup -> Schedule Backup and you will have the menu below:

Encrypt, Compress and Archive Offsite

AES-256-CBC is an encryption algorithm that has been proven to have sufficient strength for encrypting blocks of data such as database backups.  ClusterControl can help you achieve encryption for your database using this algorithm by navigating from ClusterControl -> Select your MariaDB Cluster -> Security -> SSL Encryption -> Enable.

Aside from encrypting your database, you can also encrypt your backups.  As you are creating your backup,  as described in the create backups section above, you will go through the first step of scheduling your backups, next on will be a section that will require you to specify the backup settings. In that section, you can toggle the options to encrypt and compress your database backups as shown in the diagram in the section below.

A point to note though is that encryption in and of itself doesn’t protect you from privacy breaches as a result of ransomware. An encrypted backup or encrypted data could also be stolen or re-encrypted by an attacker which could escalate into a breach if there are insufficient redundancies to ensure that the personal/critical data in question is available.

Define Retention Policies

You also have the option of specifying the backup retention period for your backups. By default the retention period is 31 days but you could specify a custom retention period if you wish.

Post-Backup Logs and Notifications 

Doing backups is not a fire-and-forget kind of activity, you will need to keep track of the status of your ongoing backups and a means to verify if your backups are running as scheduled. Using ClusterControl provides you with job logs, as shown below, that you can check through to keep track of the status of your log activity. 

Further to checking through job logs, you can also create email notifications that will provide you with the status of your backups. To create a notification go to Settings -> General Settings -> Email Notification Settings -> Backup:

Deliver option means that you will get notification immediately. You can also select Ignore or Digest and in that case, you will get a daily summary of alarms raised.

Backup Verification through Restoration

Based on the status of your notification, you need to verify if you can restore the backup. You can verify using the Restore and verify on a standalone option as shown below:

Final Thoughts

Backups are essential, although there is much more to consider aside from merely keeping backups. Proper planning should be done prior to starting any backup schedule. As part of the backup planning, you should think about the kind/nature of your data you are handling and if that imposes any regulations, privacy, or security compliance needs. If so, it may lead to further considerations such as encryption, the number of backup redundancies and possibly retention requirements.