We’ve had Zabbix templates for ClusterControl available for three years now, so that Zabbix is able to connect to ClusterControl and retrieve monitoring data as well as alerts. We recently rewrote major parts of the templates to use the ClusterControl RPC interface.

In this blog post, we’ll have a look at what’s new. Note that we provide a bunch of other integrations with third party tools that you can take advantage of.

What has Changed?

In the older version, the Zabbix plugin made use of the ClusterControl CMONAPI to retrieve monitoring data, process the output and make it understandable to the Zabbix agent. Since version 1.3, ClusterControl switched to a new interface called CMON RPC which listens on port 9500 on the ClusterControl node (9501 if you run on TLS). This new API is an improved version of the CMONAPI interface. So the agent script now communicates with ClusterControl through this RPC interface. If you are on version 1.3 and later, you should upgrade to this new script as the old one will not work anymore. Some system calls in CMONAPI have been completely removed and replaced by CMON RPC.

Another significant change is you don't need to configure the RPC URL and token directly, as the script will automatically read the configuration options directly from ClusterControl's bootstrap.php file (default to /var/www/html/clustercontrol/bootstrap.php). Due to these changes, we deprecated another 2 files inside this template, which are clustercontrol.conf (configuration file for CMONAPI URL and token) and clustercontrol_api.sh (curl wrapper to connect to CMONAPI). The same dependencies are still required, like php-curl and php-cli, as explained in the readme file of this template.

Monitoring Data

The ClusterControl template returns the following monitoring data from multiple clusters:

• Database cluster’s status - Report the cluster state - active, failed, degraded and unknown.
• ClusterControl alarms (critical/warning) - Report the number of ClusterControl alarms based on severity.

As well as ClusterControl related services:

• ClusterControl controller service (cmon)
• ClusterControl web SSH service (cmon-ssh)
• ClusterControl notification service (cmon-events)

It also supports multiple clusters, where you can specify the components and clusters to be monitored by the agent. You may skip some clusters (e.g, test cluster) for alerting by simply excluding the cluster ID from the list. The previous version had backup status as part of the monitoring data. However in this version, backup is a sub-component of alarms so monitoring the alarms should be enough.

Example Deployment

Installing the template should be pretty straightforward. Consider the following setup:

Our setup consists of a Galera Cluster, a standalone PostgreSQL server and two MySQL Replication nodes, all managed by a ClusterControl instance. The Zabbix agent will be installed on the ClusterControl node, and it uses some reporting scripts to talk to ClusterControl to retrieve monitoring data. The Zabbix server will use a ClusterControl template to talk to the Zabbix agent.

Zabbix Agent

1. Installation instructions can be found on our GitHub repository page. To get the template, just clone the s9s-admin repository:

   $ git clone https://github.com/severalnines/s9s-admin

2. Create a template directory for ClusterControl under /var/lib/zabbix and copy the scripts directory into it:

   $ mkdir -p /var/lib/zabbix/clustercontrol
   $ cp -Rf ~/s9s-admin/plugins/zabbix/agent/scripts /var/lib/zabbix/clustercontrol

3. Copy the ClusterControl template user parameter file into /etc/zabbix/zabbix.agent.d/:

   $ cp -f ~/s9s-admin/plugins/zabbix/agent/userparameter_clustercontrol.conf /etc/zabbix/zabbix.agent.d/

4. This template uses the ClusterControl CMON RPC interface to collect stats. The script will copy /var/www/html/clustercontrol/bootstrap.php into the template directory to read ClusterControl configuration options. If you are running on a non-default path for the ClusterControl UI, configure the exact path manually inside clustercontrol_stats.php, similar to the example below:

   $BOOTSTRAP_PATH = '/var/www/html/clustercontrol/bootstrap.php';

5. Test the script by invoking a cluster ID and test argument:

   $ /var/lib/zabbix/clustercontrol/scripts/clustercontrol_stats.sh 1,2,3,4,5 test
   Cluster ID: 1, Cluster Name: MariaDB 10.1, Cluster Type: galera, Cluster Status: STARTED
   Cluster ID: 2, Cluster Name: PostgreSQL, Cluster Type: postgresql_single, Cluster Status: STARTED
   Cluster ID: 3, Cluster Name: MySQLRep, Cluster Type: replication, Cluster Status: STARTED
   Cluster ID 4 not found.
   Cluster ID 5 not found.

   ** This example shows that the ClusterControl instance has 3 clusters, although we defined 5 cluster IDs in the command line.

   You should get an output of your database cluster summary, indicating the script is able to retrieve information using the provided ClusterControl RPC interface with the correct token in bootstrap.php.

6. Finally, restart the Zabbix agent:

   $ service zabbix-agent restart

Installation of the Zabbix agent is now complete.

Zabbix Server

1. Download the Zabbix template file from here to your desktop.

2. Import the XML template using the Zabbix UI (Configuration -> Templates -> Import). Verify if the import is OK by going to Configuration -> Templates -> ClusterControl Template -> Items and you should see something like this:

3. Create/edit hosts and link them to the template "ClusterControl Template" (Configuration -> Hosts -> choose a host -> Templates tab):

You are done. The following is what you can expect to see in your Zabbix frontend UI if something goes wrong:

Summary

ClusterControl has a sophisticated alarming system. In cases where Zabbix is the main monitoring and alerting tool, this simple integration will unify your database alerts into Zabbix.

We would be happy to continue and improve this template, do let us know what kind of information that you would like to see in Zabbix in the comments section below.

Join us on November 14th 2017 as we combine forces with the Codership Galera Cluster Team to talk about how to manage Galera Cluster using ClusterControl!

Galera Cluster has become one of the most popular high availability solution for MySQL and MariaDB; and ClusterControl is the de facto automation and management system for Galera Cluster.

We’ll be joined by Seppo Jaakola, CEO of Codership - Galera Cluster, and together, we’ll demonstrate what it is that makes Galera Cluster such a popular high availability solution for MySQL and MariaDB and how to best manage it with ClusterControl.

We’ll discuss the latest features of Galera Cluster with Seppo, one of the creators of Galera Cluster. We’ll also demo how to automate it all from deployment, monitoring, backups, failover, recovery, rolling upgrades and scaling using the new ClusterControl CLI.

Sign up below!

Date, Time & Registration

Europe/MEA/APAC

Tuesday, November 14th at 09:00 GMT / 10:00 CET (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, November 14th at 09:00 PT (US) / 12:00 ET (US)

Register Now

Agenda

• Introduction
  • About Codership, the makers of Galera Cluster
  • About Severalnines, the makers of ClusterControl
• What’s new with Galera Cluster
  • Core feature set overview
  • The latest features
  • What’s coming up
• ClusterControl for Galera Cluster
  • Deployment
  • Monitoring
  • Management
  • Scaling
• Live Demo
• Q&A

Speakers

Seppo Jaakola, Founder of Codership, has over 20 years experience in software engineering. He started his professional career in Digisoft and Novo Group Oy working as a software engineer in various technical projects. He then worked for 10 years in Stonesoft Oy as a Project Manager in projects dealing with DBMS development, data security and firewall clustering. In 2003, Seppo Jaakola joined Continuent Oy, where he worked as team leader for MySQL clustering product. This position linked together his earlier experience in DBMS research and distributed computing. Now he’s applying his years of experience and administrative skills to steer Codership to a right course. Seppo Jaakola has MSc degree in Software Engineering from Helsinki University of Technology.

Krzysztof Książek, Senior Support Engineer at Severalnines, is a MySQL DBA with experience managing complex database environments for companies like Zendesk, Chegg, Pinterest and Flipboard.

Many thanks to everyone who participated in this week’s webinar on how to manage MongoDB and Percona Server for MongoDB! The replay is now available for viewing online.

For this webinar we’d teamed up with Percona’s Tyler Duzan, Product Manager, who talked to us about some of the key features and aspects of Percona Server for MongoDB.

And our colleague Ruairi Newman compared the MongoDB-relevant functionality of MongoDB’s Ops Manager and ClusterControl. Participants learned about the differences between these systems, and how they help automate and manage MongoDB operations.

View the replay to find out more

Percona Server for MongoDB is a fully compatible, open source, drop-in replacement for the MongoDB®Community Server and provides MongoDB© Enterprise Edition features at no licensing cost, with additional storage engines and performance improvements.

ClusterControl is the all-inclusive management system for open source databases.

With Percona Server for MongoDB and Severalnines ClusterControl together, users benefit from à cost-efficient solution with additional features and capabilities.

View the replay to learn more

Agenda

• Introduction to Percona Server for MongoDB
• How to automate and manage MongoDB
  • Installation and maintenance
  • Complexity of architecture
  • Options for redundancy
  • Comparative functionality
  • Monitoring, Dashboard, Alerting
  • Backing up your deployments
  • Automated deployment of advanced configurations
  • Upgrading existing deployments

Speakers

Ruairí Newman is passionate about all things cloud and automation and has worked for MongoDB, VMware and Amazon Web Services among others. He has a background in Operational Support Systems and Professional Services.

Prior to joining Severalnines, Ruairí worked for Huawei Ireland as Senior Cloud Solutions Architect on their Web Services project, where he advised on commodity cloud architecture and Monitoring technologies, and deployed and administered a Research & Development Openstack lab.

Prior to joining Percona as a Product Manager, Tyler Duzan spent almost 13 years as an operations and security engineer in a variety of different industries. Deciding to take his analytical mindset and strategic focus into new territory, Tyler is applying his knowledge to solving business problems for Percona customers with inventive solutions combining technology and services.

Load balancers are an essential component in MySQL and MariaDB database high availability; especially when making topology changes transparent to applications and implementing read-write split functionality.

HAProxy is free, open source software that provides a high availability load balancer and proxy server for TCP and HTTP-based applications that spreads requests across multiple servers.

ClusterControl provides support for deployment, configuration and optimization of HAProxy as well as for other popular load balancing and caching technologies for MySQL and MariaDB databases.

Here are our top resources for HAProxy to get you started with this widely used technology.

Tutorials

MySQL Load Balancing with HAProxy - Tutorial

We have recently updated our tutorial on MySQL Load Balancing with HAProxy. Read about deployment and configuration, monitoring, ongoing maintenance, health check methods, read-write splitting, redundancy with VIP and Keepalived and more.

Read More

On-Demand Webinars

How to deploy and manage HAProxy, MaxScale or ProxySQL with ClusterControl

In this webinar we talk about support for proxies for MySQL HA setups in ClusterControl: how they differ and what their pros and cons are. And we show you how you can easily deploy and manage HAProxy, MaxScale and ProxySQL from ClusterControl during a live demo.

Watch the replay

How To Set Up SQL Load Balancing with HAProxy

In this webinar, we cover the concepts around the popular open-source HAProxy load balancer, and shows you how to use it with your SQL-based database clusters.

Watch the replay

Performance Tuning of HAProxy for Database Load Balancing

This webinar discusses the performance tuning basics for HAProxy and explains how to take advantage of some of the new features in 1.5, which was released in June 2014 after 4 years of development work.

Watch the replay

Introducing the Severalnines MySQL© Replication Blueprint

The Severalnines Blueprint for MySQL Replication includes all aspects of a MySQL Replication topology with the ins and outs of deployment, setting up replication, monitoring, upgrades, performing backups and managing high availability using proxies as ProxySQL, MaxScale and HAProxy. This webinar provides an in-depth walk-through of this blueprint and explains how to make best use of it.

Watch the replay

Top Blogs

HAProxy Connections vs MySQL Connections - What You Should Know

Max connections determines the maximum number of connections to the database server. This can be set on both the database server, or the proxy in front of it. In this blog post, we’ll dive into HAProxy and MySQL maximum connections variables, and see how to get the best of both worlds.

Read More

SQL Load Balancing Benchmark - Comparing Performance of MaxScale vs HAProxy

In a previous post, we gave you a quick overview of the MaxScale load balancer and walked through installation and configuration. We did some quick benchmarks using sysbench, a system performance benchmark that supports testing CPU, memory, IO, mutex and also MySQL performance. We will be sharing the results in this blog post.

Read More

Load balanced MySQL Galera setup - Manual Deployment vs ClusterControl

Deploying a MySQL Galera Cluster with redundant load balancing takes a bit of time. This blog looks at how long it would take to do it manually vs using ClusterControl to perform the task.

Read More

Read-Write Splitting for Java Apps using Connector/J, MySQL Replication and HAProxy

In this blog post, we will play around with Java and MySQL Replication to perform read-write splitting for Java Apps using Connector/J.

Read More

High availability read-write splitting with php-mysqlnd, MySQL Replication and HAProxy

In this blog post, we explore the use of php-mysqlnd_ms with a PHP application (Wordpress) on a standard MySQL Replication backend.

Read More

Become a ClusterControl DBA: Making your DB components HA via Load Balancers

There are various ways to retain high availability with databases. You can use Virtual IPs (VRRP) to manage host availability, you can use resource managers like Zookeeper and Etcd to (re)configure your applications or use load balancers/proxies to distribute the workload over all available hosts.

Read More

Wordpress Application Clustering using Kubernetes with HAProxy and Keepalived

In this blog post, we’re going to play with Kubernetes application clustering and pods. We’ll use Wordpress as the application, with a single MySQL server. We will also have HAProxy and Keepalived to provide simple packet forwarding (for external network) with high availability capability.

Read More

How Galera Cluster Enables High Availability for High Traffic Websites

This post gives an insight into how Galera can help to build HA websites.

Read More

ClusterControl is infrastructure-agnostic - it can be used in your own datacenter on physical hosts, as well as in virtualized cloud environments. All you need is ssh access from the ClusterControl host to the database nodes, and you can then deploy standalone/replicated/clustered MySQL/MariaDB, MongoDB (replica sets or sharded clusters) or PostgreSQL (streaming replication). In this blog post, we will walk you through the steps to deploy ClusterControl on EC2.

Setting up instances in EC2

The hardware requirements for ClusterControl are described here. Those are meant to create a performant platform for the ClusterControl server. Having said that, we will use a small instance for our testing purposes (t2.micro) - it should be enough for us.

First, we need to pick an AMI. ClusterControl supports:

• Redhat/CentOS/Oracle Linux 6 and later
• Ubuntu 12.04/14.04/16.04 LTS
• Debian 7.0 and later

We are going to use Ubuntu 16.04.

Next step - instance type. We will pick t2.micro for now, although you will want to use larger instances for production setups. For other cloud providers, pick instances with at least 1 GB of memory.

We are going to deploy four instances at once, one for ClusterControl and three for Percona XtraDB Cluster. You need to decide where those instances should be deployed (VPC or not, which subnet etc). For our testing purposes, we are going to use a VPC and a single subnet. Of course, deploying nodes across the subnets (Availability Zones) makes your setup more likely to survive if one of the AZ’s would become unavailable.

For storage we’ll use 100GB of general purpose SSD volume (GP2). This should be enough to perform some tests with a reasonable volume of data.

Next - security groups. SSH access is a requirement. Other than that, you need to open ports required by the database you plan to deploy. You can find more information on which ports are required in our support portal.

Finally, you need to either pick one of the existing key pairs or you can create a new one. After this step your instances will be launched.

Once the instances are up and running, it’s time to install ClusterControl. For that, log into one of the instances and download the ClusterControl installation script, install-cc:

ubuntu@ip-172-30-4-20:~$ wget http://www.severalnines.com/downloads/cmon/install-cc
--2017-09-06 11:13:10--  http://www.severalnines.com/downloads/cmon/install-cc
Resolving www.severalnines.com (www.severalnines.com)... 107.154.146.155
Connecting to www.severalnines.com (www.severalnines.com)|107.154.146.155|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.severalnines.com/downloads/cmon/install-cc [following]
--2017-09-06 11:13:10--  https://www.severalnines.com/downloads/cmon/install-cc
Connecting to www.severalnines.com (www.severalnines.com)|107.154.146.155|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://severalnines.com/downloads/cmon/install-cc [following]
--2017-09-06 11:13:11--  https://severalnines.com/downloads/cmon/install-cc
Resolving severalnines.com (severalnines.com)... 107.154.238.155, 107.154.148.155
Connecting to severalnines.com (severalnines.com)|107.154.238.155|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56913 (56K) [text/plain]
Saving to: ‘install-cc’

install-cc                                           100%[=====================================================================================================================>]  55.58K   289KB/s    in 0.2s

2017-09-06 11:13:12 (289 KB/s) - ‘install-cc’ saved [56913/56913]

Then, make sure it can be executed before running it:

ubuntu@ip-172-30-4-20:~$ chmod +x install-cc
ubuntu@ip-172-30-4-20:~$ sudo ./install-cc

At the beginning, you’ll get some information about the requirements on supported Linux distributions:

!!
Only RHEL/Centos 6.x|7.x, Debian 7.x|8.x, Ubuntu 12.04.x|14.04.x|16.04.x LTS versions are supported
Minimum system requirements: 2GB+ RAM, 2+ CPU cores
Server Memory: 990M total, 622M free
MySQL innodb_buffer_pool_size set to 512M

Severalnines would like your help improving our installation process.
Information such as OS, memory and install success helps us improve how we onboard our users.
None of the collected information identifies you personally.
!!
=> Would you like to help us by sending diagnostics data for the installation? (Y/n):

This script will add Severalnines repository server for deb and rpm packages and
install the ClusterControl Web Applicaiton and Controller.
An Apache and MySQL server will also be installed. An existing MySQL Server on this host can be used.

At some point you will have to answer some questions about hostnames, ports and passwords.

=> The Controller hostname will be set to 172.30.4.20. Do you want to change it? (y/N):
=> Creating temporary staging dir s9s_tmp

=> Setting up the ClusterControl Web Application ...
=> Using web document root /var/www/html
=> No running MySQL server detected
=> Installing the default distro MySQL Server ...
=> Assuming default MySQL port is 3306. Do you want to change it? (y/N):

=> Enter the MySQL root password:
=> Enter the MySQL root password again:
=> Importing the Web Application DB schema and creating the cmon user.

=> Importing /var/www/html/clustercontrol/sql/dc-schema.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
=> Set a password for ClusterControl's MySQL user (cmon) [cmon]
=> Enter a CMON user password:
=> Enter the CMON user password again: => Creating the MySQL cmon user ...
mysql: [Warning] Using a password on the command line interface can be insecure.
=> Creating UI configuration ...

Finally, you’ll get the confirmation that ClusterControl has been installed. Install script will also attempt to detect your public IP and print out a link that can be used in your browser to access ClusterControl.

=> ClusterControl installation completed!
Open your web browser to http://172.30.4.20/clustercontrol and
enter an email address and new password for the default Admin User.

Determining network interfaces. This may take a couple of minutes. Do NOT press any key.
Public/external IP => http://34.230.71.40/clustercontrol
Installation successful. If you want to uninstall ClusterControl then run install-cc --uninstall.

Once the installation is done, there’s still one thing to take care of - SSH access from ClusterControl to the remaining hosts. Unless you already have SSH access between nodes (and you can use ssh-copy-id), this will be a manual process. First and the foremost, we need to generate a new ssh key:

root@ip-172-30-4-20:~# ssh-keygen -C 'galera_cluster' -f id_rsa_galera -t rsa -b 4096
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_galera.
Your public key has been saved in id_rsa_galera.pub.
The key fingerprint is:
SHA256:2tWOGXbrtc0Qh45NhNPzUSVdDzE9ANV1TJ0QBE5QrQY galera_cluster
The key's randomart image is:
+---[RSA 4096]----+
|         .o=**X*&|
|         Eo  + BO|
|          ..+ +.o|
|           + o +.|
|        S = o + o|
|       o o * * o |
|      . . o + =  |
|           . . = |
|            . . o|
+----[SHA256]-----+
root@ip-172-30-4-20:~#

We can verify it has been created correctly. You will also want to copy the contents of the public key - we will use it to create its copies on remaining nodes.

root@ip-172-30-4-20:~# cat id_rsa_galera.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDYKil17MzTrNc70GIQlVoK6xLop9acfT3W6kBUGO4ybsvIA5Fss+WvT/DLsYMtukq2Eih93eO4amLRYQIeyWSjJ/bBwIF/LXL4v04GF8+wbDgCyiV/t9dSuXna9qFeawkUVcPjnmWnZqUoaP5QeovXTluxl54xEwbFm1oLT+fgWbaim5w9vVUK4+hAHaZ7wVvTPVsIt1B3nJgWJF0Sz+TJN87vSUg7xdshgzhapUotXlguFGVzmNKWLnEFDCK7RT41oh4y4rkKP7YLc+wFfRHYTnKyMIcf0/0VMyL+2AdwQp8RThbBommf2HGimF1hSyA9/fc+tLi7FVTg1bKKeXj4hwexeFAJZwoy3HyD3wQ/NwadpDVk5Pg7YYzdN2aCZfvo27qp3gdQQ2H+LF6LvDyQEkgRpFN+pHoWQvPjJJasjfIcfdaC9WmDiL4s5fXyCTQz/x0NaTXVkLBS9ibfOUw8AGdd36FvdqnNOFOlMLKLa359JhdpqXnH7ksiThcotQuFmV5Dc8M66vTDz9rvVZhNC0nME478RNBP0Bgj1BM26XdQlzozeaRmHGoZXcSQVJTXBC93+QN4+bRmWmxhhj5G5M7bFiQyal1VtugoUt8ZV4NiiG+KDd6yj5um8+CffD/BASGrv3vffH+AK7xtjchIv5su40+unecfSOtO98TiQw== galera_cluster

Now, on every remaining node, you need to add this public key into the authorized_keys file. For ubuntu, you may want to clean its contents first if you want to use a root login. By default, only ubuntu user can be used to connect through SSH. Such setup (regular user and sudo) is also possible to use with ClusterControl but here we’ll go for a root user.

root@ip-172-30-4-198:~# vim ~/.ssh/authorized_keys

Once authorized_keys files on all nodes contain our public key, we will copy our public key to .ssh directory and make necessary changes in access rights:

root@ip-172-30-4-20:~# cp id_rsa_galera /root/.ssh/
root@ip-172-30-4-20:~# chmod 600 /root/.ssh/id_rsa_galera

Now we can test if SSH access works as expected:

root@ip-172-30-4-20:~# ssh -i /root/.ssh/id_rsa_galera 172.30.4.46
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1022-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


root@ip-172-30-4-46:~# logout
Connection to 172.30.4.46 closed.

All’s good. It’s time to configure ClusterControl.

Fill in some registration details.

Once you have logged in, a wizard will appear with an option to either deploy new cluster or import an existing one.

We want to deploy Percona XtraDB Cluster so we’ll go for “Deploy Database Cluster” and pick the “MySQL Galera” tab. Here we have to fill in access details required for SSH connectivity. We’ll set SSH User to root and we will fill in the path to our SSH key.

Next, we’ll define a vendor, version, password and IP addresses for our database hosts. Please keep in mind that ClusterControl will check SSH connectivity to the target database hosts. If everything works ok, you’ll see green ticks. If you see that SSH authentication failed, then you will need to investigate as the ClusterControl server is not able to access your database hosts.

Then, click on Deploy to start the deployment process.

You can track the deployment progress in the activity monitor.

Remember that deployment is only the first step. Operating a database requires you to monitor performance of your hosts, database instances, queries, and manage backups, fix failures and other anomalies, manage proxies, upgrades, etc. ClusterControl can manage all these aspects for you, so do give it a try and let us know how you get on.

Severalnines is excited to announce it’s newest customer VentraIP, an Australian-based web hosting, domain names, and SSL Certificate provider.

VentraIP Australia is the largest privately owned web host and domain name registrar in Australia, backed by a team of industry veterans and local technical professionals who ensure their 150,000 customers always get the best customer service and technical support 24 hours a day, 7 days a week.

In the case study, you can learn how VentraIP went from using a ill-performing MySQL standalone instance to power their front end and replaced it with Galera Cluster using ClusterControl to deliver a high-performance and redundant system that met their customers needs and the needs of the future.  

Read the case study to learn more.

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that let’s you automate many of the database tasks you have to perform regularly, like deploying new databases, detecting anomalies, recovering nodes from failures, adding and scaling new nodes, running backups and upgrades, and more.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

Today we are excited to announce the 1.5 release of ClusterControl - the all-inclusive database management system that lets you easily deploy, monitor, manage and scale highly available open source databases - and load balancers - in any environment: on-premise or in the cloud.

ClusterControl 1.5 provides an array of exciting new backup functionalities to ensure that your data is secure and available whenever disaster strikes. The release also provides expanded PostgreSQL, MariaDB and MySQL NDB Cluster support.

Release Highlights

Cloud Services (AWS S3 and Google Cloud Storage)

• Manual upload or schedule backups to be uploaded after completion to the cloud.
• Download and restore backups from a cloud storage.

Backup (MySQL)

• Backup individual databases separately (mysqldump only).
• Upload, download and restore backups stored in the cloud.
• Trigger a verification and restore of a backup after N hours of completion.
• Rebuild a replication slave by staging it from an existing backup.
• Add a new replication slave by staging it from an existing backup.

PostgreSQL

• New backup method pg_basebackup which makes a binary copy of the database files.
• Synchronous replication failover (support for synchronous_standby_names).
• Support for HAProxy with Keepalived.
• Support for PostgreSql 10.

ProxySQL

• Mass import existing database users into ProxySQL.
• Add and modify scheduler scripts.

Misc

• MariaDB v10.2 support (Galera and MySQL Replication).
• MySQL Cluster(NDB) v7.5 support.
• Added support to show and filter DB status variables for MongoDB nodes.
• HTML formatted alarm and digest emails.
• Multiple NICs supports when deploying Load balancers (HAProxy).
• Continuous improvements to UX/UI and performance.
• New cmon-cloud process and clud client to handle cloud services.
• New Ops Report: Database Growth

View Release Details and Resources

• 1.5 ChangeLog
• ClusterControl Administration Guide
• Upgrade instructions

New and Enhanced Backup Functions

ClusterControl 1.5 marks the first steps towards providing full seamless integration between ClusterControl and the Cloud. It provides our users with the options of migrating their data to the cloud while still benefiting from the features and functionality that ClusterControl has to offer.

• AWS & Google Cloud Services Integration: ClusterControl now offers full integration for database backups for Amazon Web Services (AWS) and Google Cloud Services. Backups can now be executed, scheduled, downloaded and restored directly from your cloud provider of choice. This ability provides increased redundancy, better disaster recovery options, and benefits in both performance and cost savings.
• New Backup Wizard: ClusterControl 1.5 comes with a complete redesign of our backup wizard. The wizard lets you perform and schedule backups using a variety of options and technologies to ensure your data is protected, accurate and ready to be restored on a moment's notice.
• Single Database Backup & Restores for MySQL: With ClusterControl 1.5 you now have the ability to backup and restore individual MySQL databases rather than being forced to do it all at one time. This new feature can reduce the burden on those users who manage hundreds of databases within their environment.
• Automatic Backup Verification: ClusterControl 1.5 brings you peace of mind by automatically verifying the integrity of your backup data. It will perform a restore of your backup data, and alert in case of issues.
• Creating & Restoring Slaves from Backup: In the past a user would need to leverage their Master to create or restore new slaves… this creates an undue burden on the system as it performs a full backup on the master. In ClusterControl 1.5, you can now create new nodes or rebuild existing slaves from existing backup files, removing the burden on the master and ensuring you continue to operate at peak performance.

Improved Support for PostgreSQL

ClusterControl continues to expand its support for PostgreSQL. In our new 1.5 release, we can now support PostgreSQL 10 which offers a number of new features such as support for local replication, declarative table partitioning, improved query parallelism, quorum commit for synchronous replication, and more!

• PostgreSQL Proxy and Virtual IP: ClusterControl now supports failover of database connections with HAProxy, Keepalived and Virtual IP - a battle-tested combination.
• PostgreSQL Synchronous Replication Failover: ClusterControl 1.5 now fully supports Synchronous Replication Failover for PostgreSQL. This function minimizes the risk of data loss between Master and Slaves should something break with the Master before it has a chance to fully synchronize. This allows you to increase reliability across your servers.
• New Backup Method: PostgreSQL has an additional backup method pg_basebackup that can be used for online binary backups. Backups taken with pg_basebackup can be used later for point-in-time recovery and as the starting point for a log shipping or streaming replication standby servers.

Improved Support for ProxySQL

ProxySQL enables MySQL, MariaDB and Percona XtraDB database systems to easily manage intense, high-traffic database applications without losing availability.

• Script Scheduling: ClusterControl now provides support for the ProxySQL scheduler, a cron-like module which allows the load balancer to start external scripts on a regular interval. This provides users with full control of what scripts run and when, allowing you to tweak your instances performance at a very granular level.
• User Management: Rather than create or removing users one at a time from ProxySQL, ClusterControl now allows you to import a large batch of users.

Support for New Versions of MariaDB & MySQL NDB Cluster

• MariaDB 10.2: ClusterControl now supports MariaDB Server 10.2. This new version provides even greater integration with Galera Cluster, MariaDB’s HA solution of choice, and also features enhancements to SQL like windows functions, common table expressions, and JSON functions.
• MySQL NDB Cluster 7.5: ClusterControl now supports MySQL Cluster (NDB) 7.5. The latest MySQL Cluster release using the NDB file system offers a variety of improvements and optimizations. These improvements include enhancing the ndbinfo database, records-per-key optimization, and many more!

Additional New Functionalities

• New Notification Emails: In the 1.5 release of ClusterControl, we have revamped the design of the email notifications and alerts which are sent from the system to make them easier to read and understand.
• New Operational Report: ClusterControl 1.5 introduces a new operational report called “Database Growth.” The report shows a snapshot of your database as it exists today as well as data for the last day, last week, and last month - useful for capacity planning. The report also allows you to click into each period to deep dive into the tables to track growth at an even more granular level.

As you might have seen in today’s announcement, ClusterControl now supports the latest versions of MariaDB, MySQL NDB Cluster and PostgreSQL; and introduces a series of new database backup functionalities that range from AWS & Google Cloud integration backup services to automatic backup verifications, making it ever more efficient to run a solid backup strategy for open source database infrastructures.

And with the release of ClusterControl 1.5, we’re also announcing a new webinar in partnership with MariaDB!

Join Johan Andersson, CTO at Severalnines, and Ralf Gebhardt, Product Manager at MariaDB, on December 5th as they unveil ClusterControl 1.5, the all-inclusive database management system that lets you easily deploy, monitor, manage and scale highly available open source databases - and load balancers - in any environment: on-premise or in the cloud.

We’ll have a particular focus on MariaDB 10.2, thanks to Ralf’s participation, who will talk us through the latest features, and give us a sneak preview of what to expect in MariaDB 10.3.

We’ll also look at our new operational reports and email notification features - all in a live demo that you don’t want to miss.

Sign up below!

Date, Time & Registration

Europe/MEA/APAC

Tuesday, December 5th at 09:00 GMT / 10:00 CET (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, December 5th at 09:00 PT (US) / 12:00 ET (US)

Register Now

Agenda

MariaDB 10.2: all the new features and a first look at MariaDB 10.3

• ClusterControl 1.5
  • What’s new:
    • MariaDB 10.2 support
    • AWS & Google Cloud services integration
    • Enhanced backup functions
  • New features & support for:
    • PostgreSQL
    • MySQL NDB Cluster
    • ProxySQL
  • Operational reports
• Live demo
• Q&A

Speakers

Johan Andersson, CTO, Severalnines - Johan's technical background and interest are in high performance computing as demonstrated by the work he did on main-memory clustered databases at Ericsson as well as his research on parallel Java Virtual Machines at Trinity College Dublin in Ireland. Prior to co-founding Severalnines, Johan was Principal Consultant and lead of the MySQL Clustering & High Availability consulting group at MySQL / Sun Microsystems / Oracle, where he designed and implemented large-scale MySQL systems for key customers. Johan is a regular speaker at MySQL User Conferences as well as other high profile community gatherings with popular talks and tutorials around architecting and tuning MySQL Clusters.

Ralf Gebhardt is Product Manager at MariaDB Corporation. He is responsible for MariaDB Server and MariaDB Connectors. He joined MariaDB/SkySQL in 2011 as Principal Sales Engineer.
After 10 years professional experience in Software Development, Support, Training and Consulting, He started working at MySQL GmbH as Sales Engineer in 2002. In the course of the acquisition of Sun Microsystems he joined Oracle, still responsible for MySQL.
He holds a masters degree in Computer Engineering from the University of Cooperative Education (in cooperation with IBM Deutschland).

At the core of ClusterControl is its automation, as is ensuring that your data is is securely backed up and ready for restoration whenever something goes wrong. Having an effective backup strategy and disaster recovery plan is key to the success of any application or environment.

In our latest release, ClusterControl 1.5, we have introduced a number of enhancements for backing up MySQL and MariaDB-based systems.

One of the key improvements is the ability to backup from ClusterControl to the cloud provider of your choice. Cloud providers like Google Cloud Services and Amazon S3 each offer virtually unlimited storage, reducing local space needs. This allows you to retain your backup files longer, for as long as you would like and not have concerns around local disk space.

Let’s explore all the exciting new backup features for ClusterControl 1.5...

Backup/Restore Wizard Redesign

First of all, you will notice backup and restore wizards have been revamped to better improve the user experience. It will now load as a side menu on the right of the screen:

The backup list is also getting a minor tweak where backup details are displayed when you click on the particular backup:

You will be able to view backup location and which databases are inside the backup. There are also options to restore the backup or upload it into the cloud.

PITR Compatible Backup

ClusterControl performs the standard mysqldump backup with separate schema and data dumps. This makes it easy to restore partial backups. However, it breaks the consistency of the backup (schema and data are dumped in two separate sessions), thus it cannot be used to provision a slave or point-in-time recovery.

A mysqldump PITR-compatible backup contains one single dump file, with GTID info, binlog file and position. Thus, only the database node that produces binary log will have the "PITR compatible" option available, as highlighted in the screenshot below:

When PITR compatible option is toggled, the database and table fields are greyed out since ClusterControl will always perform the backup against all databases, events, triggers and routines of the target MySQL server.

The following lines will appear in the first ~50 lines of the completed dump file:

$ head -50 mysqldump_2017-11-07_072250_complete.sql
...
-- GTID state at the beginning of the backup
--
SET @@GLOBAL.GTID_PURGED='20dc5247-4a98-ee18-73af-5c79373388ee:1-1681';

--
-- Position to start replication or point-in-time recovery from
--
CHANGE MASTER TO MASTER_LOG_FILE='binlog.000001', MASTER_LOG_POS=2457790;
...

The information can be used to build slaves from backup, or perform point-in-time recovery together with binary logs, where you can start the recovery from the MASTER_LOG_FILE and MASTER_LOG_POS reported in the dump file using "mysqlbinlog" utility. Note that binary logs are not backed up by ClusterControl.

Build Slaves from Backup

Another feature is the ability to build a slave directly from a PITR-compatible backup, instead of doing it from a chosen master. This is a huge advantage as it offloads the master server. This option can be used with MySQL Replication or Galera Cluster. An existing backup can be used to rebuild an existing replication slave or add a new replication slave during the staging phase, as shown in the following screenshot:

Once the staging completes, the slave will connect to the chosen master and start catching up. Previously, ClusterControl performed a streaming backup directly from the chosen master using Percona Xtrabackup. This could impact performance of the master when scaling out a large dataset, despite the operation being non blocking on the master. With the new option, if the backup is stored on ClusterControl, only these hosts (ClusterControl + the slave) will be busy when staging the data on the slave.

Backup to Cloud

Backups can now be automatically uploaded in the cloud. This requires a ClusterControl module to be installed, called clustercontrol-cloud (Cloud integration module) and clustercontrol-clud (Cloud download/upload CLI) which are available in v1.5 and later. The upgrade instructions have been included with these packages and they come without any extra configuration. At the moment, the supported cloud platforms are Amazon Web Services and Google Cloud Platform. Cloud credentials are configured under ClusterControl -> Settings -> Integrations -> Cloud Providers.

When creating or scheduling a backup, you should see the following additional options when "Upload Backup to the cloud" is toggled:

The feature allows a one time upload or to schedule backups to be uploaded after completion (Amazon S3 or Google Cloud Storage). You can then download and restore the backups as required.

Custom Compression for mysqldump

This feature was in fact first introduced with ClusterControl v1.4.2 after its release. We added a backup compression level based on gzip. Previously, ClusterControl used the default backup compression (level 6) if the backup destination was on the controller node. The lowest compression (level 1 - fastest, less compression) was used if the backup destination was on the database host itself, to ensure minimal impact to the database during the compressing operation.

In this version, we have polished the compression aspect and you can now customize the compression level, regardless of the backup destination. When upgrading your ClusterControl instance, all the scheduled backups will be automatically converted to use level 6, unless you explicitly edit them in v1.5.

Backup compression is vital when your dataset is large, combined with a long backup retention policy, while storage space is limited. Mysqldump, which is text-based, can benefit from compression with savings of up to 60% of disk space of the original file size. On some occasions, the highest compression ratio is the best option to go, although it comes at the price of longer decompression when restoring.

Bonus Feature: Automatic Backup Verification

As old sysadmins say - A backup is not a backup if it's not restorable. Backup verification is something that is usually neglected by many. Some sysadmins have developed in-house routines for this, usually more manual than automated. Automating it is hard, mainly due to the complexity of the operation as a whole - starting from host provisioning, MySQL installation and preparation, backup files transfer, decompression, restore operation, verification procedures and finally cleaning up the system after the process. All these hassles make people neglect such an important aspect of a reliable backup. In general a backup restore test should be done at least once a month, or in case of significant changes in data size or database structure. Find a schedule that works for you and formalize it with a scheduled event.

ClusterControl can automate the backup verification by performing the restoration on a fresh host, without compromising any of the verification procedures mentioned above. This can be done after some delay, or right after the backup has completed. It will report the backup status based on the exit code of the restore operation, perform automatic shutdown if the backup is verified, or simply let the restored host run so you perform additional manual verifications on the data.

When creating or scheduling a backup, you will have additional options if "Verify Backup" is toggled:

If "Install Database Software" is enabled, ClusterControl will remove any existing MySQL installation on the target host and reinstall the database software with the same version as the existing MySQL server. Otherwise, if you have a specific setup for the restored host, you can skip this option. The rest of the options are self-explanatory.

Bonus Feature: Don’t Forget PostgreSQL

In addition to all this great functionality for MySQL and MariaDB ClusterControl 1.5 also now provides PostgreSQL with an additional backup method (pg_basebackup) that can be used for online binary backups. Backups taken with pg_basebackup can be used later for point-in-time recovery and as the starting point for a log shipping or streaming replication standby servers.

That’s it for now. Do give ClusterControl v1.5 a try, play around with the new features and let us know what you think.

Load balancers are an essential component in database high availability; especially when making topology changes transparent to applications and implementing read-write split functionality. ClusterControl provides an array of features to securely deploy, monitor and configure the industry's leading open source load balancing technologies.

In the past year we have added support for ProxySQL and added multiple enhancements for HAProxy and MariaDB’s Maxscale. We continue this tradition with the latest release of ClusterControl 1.5.

Based on feedback we received from our users, we have improved how ProxySQL is managed. We also added support for HAProxy and Keepalived to run on top of PostgreSQL clusters.

In this blog post, we’ll have a look at these improvements...

ProxySQL - User Management Enhancements

Previously, the UI would only allow you to create a new user or add an existing one, one at a time. One feedback we got from our users was that it is quite hard to manage a large number of users. We listened and in ClusterControl 1.5, it is now possible to import large batches of users. Let’s take a look at how you can do that. First of all, you need to have your ProxySQL deployed. Then, go to the ProxySQL node, and in the Users tab, you should see an “Import Users” button.

Once you click on it, a new dialog box will open:

Here you can see all of the users that ClusterControl detected on your cluster. You can scroll through them and pick the ones you want to import. You can also select or deselect all of the users from a current view.

Once you start to type in the Search box, ClusterControl will filter out non-matching results, narrowing the list only to users relevant to your search.

You can use the “Select All” button to select all users which match your search. Of course, after you selected users you want to import, you can clear the search box and start another search:

Please note “(7 selected)” - it tells you how many users, in total (not just from this search), you have selected to import. You can also click on it to see only the users you selected to import.

Once you are happy with your choice, you can click “Next” to go to the next screen.

Here you need to decide what should be the default hostgroup for each user. You can do that on per-user basis or globally, for the whole set or a subset of users resulting from a search.

Once you click on the “Import Users” button, users will be imported and they will show up in the Users tab.

ProxySQL - Scheduler Management

ProxySQL’s scheduler is a cron-like module which allows ProxySQL to start external scripts on a regular interval. The schedule can be quite granular - up to one execution every millisecond. Typically, the scheduler is used to execute Galera checker scripts (like proxysql_galera_checker.sh), but it can also be used to execute any other script that you like. In the past, ClusterControl used the scheduler to deploy the Galera checker script but this was not exposed in the UI. Starting ClusterControl 1.5, you now have full control.

As you can see, one script has been scheduled to run every 2 seconds (2000 milliseconds) - this is the default configuration for Galera cluster.

The above screenshot shows us options for editing existing entries. Please note that ProxySQL supports up to 5 arguments to the scripts it’ll execute through the scheduler.

If you want a new script to be added to the scheduler, you can click on the “Add New Script” button and you will be presented with a screen like the above. You can also preview how the full script will look like when executed. After you have filled all “Argument” fields and defined the interval, you can click on “Add New Script” button.

As a result, a script will be added to the scheduler and it’ll be visible on the list of scheduled scripts.

PostgreSQL - Building the High Availability Stack

Setting up replication with auto failover is good, but applications need a simple way to track the writeable master. So we added support for HAProxy and Keepalived on top of the PostgreSQL clusters. This allows our PostgreSQL users to deploy a full high availability stack using ClusterControl.

From the Load Balancer sub tab, you can now deploy HAProxy - if you are familiar with how ClusterControl deploys MySQL replication, it is a very similiar setup. We install HAProxy on a given host, two backends, reads on port 3308 and writes on port 3307. It uses tcp-check, expecting a particular string to return. To produce that string, the following steps are executed on all of the database nodes. First of all, xinet.d is configured to run a service on port 9201 (to avoid confusion with MySQL setup, which uses port 9200).

# default: on
# description: postgreschk
service postgreschk
{
        flags           = REUSE
        socket_type     = stream
        port            = 9201
        wait            = no
        user            = root
        server          = /usr/local/sbin/postgreschk
        log_on_failure  += USERID
        disable         = no
        #only_from       = 0.0.0.0/0
        only_from       = 0.0.0.0/0
        per_source      = UNLIMITED

The service executes /usr/local/sbin/postgreschk script, which validates the state of PostgreSQL and tells if a given host is available and what type of host it is (master or slave). If everything is ok, it returns the string expected by HAProxy.

Just like with MySQL, HAProxy nodes in PostgreSQL clusters are seen in the UI and the status page can be accessed:

Here you can see both backends and verify that only the master is up for the r/w backend and all nodes can be accessed through the read-only backend. You can also get some statistics about traffic and connections.

HAProxy helps to improve high availability, but it can become a single point of failure. We need to go the extra mile and configure redundancy with the help of Keepalived.

Under Manage -> Load balancer -> Keepalived, you pick the HAProxy hosts you’d like to use and Keepalived will be deployed on top of them with a Virtual IP attached to the interface of your choice.

From now on, all connectivity should go to the VIP, which will be attached to one of the HAProxy nodes. If that node goes down, Keepalived will take the VIP down on that node and bring it up on another HAProxy node.

That’s it for the load balancing features introduced in ClusterControl 1.5. Do try them and let us know how yo

Announced as part of the ClusterControl 1.5 release, we now provide full support for MariaDB version 10.2. This new version provides even greater integration with Galera Cluster, MariaDB’s HA solution of choice, and also features enhancements to SQL like window functions, common table expressions, and JSON functions.

MariaDB is the fastest growing open source database, reaching more than 60 million developers worldwide through its inclusion in every major Linux distribution, as well as a growing presence in the world’s leading cloud providers. Its widespread use across Linux distributions and cloud platforms, as well as its ease of use, have quickly made MariaDB the open source database standard for the modern enterprise.

MariaDB Server was listed in the recent OpenStack survey as the number one and two database technologies in use today.

What’s New in Version 10.2?

MariaDB Server 10.1 brought the default built-in integration of Galera Cluster to allows its users to achieve the ultimate in high availability. Severalnines was an early adopter of this clustering technology and was excited to see MariaDB embrace it for HA.

Here are some of the enhancements included in the new 10.2 version as announced by MariaDB…

• SQL enhancements like window functions, common table expressions and JSON functions allow new use cases for MariaDB Server
• Standard MariaDB Server replication has further optimizations
• Many area limitations have been removed, which allows easier use and there is no need for limitation handling on the application level
• MyRocks, a new storage engine developed by Facebook, has been introduced, which will further enrich the use cases for MariaDB Server (NOTE: This new Storage Engine is also now available for MariaDB deployments in ClusterControl, however ClusterControl does not yet support MyRocks specific monitoring.)

Window Functions

Window functions are popular in Business Intelligence (BI) where more complex report generation is needed based on a subset of the data, like country or sales team metrics. Another common use case is where time-series based data should be aggregated based on a time window instead of just a current record, like all rows inside a certain time span.

As analytics is becoming more and more important to end users, window functions deliver a new way of writing performance optimized analytical SQL queries, which are easy to read and maintain, and eliminates the need to write expensive subqueries and self-joins.

Common Table Expressions

Hierarchical and recursive queries are usually implemented using common table expressions (CTEs). They are similar to derived tables in a FROM clause, but by having an identification keyword WITH, the optimizer can produce more efficient query plans. Acting as an automatically created temporary and named result set, which is only valid for the time of the query, it can be used for recursive and hierarchical execution, and also allows for reuse of the temporary dataset. Having a dedicated method also helps to create more expressive and cleaner SQL code.

JSON Functions

JSON (JavaScript Object Notation), a text-based and platform independent data exchange format, is used not only to exchange data, but also as a format to store unstructured data. MariaDB Server 10.2 offers more than 24 JSON functions to allow querying, modification, validation and indexing of JSON formated data, which is stored in a text-based field of a database. As a result, the powerful relational model of MariaDB can be enriched by working with unstructured data, where required.

Through the use of virtual columns, the JSON function, JSON_VALUE and the newest indexing feature of MariaDB Server 10.2 on virtual columns, JSON values will be automatically extracted from the JSON string, stored in a virtual column and indexed providing the fastest access to the JSON string.

Using the JSON function JSON_VALID, the new CHECK CONSTRAINTS in MariaDB Server 10.2 guarantee that only JSON strings of the correct JSON format can be added into a field.

Binary Log Based Rollback

The enhanced mysqlbinlog utility delivered with MariaDB Server 10.2 includes a new point-in-time rollback function, which allows a database or table to revert to an earlier state, and delivers binary log based rollback of already committed data. The tool mysqlbinlog is not directly modifying any data, it is generating an “export file” including the reverted statements of the transactions, logged in a binary log file. The created file can be used with the command line client or other SQL tool to execute the included SQL statements. This way all committed transactions up to a given timestamp will be rolled back.

In the case of addressing logical mistakes like adding, changing or deleting data, so far the only possible way has been to use mysqlbinlog to review transactions and fix the problems manually. However, this often leads to data inconsistency because corrections typically only address the wrong statement, thereby ignoring other data dependencies.

Typically caused by DBA or user error, restoring a huge database can result in a significant outage of service. Rolling back the last transactions using point-in-time roll back takes only the time of the extract, a short review and the execution of the reverted transactions – saving valuable time, resources and service.

Why MariaDB?

With several MySQL options to choose from, why select MariaDB as the technology to power your application? Here are some of the benefits to selecting MariaDB...

• MariaDB is built on a modern architecture that is extensible at every layer: client, cluster, kernel and storage. This extensibility provides two major advantages. It allows for continual community innovation via plugins and it makes it easy for customers to configure MariaDB to support a wide variety of use cases from OLTP to OLAP.
• MariaDB develops features and enhancements that are part of its own roadmap, independent from Oracle / MySQL. This allows MariaDB to accept and attract broader community innovation, as well as to add internally developed new features that make it easier to migrate from proprietary systems to open source MariaDB.
• MariaDB is engineered to secure the database at every layer, making it a trusted general-purpose database used in industries such as government and banking that require the highest level security features.
• MariaDB offers support for a variety of storage engines, including NoSQL support, giving its users several choices to determine the one which will work best with their environment.
• MariaDB has deployed many performance enhancing improvements including query optimizations which, in several benchmark tests, let's MariaDB perform 3-5% better than a similarly configured MySQL environment.

ClusterControl for MariaDB

ClusterControl provides support for each of the top MariaDB technologies...

• MariaDB Server: MariaDB Server is a general purpose database engineered with an extensible architecture to support a broad set of use cases via pluggable storage engines – such as InnoDB, MyRocks and Spider.
  • Built-in asynchronous master/slave replication
  • Dynamic columns that allows different rows to store different data in the same column
  • Built-in encryption
  • Query optimization
  • Improved schema compatibility
• MariaDB Cluster: MariaDB Cluster is made for today’s cloud based environments. It is fully read-write scalable, comes with synchronous replication, allows multi-master topologies, and guarantees no lag or lost transactions.
  • Synchronous replication with no slave lag or lost transactions
  • Active-active multi-master topology
  • Read and write to any cluster node
  • Automatic membership control, with failed nodes dropped from the cluster
  • Automatic node joining
  • True row-level parallel replication
  • Direct client connections, native MariaDB look and feel
  • Both read and write scalability
• MariaDB MaxScale: MariaDB MaxScale is a database proxy that extends the high availability, scalability, and security of MariaDB Server while at the same time simplifying application development by decoupling it from underlying database infrastructure.
  • Includes Database Firewall and DoS protection
  • Read-Write Splitting
  • Data Masking
  • Schema-based Sharding
  • Query Caching

We’re happy to announce that our new whitepaper PCI Compliance for MySQL & MariaDB with ClusterControl is now available to download for free!

The Payment Card Industry Data Security Standard (PCI-DSS) is a set of technical and operational requirements to protect cardholder data and payment information. Any organisation that uses and stores such information is required to comply with the PCI standard.

Achieving full compliance with the standard is not easy. And even when it is achieved, one can easily fall out of compliance without the proper controls continuously in place. According to Verizon’s 2017 Payment Security Report, about half of businesses fail their interim PCI compliance assessment.

Access control, data protection and configuration management policies need to be implemented, and their continuous enforcement proven to auditors, in order to maintain compliance over time.

This guide discusses the essentials of a PCI compliance program from a database perspective. It will show to the reader how ClusterControl can be deployed to help address the requirements defined in the standard.

As a reminder, the PCI Data Security Standard Requirements (covered in this white paper) are:

• Building & Maintaining Secure Networks and Systems
• Vendor Supplied Default Settings
• Protecting Cardholder Data
• Encrypting Transmissions Across Public Networks
• Protection From Malware
• Maintaining Secure Applications
• Restricting Access to Cardholder Data
• Authentication & Access to System Components
• Monitoring & Testing Your Network
• Testing Security Systems
• Maintaining a Security Policy
• Onsite vs Cloud Storage

The whitepaper was written by Laurent Blume, Unix Systems Engineer and PCI-DSS expert and Severalnines CEO Vinay Joosery.

If your organization is or plans on performing payment processing on MySQL or MariaDB database infrastructures, this whitepaper will help you better understand what you need to do to stay compliant.

About the Authors

Laurent Blume, Unix Systems Engineer

Laurent's career in IT started in 2000, and even before then, he was evolving with modern computing and the advent of the Internet: from POS terminals for a jewelry stores chain to infrastructure servers in a government aerospace R&D organization, even touching supercomputers. One constant throughout was always the increasing need for security.

For the past 6 years, he has been in charge of first implementing, then keeping up with the PCI-DSS compliance of critical transnational payment authorization systems. Its implementation for databases has been an essential part of the task. For the last few years, it has expanded to the design and productization of a MariaDB cluster backend for mobile contactless payments.

Vinay Joosery, CEO & Co-Founder, Severalnines

Vinay is a passionate advocate and builder of concepts and business around distributed database systems.

Prior to co-founding Severalnines, Vinay held the post of Vice-President EMEA at Pentaho Corporation - the Open Source BI leader. He has also held senior management roles at MySQL / Sun Microsystems / Oracle, where he headed the Global MySQL Telecoms Unit, and built the business around MySQL's High Availability and Clustering product lines. Prior to that, Vinay served as Director of Sales & Marketing at Ericsson Alzato, an Ericsson-owned venture focused on large scale real-time databases.

The ClusterControl Community Edition is a free-to-use, all-in-one database management system that allows you to easily deploy and monitor the top open source database technologies like MySQL, MariaDB, Percona, MongoDB, PostgreSQL, Galera Cluster and more. It also allows you to import and monitor your existing database stack.

Free Database Deployment

The ClusterControl Community Edition ensures your team can easily and securely deploy production-ready open source database stacks that are built using battle-tested, proven methodologies. You don’t have to be a database expert to utilize the ClusterControl Community Edition - deploying the most popular open sources databases is easy with our point-and-click interface. Even if you are a master of deploying databases, ClusterControl’s point-and-click deployments will save you time and ensure your databases are deployed correctly, removing the chance for human error. There is also a CLI for those who prefer the command line, or need to integrate with automation scripts.

The ClusterControl Community Edition is not restricted to a single database technology and supports the major flavors and versions. With it you’re able to apply point-and-click deployments of MySQL standalone, MySQL replication, MySQL Cluster, Galera Cluster, MariaDB, MariaDB Cluster, Percona XtraDB and Percona Server for MongoDB, MongoDB itself and PostgreSQL!

Free Database Monitoring

The ClusterControl Community Edition makes monitoring easy by providing you the ability to look at all your database instances across multiple data centers or drill into individual nodes and queries to pinpoint issues. Offering a high-level, multi-dc view as well as a deep-dive view, ClusterControl lets you keep track of your databases so you can keep them running at peak performance.

In addition to monitoring the overall stack and node performance you can also monitor the specific queries to identify potential errors that could affect performance and uptime.

Why pay for a monitoring tool when the ClusterControl Community Edition gives you a great one for free!

Free Database Developer Studio

The Developer Studio provides you a set of monitoring and performance advisors to use and lets you create custom advisors to add security and stability to your database infrastructures. It lets you extend the functionality of ClusterControl, which helps you detect and solve unique problems in your environments.

We even encourage our users to share the advisors they have created on GitHub by adding a fork to our current advisor bundle. If we like them and think that they might be good for other users we’ll include them in future ClusterControl releases.

Why Should I Use the ClusterControl Community Edition?

These are just a few of the reasons why you should use ClusterControl as your system to deploy and monitor your open source database environments…

• You can deploy knowing you are using proven methodologies and industry best practices.
• If you are just getting started with open source database technology ClusterControl makes it easy for the beginner to deploy and monitor your stacks removing human error and saving you time.
• If you are not familiar with orchestration programs like Puppet and Chef? Don’t worry! The ClusterControl Community Edition uses a point-and-click GUI to make it easy to get your environment production-ready.
• The ClusterControl Community Edition gives you deployment and monitoring in one battle-tested all-in-one system. Why use one tool for scripting only to use a different tool for monitoring?
• If you are not sure what database technology is right for your application? The ClusterControl Community Edition supports nearly two dozen database versions that you can try.
• Have a load balancer running on an existing stack? With the ClusterControl Community Edition you can import and deploy your existing and already configured load balancer to run alongside your database instances.

If you are ready to give it a try click here to download and install the latest version of ClusterControl. Each install comes with the option to activate a 30-day enterprise trial as well.

The ClusterControl Enterprise Edition provides you will a full suite of management and scaling features in addition to the deployment and monitoring functions offered as part of the free Community Edition. You also have the ability to deploy, configure and manage the top open source load balancing and caching technologies to drive peak performance for your mission-critical applications.

Whether you have been benefiting from the free resources included in the Community Edition or have evaluated the product through the Enterprise Trial, we’ll walk you through how our licensing works and explain how to get you up-and-running with all the automation and scaling that ClusterControl Enterprise has to offer.

How to Upgrade from Community to Enterprise

While using the ClusterControl Community Edition you may have clicked on a feature and got a pop-up indicating that it was not included in the version you are using. When this happens you have two options. You can activate (or extend) your Enterprise Trial OR you can contact sales to purchase an enterprise license.

Enterprise Trial

The ClusterControl Enterprise trial provides you with free access to our full suite of features for 30 days. The purpose of this trial is to allow you to “kick the tires” using your environments and applications to make sure that ClusterControl meets your needs.

With the trial you have access to all our Community features plus: Custom Dashboards, Load Balancers, Configuration Management, Backup and Restore, Automatic Node and Cluster Recovery, Role Based Access Control, Key Management, LDAP, SSL Encryption Scaling, and more!

The trial also grants you Enterprise Level access to our support teams 24/7. We want to make sure that you have the best experience during your trial and also introduce you to our amazing support that you can count on when you become a customer of Severalnines.

At the end of your trial, you will have the option to meet with our sales team to continue with ClusterControl Enterprise on a paid license. Or you may also continue with our ClusterControl Community Edition, which you can use for free - forever.

Extending Your Trial

Sometimes thirty days isn’t enough time to evaluate a product as extensive as ClusterControl. In these situations we can sometimes grant an extension to allow you some more time to evaluate the product. This extension can be requested from the product itself and you will be contacted by an account manager to arrange for the extension.

Purchasing a Commercial License

ClusterControl offers three separate plans and different support options. Our account managers are available to assist, and recommend the best plan. We also offer volume discounts for larger orders. In short, we will work very hard to make sure our price meets your needs and budget. Once we’ve all signed on the dotted line, you will then be provided with Commercial License keys that you can put into your already deployed environment (or into a new one) which will then immediately grant you full access to the entire suite of ClusterControl features that you have contracted.

Benefits of Upgrading

While the free ClusterControl community version provides rich features that allow you to easily and securely deploy and monitor your open source databases, the Enterprise Edition provides much much more!

These are just some of the features awaiting you in the Enterprise Edition...

• Advanced Backup & Restoration: With ClusterControl you can schedule logical or physical backups with failover handling and easily restore backups to bootstrap nodes or systems.
• Automated Failover: ClusterControl includes advanced support for failure detection and handling; it also allows you to deploy different proxies to integrate them with your HA stack.
• Topology Changes: Making topology changes with ClusterControl is easy; it does all the background work to elect a new master, deploy fail-over slave servers, rebuild slaves in case of data corruption, and maintain load balancer configurations to reflect all the changes.
• Load Balancing: Load balancers are an essential component in database high availability; especially when making topology changes transparent to applications and implementing read-write split functionality and ClusterControl provides support for ProxySQL, HAProxy, and Maxscale.
• Advanced Security: ClusterControl removes human error and provides access to a suite of security features automatically protecting your databases from hacks and other threats. Operational Reports come in handy, whether you need to show you are meeting your SLAs or wish to keep track of the historical data of your cluster.
• Scaling: Easily add and remove nodes, resize instances, and clone your production clusters with ClusterControl.

In short, ClusterControl is an all-inclusive database management system that removes the need for your team to have to cobble together multiple tools, saving you time and money.

If you ever have any issues during this process you can always consult the documentation or contact us. If you need support you can contact us here.

If you haven’t upgraded to ClusterControl 1.5, you should! It’s full of great new features and enhancements. We have lots of documentation to guide you and help you get started. Documentation on older versions is still available in our Github repository.

Apart from the standard routine updates in the user and admin guides to reflect the latest features , we also made some enhancements in other areas.

Improved Readability

We received some feedback on the readability of the documentation page. We have now improved the look and feel by switching to a better CSS template. Syntax highlighting and source code tag are now following the same style we use in our blogs. We have also fixed a number of 404 errors and broken links across the documentation site.

ClusterControl Components

ClusterControl 1.5 introduced two new packages to handle cloud integration and management features. These packages are specifically for handling offsite backup to cloud storage, and will be a base to be extended when adding new functionality in upcoming releases. The ClusterControl modular setup now consists of a number of components:

• ClusterControl Controller (cmon)
• ClusterControl REST API
• ClusterControl UI
• ClusterControl SSH
• ClusterControl Notifications
• ClusterControl Cloud
• ClusterControl Cloud File Manager
• ClusterControl CLI

Each of the components is described in detail with examples. We also deprecated an older component called ClusterControl NodeJS which was available since v1.3, replaced by an improved version called ClusterControl Notifications.

Learn about them here.

Graceful Shutdown

We also got some requests to provide recommended ways to perform a clean shutdown of the database cluster, which can be useful during testing or maintenance. Depending on the clustering technology, the order of startup and shut down is vital to keep the whole cluster in sync and ensure the system will start up without any issues in the future.

It is recommended that the ClusterControl node be the last one to shut down, since it needs to oversee the state of the monitored hosts and save it into CMON database. When starting up the database cluster at the later stage, ClusterControl will perform a proper start-up procedure based on the last known state of the monitored hosts.

Check the recommended graceful shutdown steps here.

Troubleshooting Guide

We have listed out common issues, with troubleshooting steps to help you fix some common issues you might bump into. We are in the midst of migrating popular technical support cases from our Support forum, so you can find all the information in one place.

Check out the Troubleshooting Guide here.

We welcome any feedback, suggestions or comments in regards to our documentation to make sure you find everything you need. Happy clustering!

The ability to perform backup and restore of your data is one of the fundamental requirements of any DBMS. In a previous post, we discussed Postgres backups in general. In this blog post, we’ll look at how to do point-in-time recovery for PostgreSQL.

Point-in-time recovery (PITR) is a pretty useful feature. You need to keep in mind that the backup is a “snapshot” of your dataset at a given point of time. If you take backups on a daily basis, in the worst case scenario, you will have almost 24 hours of data that is not in more than you had in your last backup. So restoring from the last backup means losing a day’s worth of data. In general, it’s not really ok to lose so much data. Therefore you need to think about how you can recover all the changes that happened since you took the last backup. Let’s see how the whole process of backing up data and restoring it after an incident may look like on PostgreSQL 10.1 (which you can deploy using ClusterControl).

Setting up WAL archiving

PostgreSQL stores all modifications in form of Write-Ahead Logs (WAL). These files can be used for replication, but they can also be used for PITR. First of all, we need to setup WAL archiving. We’ll start with creating a directory in which the WAL files will be stored. This can be in any location: on disk, on NFS mount, anywhere - as long as PostgreSQL can write to it, it’ll do.

mkdir /wal_archive
chown -R postgres.postgres /wal_archive

We’ll create a simple directory and make sure it’s owned by ‘postgres’ user. Once this is done, we can start editing the PostgreSQL configuration.

archive_mode = on                       # enables archiving; off, on, or always
                                # (change requires restart)
archive_command = 'test ! -f /wal_archive/%f && cp %p /wal_archive/%f'                          # command to use to archive a logfile segment

wal_level = archive     # minimal, replica, or logical

The settings above have to be configured - you want to enable archiving (archive_mode = on), you want also to set wal_lever to archive. Archive_command requires more explanation. PostgreSQL doesn’t care where you will put the files, as long as you’ll tell it how to do it. This command will be used to copy the files to their destination. It can be a simple one-liner, as in the example above, it can be a call to a complex script or binary. What’s important is that this command should not overwrite existing files. It also should return zero exit status only when it managed to copy the file. If the return code will be different than 0, PostgreSQL will attempt to copy that file again. In our case, we verify if the file already exists or not. If not, we will copy it to the destination. ‘%p’ represents full path to the WAL file to be copied while ‘%f’ represents the filename only.

Once we are done with those changes, we need to restart PostgreSQL:

service postgresql restart

At this point, you should see WAL files appearing in your target directory. Please keep in mind that the process is triggered after a WAL file rotation - if you have very low traffic, it may take a while before WAL archiving will be started.

Base backup

The next step will be to take a base backup that we’ll use as a base for our point-in-time restore. You can take it using pg_basebackup:

pg_basebackup -Umyuser -h127.0.0.1 --progress -D /basebackup/

Or you can use ClusterControl for that:

When dealing with it manually, you may need to edit your pg_hba.conf to allow replication connections for the user you’ll be using for pg_basebackup. ClusterControl will take care of it for you.

Restore

Once you have a base backup done and WAL archiving configured, you are good to go. First of all, you need to figure out the point at which you should restore your data. Ideally, you’ll be able to give an exact time. Other options are: a named restore point (pg_create_restore_point()), transaction ID or WAL location LSN. You can try to use the pg_waldump utility to parse WAL files and print them in human-readable format, but it’s actually far from being human-readable so it can still be tricky to pinpoint the exact location.

rmgr: Transaction len (rec/tot):     34/    34, tx:     801534, lsn: 3/726587F0, prev 3/726587A8, desc: COMMIT 2017-11-30 12:39:05.086674 UTC
rmgr: Standby     len (rec/tot):     42/    42, tx:     801535, lsn: 3/72658818, prev 3/726587F0, desc: LOCK xid 801535 db 12938 rel 16450
rmgr: Storage     len (rec/tot):     42/    42, tx:     801535, lsn: 3/72658848, prev 3/72658818, desc: CREATE base/12938/16456
rmgr: Heap        len (rec/tot):    123/   123, tx:     801535, lsn: 3/72658878, prev 3/72658848, desc: UPDATE off 7 xmax 801535 ; new off 9 xmax 0, blkref #0: rel 1663/12938/1259 blk 0

When you know at which point you need to restore to, you can proceed with the recovery. First, you should stop the PostgreSQL server:

service postgresql stop

Then you need to remove all of the data directory, restore the base backup and then remove any existing WAL files. PostgreSQL will copy the data from our WAL archive directory.

rm -rf /var/lib/postgresql/10/main/*
cp -r /basebackup/* /var/lib/postgresql/10/main/
rm -rf /var/lib/postgresql/10/main/pg_wal/*

Now, it’s time to prepare the recovery.conf file, which will define how the recovery process will look like.

vim /var/lib/postgresql/10/main/recovery.conf
restore_command = 'cp /wal_archive/%f "%p"'
recovery_target_lsn = '3/72658818'

In the example above, we defined a restore command (simple cp from our /wal_archive directory into PostgreSQL pg_wal directory). We also should decide where to stop - we decided to use a particular LSN as a stop point, but you can also use:

recovery_target_name
recovery_target_time
recovery_target_xid

for a named restore point, timestamp and transaction ID.

Finally, make sure that all of the files in the PostgreSQL data directory have the correct owner:

chown -R postgres.postgres /var/lib/postgresql/

Once this is done, we can start PostgreSQL:

service postgresql start

In the log you should see entries like this:

2017-12-01 10:45:56.362 UTC [8576] LOG:  restored log file "000000010000000300000034" from archive
2017-12-01 10:45:56.401 UTC [8576] LOG:  restored log file "00000001000000030000001D" from archive
2017-12-01 10:45:56.419 UTC [8576] LOG:  redo starts at 3/1D9D5408
2017-12-01 10:45:56.464 UTC [8576] LOG:  restored log file "00000001000000030000001E" from archive
2017-12-01 10:45:56.526 UTC [8576] LOG:  restored log file "00000001000000030000001F" from archive
2017-12-01 10:45:56.583 UTC [8576] LOG:  restored log file "000000010000000300000020" from archive
2017-12-01 10:45:56.639 UTC [8576] LOG:  restored log file "000000010000000300000021" from archive
2017-12-01 10:45:56.695 UTC [8576] LOG:  restored log file "000000010000000300000022" from archive
2017-12-01 10:45:56.753 UTC [8576] LOG:  restored log file "000000010000000300000023" from archive
2017-12-01 10:45:56.812 UTC [8576] LOG:  restored log file "000000010000000300000024" from archive
2017-12-01 10:45:56.870 UTC [8576] LOG:  restored log file "000000010000000300000025" from archive

Once done, the server will still be in recovery mode. You can switch it to normal operations by running:

postgres=# create database mydb;
ERROR:  cannot execute CREATE DATABASE in a read-only transaction
postgres=# SELECT pg_wal_replay_resume();
 pg_wal_replay_resume
----------------------

(1 row)

postgres=# create database mydb;
CREATE DATABASE

You can also define in recovery.conf how PostgreSQL should behave after the recovery using:

recovery_target_action

There are three options:

• ‘pause’, the default, keeps the server in recovery mode. It is useful if you want to verify that everything is indeed in place before you allow normal traffic to hit the server and modify data.
• ‘promote’ will end the recovery mode as soon as the recovery itself completes.
• ‘shutdown’ will stop the PostgreSQL server after the recovery.

After recovery is complete, recovery.conf file will be renamed to recovery.done. Please keep in mind that this won’t happen with:

recovery_target_action = ‘shutdown’

In this case you need to remove or rename the file manually, otherwise the server will keep restarting over and over again.

As you can see, there is good support for performing point in time recovery in PostgreSQL. We hope you that you won’t have to go through this process on your production servers. It’s still good practice to test your organization’s skills in performing point-in-time recovery every now and then, so you are prepared in case of emergency.

With four major releases in 2017 ClusterControl is better than ever at supporting your MySQL, MariaDB, MongoDB& PostgreSQL environments.

When thinking about the features and functions released in 2017 three main themes emerge…

Delivering High Availability

2017 meant the introduction of ProxySQL, a lightweight yet complex protocol-aware proxy that sits between the MySQL clients and server. It also meant improved support for HAProxy and Keepalived and making sure that MySQL and MariaDB can fully utilize them.

Making You More Efficient

From the introduction of the new ClusterControl CLI to dozens of improvements to our UI to the new system to integration with alarms and chatops, ClusterControl now makes it even easier to manage your database environments.

Mixed Environment Support

ClusterControl has always been the system to manage multiple technologies from a single console and have them work together seamlessly. 2017 meant adding support for the latest versions of MariaDB, MongoDB, MySQL, PostgreSQL, Percona Server, and Galera Cluster.

ClusterControl 1.4.0 - January 2017

Announced in January 2017, ClusterControl version 1.4.0 brought several improvements for MySQL Replication and MongoDB. It was also the first version to introduce features for ProxySQL.

With the new version you are now able to deploy a multi-master replication setup in active - standby mode. One master will actively take writes, while the other one is ready to take over writes should the active master fail. From the UI, you can also easily add slaves under each master and reconfigure the topology by promoting new masters and failing over slaves.

Topology reconfigurations and master failovers are not always possible in case of replication problems, for instance errant transactions. In this version ClusterControl checks for issues before any failover or switchover happens. The admin can define whitelists and blacklists of which slaves to promote to master (and vice versa). This makes it easier for admins to customize failover automation in their replication setups.

For MongoDB we extended support, bringing sharded clusters in addition to replica sets. Coupled with this is the ability to retrieve more metrics for monitoring, adding new advisors and providing consistent backups for sharding. With this release, you could now convert a ReplicaSet cluster to a sharded cluster, add or remove shards from a sharded cluster as well as add Mongos/routers to a sharded cluster.

Lastly, we added our initial support for ProxySQL allowing for its deployment onto MySQL Replication setups.

ClusterControl 1.4.1 - April 2017

April was ProxySQL month at Severalnines. ClusterControl 1.4.1 focused almost exclusively on adding additional features and support for this exciting new load balancing technology.

In this version you could now easily configure and manage your ProxySQL deployments with a comprehensive UI. You could create servers, reorientate your setup, create users, set rules, manage query routing, and enable variable configurations. It was now possible to view query analytics for all queries going through the proxy, and e.g. cache any frequent queries in just a click.

ClusterControl 1.4.2 - June 2017

Coined “The DevOps Edition”, version 1.4.2 brought improved support and new features like automated failover for PostgreSQL& MongoDB and included even more features for ProxySQL.

One of the main highlights in this release is the ClusterControl CLI, which allows users who prefer to manage their databases through the command line. All actions, such as deploying a cluster, using the CLI will be visible in the UI and vice versa.

Also included in this release is the new integration system for alarm notification and chatops systems. This new integration with popular incident management and chat services lets you customise the alarms and get alerted in the ops tools you are already using - e.g., Pagerduty, VictorOps, Telegram, Opsgenie and Slack.

ClusterControl 1.5.0 - November 2017

ClusterControl 1.5 provided an array of exciting new backup functionalities to ensure that your data is secure and available whenever disaster strikes. The release also provides expanded PostgreSQL, MariaDB, MySQL NDB Cluster, and ProxySQL support.

This version introduced a new Backup Wizard with new support for AWS & Google Cloud backups, backup verification, Single Database backups and restores, and the ability to create and restore slaves from a backup rather than doing it from the master. Automatic restore testing was an awaited feature, as it is a time consuming task that is often neglected by database administrators.

PostgreSQL got a number of new features in this version including version 10 support, load balancing and virtual IP support with HAProxy and Keepalived, a new backup method, and support for synchronous replication failover.

The version also included support for MariaDB 10.2 and MySQL NDB Cluster 7.5.

If any of these features appeal to you make sure to upgrade or download the latest version of ClusterControl to take advantage of them.

We look forward to providing you even more features to help you deploy, monitor, manage and scale your open source databases further in 2018!

The picture below was taken in February of this year during our annual company meeting in Lisbon, Portugal; and while the picture was instigated by the author of this blog (I am in marketing after all), it did happen quite spontaneously as an ‘on-the-fly’ idea that my colleagues graciously indulged.

Different things could be read into this picture, but it is a picture that represents a team, which, although it’s spread across 14 different countries worldwide and only meets in person once a year, has achieved some pretty amazing things.

This is a snapshot of some of this year’s Severalnines highlights!

Four Major Releases of ClusterControl - the automation and management system for open source databases

This is at the core of our company and it’s what we’re the best at: helping MySQL, MariaDB, Percona Server, MongoDB and PostgreSQL users securely deploy, monitor, manage and scale their database infrastructures; be it on premise, in the cloud or both.

And while we keep innovating and adding features and functionalities to ClusterControl to help users future-proof their database deployments, we also keep enhancing the product in its four pillars of focus: deployment, monitoring, management and scaling - as that’s what it’s all about when looking to achieve severalnines nines high availability.

You can find an overview of all the product achievements from this year in this ClusterControl features highlight blog we published this week.

Thousands of ClusterControl Community Users and Hundreds of Enterprise Customers

Ever since we started out with Severalnines and ClusterControl, the relationships we maintain with our Community users and Enterprise customers have shaped us as a company, but first and foremost have shaped our product, ClusterControl. Indeed, user and customer feedback is what drives a good portion of our development efforts and every new release we publish includes several new functionalities that are originally user or customer feature requests. ClusterControl is a reflection of what System Administrators, DBAs, DevOps professionals or IT Leaders need in order to deploy and maintain their highly available database environments. But don’t just take our word for it …

“ClusterControl is phenomenal software…I’m usually not impressed with vendors or the software we buy, because usually it’s over promised and under delivered. ClusterControl is a nice handy system that makes me feel confident that we can run this in a production environment.” Jordan Marshall, Manager of Database Administration, Black Hills Corporation

“With quick installation, ease of use, great support, stable deployments and a scalable architecture, ClusterControl is just the solution we were looking for to provide a strong MySQL HA platform to our customers.” Xavi Morrus, CMO, MediaCloud

“Very easy to deploy a cluster, also it facilitates administration and monitoring.” Michel Berger IT Applications Manager European Broadcasting Union (EBU)

“ClusterControl is great for deploying and managing a high availability infrastructure. Also find the interface very easy to manage.” Paul Masterson, Infrastructure Architect, Dunnes

Read our case studies for more user and customer feedback.

Hundreds of Technical White Papers, Webinars, Videos, Tutorials and Blogs Published

If we’re known for being the open source database automation and management gurus, we’re also known for the content that we produce and publish on a weekly basis.

Because of the nature of our product and what we do, i.e. make the most popular open source databases in the world as easy as possible to manage from one single console, we’re led to know and write about a vast variety of technologies and related topics.

This is how we’ve progressed to being the de facto place to come to for anyone who’s interested in MySQL on Docker. But we’ve also made a bit of name for ourselves (maybe less surprisingly) for anything to do with database load balancing technologies.

But of course we blog about pretty much everything to do with MySQL, MariaDB, Percona Server, MongoDB and PostgreSQL; and produce webinars and others resources on these and related topics.

Great partners that we cooperate and (this is an open source world after all) also compete with

2017 saw us attend Percona Live and MariaDB user conferences as well as run a number of joint webinars such as with the Galera Cluster - Codership Team. These are always great opportunities for us not only to show-case what we do, but also to learn from others and hear from users and customers that are looking to automate and manage their open source databases.

We have a joint offering for instance with Percona for MongoDB users for that are looking to more securely and cost-efficiently manage their MongoDB installations, which we demonstrated during a nicely attended webinar.

And the conferences are always a good place for us to meet as a team and take a bit of time out with our colleagues ;-)

Finally, we’ve done all these things and more while having fun - whether virtually (thanks Skype and Slack) or, much better in all fairness, in person (where we also tend to eat a lot apparently …)!

So to sum things up: 2017 was a great year and here’s to an even better 2018! See you there!

Our clients are the lifeblood of Severalnines, not only because they fund our product development, but they are also our testers, our roadmap advisors and our partners in technology… pushing the limits of high availability databases through innovative design.

This year has been a great year for Severalnines and we have added many new companies to the ClusterControl family. While some have to remain quiet about their achievements, others have shared their stories, their challenges, and how ClusterControl helped them solve their problems and achieve performance and efficiency for the open source databases in their environments.

Here are just some of the stories we learned about in 2017 from our newest clients…

VentraIP is the largest privately owned web hosting and domain name registrar in Australia. They were using standalone MySQL instances running through a standard cPanel to manage their website, which also acted as the storefront for their domain registration, maintenance, transactions and payment processing. During peak times, the strain on the database caused it to lag and sometimes even crash, resulting in a frustrated client base and loss of revenue for the company.

They began investing in a MySQL Master-Master setup, but struggled to get it up-and-running at peak efficiency. After finding and installing ClusterControl, they were easily able to deploy a MariaDB Cluster and implement caching and load balancing on top. Since putting ClusterControl and this distributed database setup into production, they have eliminated all of their performance issues and no longer receive customer complaints about failed transactions or site outages. You can learn more about their story here.

Mediacloud is the division of Mediapro that manages all the data centers for one of Spain’s largest multimedia communications companies. Handling the needs of the fans of some of the largest football clubs in Europe can put a strain on any IT infrastructure. With more than 150 organizations trusting that Mediacloud would deliver seamless and highly-available content, they needed a database infrastructure they could count on.

The Mediacloud team evaluated many solutions before they decided on Galera Cluster and ClusterControl. One of the main reasons for this selection was the ability to distribute data across multiple data centers (of which Mediacloud has six), and that the distributed nature of the system lended itself well to a cloud environment. Their database infrastructure continue to deliver even as the user demands on the system continue to grow. You can learn more about their story here.

Root Level Technology is a managed hosting provider that specializes in developing custom high performance solutions with geographic redundancy with built-in DDoS Protection. The database is a key component in achieving high-availability in these custom solutions, and the team decided on ClusterControl to power the open source database aspects of their endeavors.

The combination of dependable technology, secure deployments and time savings of database automation allowed them to deliver an even higher quality offering for their clients. In addition, Root Level Tech did not have a DBA on staff, relying on ClusterControl as their ‘virtual DBA’ and their database monitoring solution. You can learn more about their story here.

Black Hills Corporation provides energy for clients in eight states in the US. While most of its massive infrastructure is ran on MSSQL Server or Oracle, there were many smaller open source database environments running what were deemed non-business critical applications. Regardless of their status, in a company like Black Hills everything needs to be managed effectively and efficiently!

The IT Team wanted to implement an enterprise-grade solution that brought these MySQL databases under one roof so they could monitor and manage them alongside their Microsoft SQL Server and Oracle databases. Their requirements were simple - migrate to MySQL Galera Cluster for improved performance, be able to monitor these databases, high availability across multiple data centers with failover, and reliable backups for disaster recovery. ClusterControl checked all these boxes and, now that they have the solution in place, they plan on utilizing MySQL for even more business applications. You can learn more about their story here.

While StreamAMG has been a user of ClusterControl and a client of ours for a while now, we were able to sit down with them this year to discuss in more detail about how they are using ClusterControl to manage their database clusters on Amazon Web Services.

StreamAMG is Europe’s largest provider of online video solutions and is used by many creators as the backbone of their streaming video content. The company’s explosive growth meant having to implement a new system to manage the databases to keep up with the growth of the rest of the AWS-based infrastructure. They chose ClusterControl to ‘future-proof’ their databases. Based in the cloud, the addition of ClusterControl let them increase their overall streaming speed by 76%. You can watch the video below of our interview this year with StreamAMG Platform Manager Andrew de Bono and also learn more about their story here.

2017 has been a great year for Severalnines and we encourage you to share your ClusterControl story by writing a review of our product on Capterra.

With 2017 quickly coming to a close and 2018 looming on the horizon we wanted to take a minute and reflect on what’s been happening in our industry in the past year and what we are excited about for the future.

Johan Andersson, Severalnines CTO and Co-Founder took a few minutes from working on the newest releases of ClusterControl to talk with us about his thoughts on 2017.

2018 Database Trends and Predictions

As technology moves fast the open source world moves even faster. Here are some predictions from around the web for 2018…

FORBES

• “In 2017, DevOps suffered from under-budgeting and a perception from management that things that were inexpensive as tools were mostly open source. However, non-standardized adoption and expensive DevOps resources skewed budget. With the realization that open source doesn’t equal free, especially as enterprise-grade support is required, there will be increased awareness of the budget needed for skilled DevOps resources. This barrier should get lowered -- organizations will need a budget for experimentation and failure.”

GITHUB

• “Data will rule all. Over the last several years, Cloud 1.0 has been about computing in big clouds, while Cloud 2.0 is all about data. This includes data movement and the tools and services that support it, like analytics and machine learning systems. Today all companies are data companies, whether they know it or not. In 2018, so long as teams know how to use it, data will become their greatest asset.”
• “A decade ago, Linux was a big deal. Now it’s standard. Back in the day, companies like Amazon, Google, and Microsoft were forced to build their own, proprietary tools because no other software existed to meet their needs. Many of these frameworks have since been open sourced—and other open source technologies, like Kubernetes, are becoming integral to developers' workflows. This shift is changing what companies are investing in, making open source software traditional software's biggest competitor.”

OPENSOURCE.COM

• “Containers gain even more acceptance. Container technology is the approach of packaging pieces of code in a standardized way so they can be "plugged and run" quickly in any environment. Container technology allows enterprises to cut costs and implementation times. While the potential of containers to revolutionize IT infrastructure has been evident for a while, actual container use has remained complex. Container technology is still evolving, and the complexities associated with the technology decrease with every advancement. The latest developments make containers quite intuitive and as easy as using a smartphone, not to mention tuned for today's needs, where speed and agility can make or break a business.”

DBTA.COM

• “Rapid Kubernetes adoption forms the foundation for multi-cloud deployments:We predict runaway success of Kubernetes, but it is running away with the prize of adoption so fast that this may quickly be more of an observation than a prediction in 2018. So far, however, almost everybody is thinking of Kubernetes as a way of organizing and orchestrating computation in a cloud. Over the next year, we expect Kubernetes to more and more be the way that leading-edge companies organize and orchestrate computation across multiple clouds, both public and private. On premises computation is moving to containers and orchestration style at light speed, but when you can interchangeably schedule work anywhere that it makes sense to do so, you will see the real revolution.”
• “Fear of cloud lock-in will result in cloud sprawl: As CIOs try to diversify investment in their compute providers, inclusive of their own on-premise capabilities, the diversification will result in data, services and algorithms spreading across multiple clouds. Finding information or code within a single cloud is tough enough. The data silos built from multiple clouds will be deep and far apart, pushing the cost of management onto humans that need to understand the infrastructure.”

Severalnines 2018 Predictions

Several members of our team took a moment to share their thoughts on 2018 and the future of the open source database world…

Vinay Joosery, CEO & Co-Founder

• Databases in Containers: I think a lot of people have gone from discussing the pros and cons of Docker for databases, and whether it is a good idea at all when it comes to running databases on Docker, to trying it not only in test/dev, but in actual live production!

Alex Yu, VP of Products

• Cloud and containerized services as the new norm - More companies will start new projects in the cloud with multiple providers and applications will be written with cloud-native architectures in mind.
• Traditional monolithic applications will continue to give away to more loosely coupled services which are easier to build, deploy, update, scale or move to other cloud and container service providers. Cloud-native services are resilient by nature and will facilitate faster development cycles and feedback loops.
• Container technologies such as Kubernetes and Docker are already de-facto standard for typical stateless applications/services using "application containers" however databases though are intrinsically stateful and we will hopefully see more use of "system containers" such as LXD coming into the mainstream and gain adoption.
• This "new world" has an impact on database management and monitoring applications where containerized services come and go frequently. The lifetime of a host is many times longer than a container where the uptime can be measured in only minutes or hours. Host centric management and monitoring will give away to a cloud-native service oriented model where the transient nature is the norm.

Jean-Jérôme Schmidt, VP of Marketing

• The European Union (EU) is the world’s 2nd largest economy after China and before the US. It’s about to enact a new piece of legislation with far-reaching consequences for anyone doing business involving its residents … and yet it seems to be going almost unnoticed. The European General Data Protection Regulation (GDPR) is due to come into effect in May 2018 and will impact anyone and any business or organisation that deals with and stores EU residents’ personal data in some form or shape. And it will make the organisations that are handling that data responsible for any breaches or misuse. It will therefore be of the highest importance how data is collected, processed and secured. In other words, databases and their infrastructures will be in the spotlight more than ever before and how these database are automated and managed will be crucial for anyone doing business with or within the EU. The GDPR is probably not getting the attention it should because the perception that’s being maintained globally is that the US (and maybe China) are the only large-scale economies worth being concerned with, but the reality is that the EU is the one to be focussed on, particularly next year. So if you’re not sure whether you have your databases and their EU residents data 99.999% under control … contact us ;-)

Ashraf Sharif, Senior Support Engineer

• We are expecting higher adoption of MySQL 8.0 once it becomes GA. It introduces many notable enhancements like transactional data dictionary, atomic DDL, invisible indexes, common table expression (CTE), windows function and MySQL roles to name some of them. More details at MySQL Documentation page. We are also forecasting a growth in MyRocks storage engine adoption which is already included in Percona Server 5.7 and MariaDB 10.2.
• MySQL on Docker will be getting much more attention in the coming years, after a number of success stories like Uber and BlablaCar. We've seen many people trying to adapt this technology as a reliable backend data service with the help of in-house automation scripts and Docker orchestration tools. Besides, Docker has announced support for Kubernetes, allowing developers and operators to build apps with Docker and seamlessly test and deploy them using both Docker Swarm and Kubernetes.

Krzysztof Książek, Senior Support Engineer

• The main new trend that I see today is a move towards column store, analytics databases. MariaDB has it as part of their offering and ClickHouse seems to get traction as a go-to analytics Database engine that works alongside MySQL. ProxySQL's support for ClickHouse also makes it easier for the application to connect to either MySQL or ClickHouse, whatever is needed at that moment. If your dataset is small, you can do analytics in MySQL but there are other tools which do it better - faster and use less disk to store the data.