We’re happy to announce that our new whitepaper Choosing a Database Proxy for MySQL and MariaDB is now available to download for free!

Load balancing and high availability go hand-in-hand. Without them, you are left with a single point of entry for your MySQL or MariaDB databases and any spike in traffic could cause your setup to crash.

High-traffic database applications draw an enormous amount of queries daily and DBAs and SysAdmins require reliable technology solutions that can automatically scale to handle those connections while remaining available for still more.

In this whitepaper, we discuss what proxies are, what their use is and how to build a highly available and highly controllable MySQL and MariaDB database environment using modern proxies.

Topics included in this whitepaper are …

• What is a database proxy?
• Overview of different proxy types
• Features of advanced, SQL-aware proxies
  • Query Routing
  • Query Rewriting
  • Query Caching
  • Killing or slowing a Query
• Designing a highly available proxy architecture
  • Dedicated proxy instances
  • Virtual IPs
  • Elastic load balancing
  • Application-side proxy discovery
  • Collocating proxies on application hosts
  • Silo approach to proxies
• Managing large proxy deployments
• Setting up highly available proxy layers with ClusterControl
  • ProxySQL
  • HAProxy
• What proxy should I pick?

Download the whitepaper today!

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that lets you automate many of the database tasks you have to perform regularly like deploying new databases, adding and scaling new nodes, running backups and upgrades, and more.

To learn more about ClusterControl click here.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 32,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States. To see who is using Severalnines today visit, https://www.severalnines.com/company.

We discussed in our previous blogs about the MySQL-related dashboards. We highlighted the things that a DBA can benefit from by studying the graphs, especially when performing their daily routines from diagnostics, metric reporting, and capacity planning. In this blog, we will discuss the InnoDB Metrics and the MySQL Performance Schema, which is very important especially on monitoring InnoDB transactions, disk/cpu/memory I/O, optimizing your queries, or performance tuning of the server.

This blog touches upon the deep topic of performance, considering that InnoDB would require extensive coverage if we tackle its internals. The Performance Schema is also extensive as it covers kernel and core parts of MySQL and storage engines.

Let’s begin walking through the graphs.

MySQL InnoDB Metrics

This dashboard is great for any MySQL DBA or ops person, as it offers a very good view into the InnoDB storage engine. There are certain graphs here that a user has to consider to enable, because not in all situations that the variables are set correctly in the MySQL configuration.

• Innodb Checkpoint Age

  According to the manual, checkpointing is defined as follows: “As changes are made to data pages that are cached in the buffer pool, those changes are written to the data files sometime later, a process known as flushing. The checkpoint is a record of the latest changes (represented by an LSN value) that have been successfully written to the data files”. This graph is useful when you would like to determine how your server is performing checkpointing data to your disk. This can be a good reference if your transaction log (redo log or ib_logfile0) is too large. This graph is also a good indicator if you need to adjust variables such as innodb_log_file_size,, innodb_log_buffer_size, innodb_max_dirty_pages_pct, or innodb_adaptive_flushing_method. The closer checkpoint age is to the max checkpoint age, the more filled are the logs and InnoDB will be doing more I/O in order to maintain some free space in the logs. Checkpointing mechanism differs in subtle details between Percona XtraDB-based flavours, MariaDB and Oracle’s version, you can also find differences in it’s implementation between MySQL versions.

• InnoDB Transactions

  Whenever there’s a large transaction on-going in your MySQL server, this graph is a good reference. It will count the transactions that were created at a specific time, and the history length (or is actually the history list length found in SHOW ENGINE INNODB STATUS) is the number of pages in the undo log. The trends you’ll see here is a good resource to check if it could mean, for example, that purge is delayed due to a very high insert rate of reloading the data or due to a long-running transaction, or if purge simply can't keep up due to a high disk I/O in the volume where your $DATADIR resides.

• Innodb Row Operations

  For certain DBA tasks, you might want to determine the number of deletes, inserts, reads, and rows updated. Then this graph is what you can use to check these.

• Innodb Row Lock Time

  This graph is a good resource to look upon when you are noticing that your application is encountering lots of occurrences for “Lock wait timeout exceeded; try restarting transaction”. This can also help you determine if you might have an indication for using bad queries on handling locks. This is also a good reference to look upon when optimizing your queries that involves locking of rows. If the time to wait is too high, you need to check the slow query log or run a pt-query-digest and see what are those suspecting queries causing that bloat in the graph.

• InnoDB I/O

  Whenever you want to determine the amount of InnoDB data reads, disk flushes, writes, and log writes, this graph has what you need to look at. You can use this graph to determine if your InnoDB variables are well tuned to handle your specific requirements. For example, if you have Battery Backup Module cache but you are not gaining much of its optimum performance, you can rely on this graph to determine if your fsyncs() are higher than expected. Then changing the variable innodb_flush_method and using O_DSYNC can resolve the issue.

• InnoDB Log File Usage Hourly

  This graph shows only the number of bytes written to the InnoDB redo log files and the growth of your InnoDB log files based on the 24-hour time range of the current date.

• InnoDB Logging Performance

  This graph is closely related to InnoDB Log File Usage Hourly graph. You have to use this graph whenever you need to determine how large your innodb_log_file_size needs to be. You can determine the number of bytes written to the InnoDB redo log files and how efficiently your MySQL flushes data from memory to disk. Whenever you are experiencing a low-time in need to use your redo log space, then it would indicate that you have to increase your innodb_log_file size. In that case, this graph would tell you that you need to do so. However, to dig more into how much you need for your innodb_log_file, it might make more sense to check the LSN (Log Sequence Number) in SHOW ENGINE INNODB STATUS. Percona has a good blog related to this which is a good source to look at.

• InnoDB Deadlocks

  In certain situations that your application client is often experiencing deadlocks or you have to look at how much your MySQL is experiencing deadlocks, this graph serves the purpose. Deadlocks indicate that you have poor SQL design which leads to your transactions creating a race condition causing deadlocks.

• Index Condition Pushdown

  A little word of caution when looking at this graph. First, you have to determine that you have your MySQL global variable innodb_monitor_enable set to the correct value that is module_icp. Otherwise, you’ll experience a “No Data Points” as shown below:

  The graph’s purpose, if has data points defined as what I have in the sample outputs, will provide a DBA with an overlook of how well your queries are benefiting with Index Condition Pushdown or ICP for short. ICP is great feature in MySQL that offers optimization to your queries. Instead of MySQL reading the full rows filtered in your WHERE queries upon retrieval, it will add more checks after your secondary indexes. This adds more granularity and saves time, otherwise the engine has to read the full-table rows instead when it is based only on the filtered index and no ICP is used. This avoids reading the full rows corresponding to your index tuples that matches your secondary indexes.

  Let me elaborate a bit about this graph, let say I have a table named:

  mysql> show create table a\G
  *************************** 1. row ***************************
         Table: a
  Create Table: CREATE TABLE `a` (
    `id` int(11) NOT NULL,
    `age` int(11) NOT NULL,
    KEY `id` (`id`)
  ) ENGINE=InnoDB DEFAULT CHARSET=latin1
  1 row in set (0.00 sec)

  And has some small data:

  mysql> select * from a;
  +----+-----+
  | id | age |
  +----+-----+
  |  1 |   1 |
  |  2 |   1 |
  |  3 |   1 |
  |  3 |  41 |
  |  4 |  41 |
  |  5 |   4 |
  |  4 |   4 |
  |  4 |   4 |
  +----+-----+
  8 rows in set (0.00 sec)

  When ICP is enabled, results is more efficient and feasible:

  mysql> explain extended select * from a where id>2 and id<4 and age=41;
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
  | id | select_type | table | partitions | type  | possible_keys | key  | key_len | ref  | rows | filtered | Extra                              |
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
  |  1 | SIMPLE      | a     | NULL       | range | id            | id   | 4       | NULL |    2 |    12.50 | Using index condition; Using where |
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+------------------------------------+
  1 row in set, 2 warnings (0.00 sec)

  Than without ICP,

  mysql> set optimizer_switch='index_condition_pushdown=off';
  Query OK, 0 rows affected (0.00 sec)
  
  mysql> explain extended select * from a where id>2 and id<4 and age=41;
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+-------------+
  | id | select_type | table | partitions | type  | possible_keys | key  | key_len | ref  | rows | filtered | Extra       |
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+-------------+
  |  1 | SIMPLE      | a     | NULL       | range | id            | id   | 4       | NULL |    2 |    12.50 | Using where |
  +----+-------------+-------+------------+-------+---------------+------+---------+------+------+----------+-------------+
  1 row in set, 2 warnings (0.00 sec)

  This is a simple example of ICP, and how this graph can benefit a DBA.

• InnoDB Buffer Pool Content

  When working with MySQL and using InnoDB engine, this graph is one of the most common values (innodb_buffer_pool*) that you have to tune up to optimize MySQL performance. Specifically speaking on its buffer pool content, it displays the trends for dirty pages against the total buffer pool content. The total buffer pool content includes the clean pages aside of dirty pages. Determining how efficient your MySQL is handling the buffer pool, this graph serves its purpose.

• InnoDB Buffer Pool Pages

  This graph is helpful when you want to check how efficient MySQL is using your InnoDB buffer pool. You can use this graph, for instance, if your daily traffic doesn’t fill up the assigned innodb_buffer_pool_size, then this could indicate that certain parts of an application aren’t useful or do not serve any purpose or if you set the innodb_buffer_pool_size very high which might be good to lower the value and reclaim back space to your memory.

• InnoDB Buffer Pool I/O

  When you have to check the number of pages created and written on InnoDB tables or page reads to InnoDB buffer pool by operations on InnoDB tables.

• InnoDB Buffer Pool Requests

  When you want to determine how efficiently are your queries are accessing the InnoDB buffer pool, this graph serves the purpose. This graph will show the trends based on the data points on how your MySQL server performs when InnoDB engine has to frequently access the disk (indication of buffer pool has not warmed up yet), how frequent the buffer pool requests were handling read requests and write requests.

• InnoDB Read-Ahead

  When you have the variable innodb_random_read_ahead set to ON, then add this graph as a valuable trend to look at as part of your DBA routine. It shows the trends on how your MySQL InnoDB storage engine manages the buffer pool by the read-ahead background thread, how it manages those subsequently evicted without having been accessed by queries, and how does InnoDB initiate the random read-ahead when a query scans a large portion of a table but in random order.

• InnoDB Change Buffer

  When you have Percona Server 5.7 running, this graph is useful when monitoring how well InnoDB has allocated change buffering. This changes includes those inserts, updates, and deletes which are specified by innodb_change_buffering variable. Change buffering helps speed up queries, avoiding substantial random access I/O that would be required to read-in secondary index pages from disk.

• InnoDB Change Buffer Activity

  This is related to the InnoDB Change Buffer graph, but dissects the information into more viable data points. These provide more information to monitor how InnoDB handles change buffering. This is useful in a particular DBA task to determine if your innodb_change_buffer_max_size is set to a too high value, since the change buffering shares the same memory of the InnoDB buffer pool reducing the memory available to cache data pages. You might have to consider to disable change buffering if the working set almost fits in the buffer pool, or if your tables have relatively few secondary indexes. Remember that change buffering does not impose extra overhead, because it only applies to pages that are not in the buffer pool. This graph is also useful if you have to determine how merges are useful if you do have to benchmark your application based on certain requests for particular scenarios. Let say you have a bulk inserts, you have to set innodb_change_buffering=insert and determine if having the values set in your buffer pool and innodb_change_buffer_max_size do not impact disk I/O, specially during recovery or slow shutdown (necessary if you want to do a failover with low downtime requirement). Also, this graph can serve your purpose to evaluate certain scenarios, since merging of change buffer may take several hours when there are numerous secondary indexes to update and many affected rows. During this time, disk I/O is increased, which can cause a significant slowdown for disk-bound queries.

MySQL Performance Schema

The MySQL Performance Schema is a complicated topic. It’s a long and hard one, but I’m going to discuss only information that is specific to the graphs we have in SCUMM. There are certain variables as well that you must consider, and ensure they are set properly. Ensure that you have your variable innodb_monitor_enable = all and userstat=1 to see data points in your graphs. As a note, when I am using the word “event” here, it does not mean that this is related to MySQL Event Scheduler. I’m talking about specific events such as MySQL parses a query, MySQL is reading or writing to relay/binary log file, etc.

Let’s proceed with the graphs then.

• Performance Schema File IO (Events)

  This graph fetches data points related to any events that occurred in MySQL which might have been instrumented to create multiple instances of the instrumented object (e.g. binary log reads or InnoDB data file reads). Each row summarizes events for a given event name. For example, if there is an instrument for a mutex that is created for each connection, then there could be many instances of this instrumented event as there are multiple connections. The summary row for the instrument summarizes over all these instances. You can check these events in MySQL manual for Performance Schema Summary Tables for more info.

• Performance Schema File IO (Load)

  This graph is same as “Performance Schema File IO (Events)” graph except that it’s instrumented based on the load.

• Performance Schema File IO (Bytes)

  This graph is same as “Performance Schema File IO (Events)” graph except that it’s instrumented based on the the size in bytes. For example, how much time did a specific event take when MySQL triggered wait/io/file/innodb/innodb_data_file event.

• Performance Schema Waits (Events)

  This graph has the data graph for all waits spent on a specific event. You can check Wait Event Summary Tables in the manual for more info.

• Performance Schema Waits (Load)

  Same as the “Performance Schema Waits (Events)” graph but this time it shows the trends for the load.

• Index Access Operations (Load)

  This graph is an aggregation of all the table index I/O wait events grouped by index(es) of a table, as generated by the wait/io/table/sql/handler instrument. You can check the MySQL manual about the Performance Schema table table_io_waits_summary_by_index_usage for more info.

• Table Access Operations (Load)

  “Same as Index Access Operations (Load)” graph, it’s an aggregation of all table I/O wait events group by table, as generated by the wait/io/table/sql/handler instrument. This is very useful to DBAs. For example, you would like to trace how fast it takes to access (fetch) or update (insert, delete, update) a specific table. You can check in the MySQL manual about the Performance Schema table table_io_waits_summary_by_table for more info.

• Performance Schema SQL & External Locks (Events)

  Related resources

   Effective Monitoring of MySQL with SCUMM Dashboards - Part 1

   Effective Monitoring of MySQL Replication with SCUMM Dashboards - Part 2

   How to Monitor MySQL or MariaDB Galera Cluster with Prometheus Using SCUMM

  This graph is an aggregation (counts of how many times it occured) of all table lock wait events, as generated by the wait/lock/table/sql/handler instrument which is group by table. The SQL lock here in the graph means of the internal locks. These internal locks are read normal, read with shared locks, read high priority, read no insert, write allow write, write concurrent insert, write delayed, write low priority, write normal. While the external locks are read external and write external. In any DBA task, this is very useful if you have to trace and investigate locks on a particular table regardless of its type. You can check the table table_lock_waits_summary_by_table for more info.

• Performance Schema SQL and External Locks (Seconds)

  Same as graph “Performance Schema SQL & External Locks (Events)”, but specified in seconds. If you want to look for your table locks based on seconds it held the locks, then this graph is your good resource.

Conclusion

The InnoDB Metrics and MySQL Performance Schema are some of the most in-depth and complicated parts in the MySQL domain, especially when there is no visualization to assist the interpretation. Thus, going to a manual trace and investigations may take some of your time and hard work. SCUMM dashboards offer a very efficient and feasible way to handle these and lower the extra load on any DBA routine task.

In this blog, you learnt how to use the dashboards for InnoDB and Performance Schema to improve database performance. These dashboards can make you more efficient at analyzing performance.

Severalnines was pleased to yet again sponsor Percona Live Europe which was held this year in Frankfurt, Germany. Thanks to the Percona Team for having us and the great organisation.

At the Conference

Severalnines team members flew in from around the world to show off the latest edition of ClusterControl in the exhibit hall and present five sessions (see below).

On our Twitter feed we live tweeted both of the keynote sessions to help keep those who weren’t able to attend up-to-speed on the latest happenings in the open source database world.

Our Sessions

Members of the Severalnines Team presented five sessions in total at the event about MySQL, MariaDB & MongoDB, each of which showcased ClusterControl and how it delivers on those topics.

Disaster Recovery Planning for MySQL & MariaDB

Presented by: Bart Oles - Severalnines AB

Session Details: Organizations need an appropriate disaster recovery plan to mitigate the impact of downtime. But how much should a business invest? Designing a highly available system comes at a cost, and not all businesses and indeed not all applications need five 9's availability. We will explain fundamental disaster recovery concepts and walk you through the relevant options from the MySQL & MariaDB ecosystem to meet different tiers of disaster recovery requirements, and demonstrate how to automate an appropriate disaster recovery plan.

MariaDB Performance Tuning Crash Course

Presented by: Krzysztof Ksiazek - Severalnines AB

Session Details: So, you are a developer or sysadmin and showed some abilities in dealing with databases issues. And now, you have been elected to the role of DBA. And as you start managing the databases, you wonder

• How do I tune them to make best use of the hardware?
• How do I optimize the Operating System?
• How do I best configure MySQL or MariaDB for a specific database workload?

If you're asking yourself the following questions when it comes to optimally running your MySQL or MariaDB databases, then this talk is for you!

We will discuss some of the settings that are most often tweaked and which can bring you significant improvement in the performance of your MySQL or MariaDB database. We will also cover some of the variables which are frequently modified even though they should not.

Performance tuning is not easy, especially if you're not an experienced DBA, but you can go a surprisingly long way with a few basic guidelines.

Performance Tuning Cheat Sheet for MongoDB

Presented by: Bart Oles - Severalnines AB

Session Details: Database performance affects organizational performance, and we tend to look for quick fixes when under stress. But how can we better understand our database workload and factors that may cause harm to it? What are the limitations in MongoDB that could potentially impact cluster performance?

In this talk, we will show you how to identify the factors that limit database performance. We will start with the free MongoDB Cloud monitoring tools. Then we will move on to log files and queries. To be able to achieve optimal use of hardware resources, we will take a look into kernel optimization and other crucial OS settings. Finally, we will look into how to examine performance of MongoDB replication.

Advanced MySql Data-at-Rest Encryption in Percona Server

Presented by: Iwo Panowicz - Percona & Bart Oles - Severalnines AB

Session Details: The purpose of the talk is to present data-at-rest encryption implementation in Percona Server for MySQL.
Differences between Oracle's MySQL and MariaDB implementation.

• How it is implemented?
• What is encrypted:
• Tablespaces?
• General tablespace?
• Double write buffer/parallel double write buffer?
• Temporary tablespaces? (KEY BLOCKS)
• Binlogs?
• Slow/general/error logs?
• MyISAM? MyRocks? X?
• Performance overhead.
• Backups?
• Transportable tablespaces. Transfer key.
• Plugins
• Keyrings in general
• Key rotation?
• General-Purpose Keyring Key-Management Functions
• Keyring_file
• Is useful? How to make it profitable?
• Keyring Vault
• How does it work?
• How to make a transition from keyring_file

Polyglot Persistence Utilizing Open Source Databases as a Swiss Pocket Knife

Presented by: Art Van Scheppingen - vidaXL & Bart Oles - Severalnines AB

Session Details: Over the past few years, VidaXL has become a European market leader in the online retail of slow moving consumer goods. When a company achieved over 50% year over year growth for the past 9 years, there is hardly enough time to overhaul existing systems. This means existing systems will be stretched to the maximum of their capabilities, and often additional performance will be gained by utilizing a large variety of datastores. Polyglot persistence reigns in rapidly growing environments and the traditional one-size-fits-all strategy of monoglots is over. VidaXL has a broad landscape of datastores, ranging from traditional SQL data stores, like MySQL or PostgreSQL alongside more recent load balancing technologies such as ProxySQL, to document stores like MongoDB and search engines such as SOLR and Elasticsearch.

If you’re looking at minimizing downtime and meet your SLAs through an automated or semi-automated approach, then this webinar is for you:

A detailed overview of what failover processes may look like in MySQL, MariaDB and PostgreSQL replication setups.

Failover is the process of moving to a healthy standby component, during a failure or maintenance event, in order to preserve uptime. The quicker it can be done, the faster you can be back online.

However, failover can be tricky for transactional database systems as we strive to preserve data integrity - especially in asynchronous or semi-synchronous topologies.

There are risks associated: from diverging datasets to loss of data. Failing over due to incorrect reasoning, e.g., failed heartbeats in the case of network partitioning, can also cause significant harm.

In this webinar we’ll cover the dangers related to the failover process, and discuss the tradeoffs between failover speed and data integrity. We’ll find out about how to shield applications from database failures with the help of proxies.

And we will finally have a look at how ClusterControl manages the failover process, and how it can be configured for both assisted and automated failover.

Date, Time & Registration

Europe/MEA/APAC

Tuesday, December 11th at 09:00 GMT / 10:00 CET (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, December 11th at 09:00 PT (US) / 12:00 ET (US)

Register Now

Agenda

• An introduction to failover - what, when, how
  • in MySQL / MariaDB
  • in PostgreSQL
• To automate or not to automate
• Understanding the failover process
• Orchestrating failover across the whole HA stack
• Difficult problems
  • Network partitioning
  • Missed heartbeats
  • Split brain
• From assisted to fully automated failover with ClusterControl
  • Demo

Speaker

I’d like to take advantage of the quiet days between holidays to look back on 2018 at Severalnines as we continue to advance automation and management of the world’s most popular open source databases: MySQL, MariaDB, PostgreSQL & MongoDB!

And take this opportunity to thank you all for your support in the past 12 months and celebrate some of our successes with you …

2018 Severalnines Momentum Highlights:

• Over 160% increase in registrations for our flagship product ClusterControl
• Over 25K ClusterControl installations to date
• New customer wins such as Mediacloud, Instant Gaming, Can’l, build.com, VentraIP, and more
• Announced an enhanced ClusterControl Community Edition with brand new monitoring features
• Introduced a new look for ClusterControl Enterprise Edition with its 10 feature modules

For those who don’t know about it yet, ClusterControl helps database users deploy, monitor, manage and scale SQL and NoSQL open source databases such as MySQL, MariaDB, PostgreSQL and MongoDB.

Automation and control of open source database infrastructure across mixed environments makes ClusterControl the ideal polyglot solution to support modern businesses - be they large or small.

The reason for ClusterControl’s popularity is the way it provides full operational visibility and control for open source databases.

But don’t take my word for it: we’ve published a year-end video this week that not only summarises our year’s achievements, but also includes customer and user quotes highlighting why they’ve chosen ClusterControl to help them administer their open source database infrastructure.

As a self-funded (mature) startup, our team’s focus is solely on solving pressing customer and community user needs. We do so with our product of course, but just as importantly also through our content contributions to the open source database community. We publish technical content daily that ranges from blogs to white papers, webinars and more.

These Are Our Top Feature & Content Hits in 2018

Top 3 New ClusterControl Features

SCUMM: agent-based monitoring infrastructure & dashboards

SCUMM - Severalnines CMON Unified Monitoring and Management - introduces new agent-based monitoring infrastructure with a server pulling metrics from agents that run on the same hosts as the monitored databases and uses Prometheus agents for greater accuracy and customization options while monitoring your database clusters.

Cloud database deployment

Introduces tighter integration with AWS, Azure and Google Cloud, so it is now possible to launch new instances and deploy MySQL, MariaDB, MongoDB and PostgreSQL directly from the ClusterControl user interface.

Comprehensive automation and management of PostgreSQL

Throughout the year, we’ve introduced a whole range of new features for PostgreSQL: from full backup and restore encryption for pg_dump and pg_basebackup, continuous archiving and Point-in-Time Recovery (PITR) for PostgreSQL, all the way to a new PostgreSQL performance dashboard.

Top 3 Most Read New Blogs

My Favorite PostgreSQL Queries and Why They Matter

Joshua Otwell presents a combination of eight differing queries or types of queries he has found interesting and engaging to explore, study, learn, or otherwise manipulate data sets.

A Performance Cheat Sheet for PostgreSQL

Sebastian Insausti discusses how one goes about analyzing the workload, or queries, that are running, as well as review some basic configuration parameters to improve the performance of PostgreSQL databases.

Deploying PostgreSQL on a Docker Container

Our team explains how to use Docker to run a PostgreSQL database.

Top 3 Most Downloaded White Papers

MySQL on Docker - How to Containerize the Dolphin

Covers the basics you need to understand when considering to run a MySQL service on top of Docker container virtualization. Although Docker can help automate deployment of MySQL, the database still has to be managed and monitored. ClusterControl can provide a complete operational platform for production database workloads.

PostgreSQL Management & Automation with ClusterControl

Discusses some of the challenges that may arise when administering a PostgreSQL database as well as some of the most important tasks an administrator needs to handle; and how to do so effectively … with ClusterControl. See how much time and effort can be saved, as well as risks mitigated, by the usage of such a unified management platform.

How to Design Highly Available Open Source Database Environments

Discusses the requirements for high availability in database setups, and how to design the system from the ground up for continuous data integrity.

Top 3 Most Watched Webinars

Our Guide to MySQL & MariaDB Performance Tuning

Watch as Krzysztof Książek, Senior Support Engineer at Severalnines, walks you through the ins and outs of performance tuning for MySQL and MariaDB, and share his tips & tricks on how to optimally tune your databases for performance.

Designing Open Source Databases for High Availability

From discussing high availability concepts through to failover or switch over mechanisms, this webinar covers all the need-to-know information when it comes to building highly available database infrastructures.

Backup Management for MySQL, MariaDB, PostgreSQL & MongoDB with ClusterControl

Whether you are looking at rebuilding your existing backup infrastructure, or updating it, then this webinar is for you: watch replay of this webinar on Backup Management for MySQL, MariaDB, PostgreSQL and MongoDB with ClusterControl.

Thanks again for your support this year and “see you” in 2019!

Happy New Year from everyone at Severalnines!

PS.: To join Severalnines’ growing customer base please click here

We are excited to announce the 1.7.1 release of ClusterControl - the only management system you’ll ever need to take control of your open source database infrastructure!

ClusterControl 1.7.1 introduces the next phase of our exciting agent-based monitoring features for MySQL, Galera Cluster, PostgreSQL & ProxySQL, a suite of new features to help users fully automate and manage PostgreSQL (including support for PostgreSQL 11), support for MongoDB 4.0 ... and more!

Release Highlights

Performance Management

• Enhanced performance dashboards for MySQL, Galera Cluster, PostgreSQL & ProxySQL
• Enhanced query monitoring for PostgreSQL: view query statistics

Deployment & Backup Management

• Create a cluster from backup for MySQL & PostgreSQL
• Verify/restore backup on a standalone PostgreSQL host
• ClusterControl Backup & Restore

Additional Highlights

• Support for PostgreSQL 11 and MongoDB 4.0

View the ClusterControl ChangeLog for all the details!

View Release Details and Resources

• Download ClusterControl
• ChangeLog
• ClusterControl Administration Guide
• Upgrade instructions

Release Details

Performance Management

Enhanced performance dashboards for MySQL, Galera Cluster, PostgreSQL & ProxySQL

Since October 2018, ClusterControl users have access to a set of monitoring dashboards that have Prometheus as the data source with its flexible query language and multi-dimensional data model, where time series data is identified by metric name and key/value pairs.

The advantage of this new agent-based monitoring infrastructure is that users can enable their database clusters to use Prometheus exporters to collect metrics on their nodes and hosts, thus avoiding excessive SSH activity for monitoring and metrics collections and use SSH connectivity only for management operations.

These Prometheus exporters can now be installed or enabled Prometheus on your nodes and hosts with MySQL, PostgreSQL and MongoDB based clusters. And you have the possibility to customize collector flags for the exporters (Prometheus), which allows you to disable collecting from MySQL's performance schema for example, if you experience load issues on your server.

This allows for greater accuracy and customization options while monitoring your database clusters. ClusterControl takes care of installing and maintaining Prometheus as well as exporters on the monitored hosts.

With this 1.7.1 release, ClusterControl now also comes with the next iteration of the following (new) dashboards:

• System Overview
• Cluster Overview
• MySQL Server - General
• MySQL Server - Caches
• MySQL InnoDB Metrics
• Galera Cluster Overview
• Galera Server Overview
• PostgreSQL Overview
• ProxySQL Overview
• HAProxy Overview
• MongoDB Cluster Overview
• MongoDB ReplicaSet
• MongoDB Server

Do check them out and let us know what you think!

MongoDB Cluster Overview

HAProxy Overview

Performance Management

Advanced query monitoring for PostgreSQL: view query statistics

ClusterControl 1.7.1 now comes with a whole range of new query statistics that can easily be viewed and monitored via the ClusterControl GUI. The following statistics are included in this new release:

• Access by sequential or index scans
• Table I/O statistics
• Index I/O statistics
• Database Wide Statistics
• Table Bloat And Index Bloat
• Top 10 largest tables
• Database Sizes
• Last analyzed or vacuumed
• Unused indexes
• Duplicate indexes
• Exclusive lock waits

Table Bloat & Index Bloat

Deployment

Create a cluster from backup for MySQL & PostgreSQL

To be able to deliver database and application changes more quickly, several tasks must be automated. It can be a daunting job to ensure that a development team has the latest database build for the test when there is a proliferation of copies, and the production database is in use.

ClusterControl provides a single process to create a new cluster from backup with no impact on the source database system.

With this new release, you can easily create MySQL Galera or PostgreSQL including the data from backup you need.

Backup Management

ClusterControl Backup/Restore

ClusterControl users can use this new feature to migrate a setup from one controller to another controller; and backup the meta-data of an entire controller or individual clusters from the s9s CLI. The backup can then be restored on a new controller with a new hostname/IP and the restore process will automatically recreate database access privileges. Check it out!

Additional New Functionalities

View the ClusterControl ChangeLog for all the details!

Download ClusterControl today!

Happy Clustering!

A few days ago was the release of a new version of ClusterControl, the 1.7.1, where we can see several new features, one of the main ones being the support for PostgreSQL 11.

To install PostgreSQL 11 manually, we must first add the repositories or download the necessary packages for the installation, install them and configure them correctly, depending on our infrastructure. All these steps take time, so let’s see how we could avoid this.

In this blog, we will see how to deploy this new PostgreSQL version with a few clicks using ClusterControl and how to manage it. As pre-requisite, please install the 1.7.1 version of ClusterControl on a dedicated host or VM.

Deploy PostgreSQL 11

To perform a new installation from ClusterControl, simply select the option “Deploy” and follow the instructions that appear. Note that if you already have a PostgreSQL 11 instance running, then you need to select the ‘Import Existing Server/Database’ instead.

ClusterControl Deploy Option

When selecting PostgreSQL, we must specify User, Key or Password and port to connect by SSH to our PostgreSQL hosts. We also need the name for our new cluster and if we want ClusterControl to install the corresponding software and configurations for us.

ClusterControl Deploy Information 1

Please check the ClusterControl user requirement for this task here.

ClusterControl Deploy Information 2

After setting up the SSH access information, we must define the database user, version and datadir (optional). We can also specify which repository to use. In this case, we want to deploy PostgreSQL 11, so just select it and continue.

In the next step, we need to add our servers to the cluster we are going to create.

ClusterControl Deploy Information 3

When adding our servers, we can enter IP or hostname.

In the last step, we can choose if our replication will be Synchronous or Asynchronous.

ClusterControl Deploy Information 4

We can monitor the status of the creation of our new cluster from the ClusterControl activity monitor.

ClusterControl Activity Section

Once the task is finished, we can see our new PostgreSQL 11 cluster in the main ClusterControl screen.

ClusterControl Main Screen

Once we have our cluster created, we can perform several tasks on it, like adding a load balancer (HAProxy) or a new replica.

ClusterControl Cluster Section

Scaling PostgreSQL 11

If we go to cluster actions and select “Add Replication Slave”, we can either create a new replica from scratch, or add an existing PostgreSQL database as a replica.

ClusterControl Add Replication Slave Option

Let's see how adding a new replication slave can be a really easy task.

ClusterControl Add Replication Slave Information

As you can see in the image, we only need to choose our Master server, enter the IP address for our new slave server and the database port. Then, we can choose if we want ClusterControl to install the software for us, and if the replication slave should be Synchronous or Asynchronous.

In this way, we can add as many replicas as we want and spread read traffic between them using a load balancer, which we can also implement with ClusterControl.

We can see more information about the HA for PostgreSQL in a related blog.

From ClusterControl, you can also perform different management tasks like Reboot Host, Rebuild Replication Slave or Promote Slave, with one click.

ClusterControl Node Actions

Backups

In previous blogs we took a look on the backup and PITR ClusterControl features for PostgreSQL. Now, in the last ClusterControl version, we have the "verify/restore backup on a standalone host" and "create a cluster from an existing backup" features.

In ClusterControl, select your cluster and go to the "Backup" section to see your current backups.

ClusterControl Backups Section

In the "Restore" option, first, you can choose which backup will be restored.

ClusterControl Restore Backup Option

There, we have three options.

ClusterControl Restore on node Option

The first one is the classic "Restore on node" option. This just restores the selected backup on a specific node.

ClusterControl Restore and verify on standalone host Option

The "Restore and verify on standalone host" option is a new ClusterControl PostgreSQL feature. This allows us to test the generated backup by restoring it on a standalone host. This is really useful to avoid any surprises in a disaster recovery scenario.

To use this feature, we need a dedicated host (or VM) that is not part of the cluster.

ClusterControl Restore and verify on standalone host Information

Add the dedicated host IP address and choose the desire options.

ClusterControl Verified Backup

When the backup is verified, you can see the "Verified" icon in the backup list.

ClusterControl Create Cluster From Backup Option

"Create Cluster From Backup" is another important new ClusterControl PostgreSQL feature.

As the name suggests it , this feature allow us to create a new PostgreSQL cluster with the data from the generated backup.

After choosing this option, we need to follow the same steps that we saw in the deploy section.

ClusterControl Create Cluster From Backup Information

All the configuration like user, number of nodes or replication type can be different in this new cluster.
When the new cluster is created, you can see both, the old and the new one in the ClusterControl main screen.

ClusterControl Main Screen

Conclusion

As we have seen above, you can now deploy the latest PostgreSQL release, version 11 using ClusterControl. Once deployed, ClusterControl provides a whole range of features, from monitoring, alerting, automatic failover, backup, point-in-time recovery, backup verification, to scaling of read replicas. This can help you manage Postgres in a friendly and intuitive way. Give it a try!

HAProxy is one of the most popular load balancers for MySQL and MariaDB.

Feature-wise, it cannot be compared to ProxySQL or MaxScale, but it is fast and robust and may work perfectly fine in any environment as long as the application can perform the read/write split and send SELECT queries to one backend and all writes and SELECT … FOR UPDATE to a separate backend.

Keeping track of the metrics made available by HAProxy is very important. You have to be able to know the state of your proxy, especially if you encountered any issues.

ClusterControl always made available an HAProxy status page, which showed the state in real time. With the new, Prometheus-based SCUMM (Severalnines ClusterControl Unified Monitoring & Management) dashboards, it is now possible to track how those metrics change in time.

In this blog post we will go over the different metrics presented in the HAProxy SCUMM dashboard.

First of all, by default Prometheus and SCUMM dashboards are disabled in ClusterControl. In that case, it’s just a matter of one click to deploy them for a given cluster. If you have multiple clusters monitored by ClusterControl you can reuse the same Prometheus instance for every cluster managed by ClusterControl.

Once deployed, we can access the HAProxy dashboard. We will go over the data shown in it.

As you can see, it starts with the information about the state of the backends. Here, please note this may depend on the cluster type and how you deployed HAProxy. In this case, it was a Galera cluster and HAProxy was deployed in a round-robin fashion; therefore you see three backends for reads and three for writes, six in total. It is also the reason why you see all backends marked as up. In case of the replication cluster, things are looking different as the HAProxy will be deployed in a read/write split, and the scripts will keep only one host (master) up and running in the writer’s backend:

This is why on the screen above, you can see two backend servers marked as “down”.

Next graph focuses on the data sent and received by both backend (from HAProxy to the database servers) and frontend (between HAProxy and client hosts).

We can also check the traffic distribution between backends that are configured in HAProxy. In this case we have two backends and the queries are sent via port 3308, which acts as the round-robin access point to our Galera Cluster.

We can also find graphs showing how the traffic was distributed across all backend servers. In this case, due to round-robin access pattern, data was more or less evenly distributed across all three backend Galera servers.

Next, we can see information about sessions. How many sessions were opened from HAProxy to the backend servers. We can also track how many times per second a new session was opened to the backend. You can also check how those metrics look like when you look at them on per backend server basis.

Next two graphs show what was the maximum number of sessions per backend server and also when some connectivity issues showed up. This can be quite useful for debugging purposes when you hit some configuration error on your HAProxy instance and connections started to be dropped.

Next graph might be even more valuable as it shows different metrics related to error handling - response errors, request errors, retries on the backend side etc. Then we have a Sessions graph, which shows the overview of the session metrics.

On the next graph we can track the connection errors in time, this can be also useful to pinpoint the time when the issue started to evolve.

Finally, two graphs related to queued requests. HAProxy queues requests to backend if the backend servers are oversaturated. This can point to, for example, the overloaded database servers, which cannot cope with more traffic.

As you can see, ClusterControl tracks the most important metrics of HAProxy and can show how they change in time. This is very useful in pinpointing when an issue started and, to some extent, what could be the root cause of it. Try it out (it’s free) for yourself.

Monitoring and management is critical to any production environment, and performance matters. Slow user interfaces that lag or do not respond, delayed alerts, cluster job timeouts when the server is starved of resources are all things that can cause trouble. There are ways to improve performance of ClusterControl, especially if you are managing multiple clusters and each cluster contains multiple nodes. This blog provides some tuning tips. The points elaborated here are curated based on our experience dealing with performance issues reported by our users and customers.

As an introduction, ClusterControl consists of several main components - a web application (frontend) based on PHP together with a number of daemonized processes (backend), these leverage a MySQL/MariaDB database for persistent storage. You are effectively controlling your cluster from the web application, which will be translated to a series of process calls executed by the backend processes to manage and monitor your database clusters.

MySQL Database

ClusterControl components rely on a MySQL or MariaDB database as the persistent store for monitoring data collected from the managed nodes, as well as all ClusterControl meta data (e.g. what jobs there are in the queue, backup schedules, backup statuses, etc.). By default, the installer script will install whatever version comes by the standard repository of the OS. The following is the MySQL/MariaDB version being installed by the installer:

• CentOS/Redhat 6 - MySQL 5.1
• CentOS/Redhat 7 - MariaDB 5.5
• Ubuntu 18.04 (Bionic) - MySQL 5.7
• Ubuntu 16.04 (Xenial) - MySQL 5.7
• Ubuntu 14.04 (Trusty) - MySQL 5.5
• Debian 9 (Stretch) - MySQL 5.5
• Debian 8 (Jessie) - MySQL 5.5
• Debian 7 (Wheezy) - MySQL 5.5

The installer script would do some basic tuning like configuring datadir location, MySQL port, user and also InnoDB buffer pool size at the very beginning of the installation stage. However, the tuning might not be suitable once you have imported or created more clusters/nodes. With an increased number of nodes to be monitored and managed, ClusterControl would use more resources and the database layer is commonly the first bottleneck that users encounter. Some further tuning might be needed to contain the incoming load.

ClusterControl is smart enough to detect any performance anomaly by writing up the following lines inside cmon_X.log (where X is the cluster ID):

2018-11-28 01:30:00 : (INFO) CmonSheetManager at 0x3839af0 loaded 70 values for key 'diskstat' between 2018-11-23 01:30:00 - 2018-11-28 01:30:0
0.
2018-11-28 01:30:00 : (INFO) SQL processing: 220.0000ms
2018-11-28 01:30:00 : (INFO) Parsing       : 0.0000ms
2018-11-28 01:30:00 : (INFO) Sum           : 220.0000ms

The above simply means it took 220ms (Sum value) to load 70 values for component "diskstat", where most of the processing time was happening at the SQL processing stage and 0 ms to parse the SQL resultset. This concludes that the SQL layer takes most of the processing time when ClusterControl was trying to query the dataset.

We believe most of the SQL queries executed by ClusterControl are properly optimized for single MySQL instance and use proper indexing. Simply said, if you see something like the above appearing regularly in the log file, some improvements to the database layer are in order, as shown in the next sections.

Tuning InnoDB Buffer Pool Size

Buffer pool size is an important component and has to be configured upfront to improve the MySQL performance. It allows MySQL processing to be happening inside memory, instead of hitting the disk. A simple rule of thumb is to check the InnoDB hit ratio and look for the following line under BUFFER POOL AND MEMORY section:

Buffer pool hit rate 986 / 1000

The hit rate of 986 / 1000 indicates that out of 1000 row reads, it was able to read the row in RAM 986 times. The remaining 14 times, it had to read the row of data from disk. Simply said, 1000 / 1000 is the best value that we are trying to achieve here, which means the frequently-accessed data fits fully in RAM.

Increasing the innodb_buffer_pool_size value will help a lot to accomodate more room for MySQL to work on. However, ensure you have sufficient RAM resources beforehand. By default, ClusterControl allocates 50% of the RAM to the buffer pool. If the host is dedicated to ClusterControl, you can even push it to a higher value like 70%, provided you spare at least 2GB of RAM to the OS processes and caches. If you can't allocate that much, increasing the RAM is the only solution.

Changing this value requires a MySQL restart (older than MySQL 5.7.5), thus the correct service restart ordering will be:

1. Modify innodb_buffer_pool_size value inside my.cnf.
2. Stop the CMON service.
3. Restart MySQL/MariaDB service.
4. Start the CMON service.

Or simply reboot the host if you can afford a longer downtime.

Tuning max_connections

By default, the installer script will configure max_connections value up to 512. This is rather high, although sane, since ClusterControl barely reaches 200 connections in total, unless the MySQL server is shared with other applications or you have tens of MySQL nodes monitored by ClusterControl (we are talking about 30 nodes and more).

A high max_connections value wastes resources and adjusting the value will affect the maximum memory configured for MySQL. If it is greater than System RAM then there is a chance that the MySQL Server process will terminate with an OOM exception, if all connections are used.

To check on this, simply look for max_used_connections MySQL status. The following is the maximum connections ever reached by MySQL on a ClusterControl node that monitors 2 clusters with 6 MySQL nodes in total:

mysql> SHOW STATUS like 'max_used_connections';
+----------------------+-------+
| Variable_name        | Value |
+----------------------+-------+
| Max_used_connections | 43    |
+----------------------+-------+

A good value to start is Max_used_connections x 2, and gradually increase it if the value is consistently growing. Modifying the max_connections variable can be done dynamically via SET GLOBAL statement.

Using MySQL socket file

By default, the installer script will automatically configure the following host value inside every ClusterControl database configuration files:

The above will force the MySQL client to connect via TCP networking, just like connecting to a remote MySQL host although ClusterControl is running on the same server as the MySQL server. We purposely configured it this way to simplify the installation process since almost every OS platform pre-configures MySQL socket file differently, which greatly reduce the installation failure rate.

Changing the value to "localhost" will force the client to use the MySQL Unix socket file instead:

On Unix based systems, MySQL programs treat the host name localhost specially, in a way that is likely different from what you expect compared to other network-based programs. For connections to localhost, MySQL programs attempt to connect to the local server by using a Unix socket file. This occurs even if a --port or -P option is given to specify a port number.

Using MySQL UNIX socket file is much more secure and will cut off the network overhead. It is always recommended over TCP. However, you need to make sure the socket file is configured correctly. It must exist on the following directives inside my.cnf and every MySQL option files on ClusterControl node, for example:

[mysqld]
socket=/var/lib/mysql/mysql.sock

[client]
socket=/var/lib/mysql/mysql.sock

[mysql]
socket=/var/lib/mysql/mysql.sock

[mysqldump]
socket=/var/lib/mysql/mysql.sock

Changing the socket file will also require a MySQL and CMON restart. If you are using the "localhost", you can then add some additional configuration options like skip-networking=1, to prevent accepting remote connections. Our ClusterControl Docker image is using this approach to overcome a limitation in docker-proxy when binding on ports.

OpenSSH with SSSD

ClusterControl uses SSH protocol as its main communication channel to manage and monitor remote nodes. The default OpenSSH configurations are pretty decent and should work in most cases. However, in some environments where SSH is integrated with other security enhancement tools like SElinux or System Security Services Daemon (SSSD), it could bring significant impact to the SSH performance.

We have seen many cases where an ever increasing amount of SSH connections established to each of the managed nodes and eventually, both the ClusterControl server and all managed nodes max out their system memory with SSH connections. In some cases, only a normal full system reboot nightly on the ClusterControl server could solve the problem.

If you running your infrastructure with System Security Services Daemon (SSSD), it's advised for you to comment the following line inside OpenSSH client configuration at /etc/ssh/ssh_config on ClusterControl node:

#ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h

The above will skip using SSSD to manage the host key, which will be handled by OpenSSH client instead.

Starting from ClusterControl 1.7.0, you have an option to use agent-based monitoring tool with Prometheus. With agent-based monitoring, ClusterControl does not use SSH to sample host metrics which can be excessive in some environments.

File System and Partitioning

ClusterControl controller writes new entry in its log file almost every second for every cluster. For those who wants to take advantage of this sequential writes on disk and would like to save cost, you can use a spindle disk for this purpose. Modify the following line inside all cmon_X.cnf:

logfile=/new/partition/log/cmon_X.log

Replace X with the cluster ID and restart CMON service to apply the changes.

If you are using ClusterControl as the backup repository, it's recommended for you to allocate sufficient disk space on a separate partition other than the root partition. It gets better if the partition resides on a networked or clustered file system for easy mounting with the targeted nodes when performing restoration operation. We have seen cases where the created backups ate up all disk space of the main partition, eventually impacting ClusterControl and its components.

Keep up to Date

ClusterControl has a short release cycle - at least one new major release every quarter of the year plus weekly maintenance patches (mostly bug fixes - if any). The reason is ClusterControl supports multiple database vendors and versions, operating systems and hardware platforms. Often there are new things being introduced and old things being deprecated from what is provided and ClusterControl has to keep up with all the changes introduced by application vendors and follow the best-practice every time.

We recommend users to always use the latest version of ClusterControl (upgrading is easy) together with the latest web browser (built and tested on Google Chrome and Mozilla Firefox), as we are very likely taking advantage of the new features available in the latest version.

Final Thoughts

Do reach us via our support channel if you face any problems or slowness issues when using ClusterControl. Suggestions and feedback are very much welcome.

Happy tuning!

ClusterControl 1.7.1 introduced a new feature which allows you to backup your ClusterControl server and restore it (together with metadata about your managed databases) onto another server. It backs up the ClusterControl application as well as all its configuration data. Migrating ClusterControl to a new server used to be a pain, but not any more.

This blog post walks you through this new feature.

We will migrate ClusterControl from one server to another, preserving all the configurations and settings.

We will also show you how to transfer the management of a cluster from one ClusterControl instance to another.

Our example architecture started with two production clusters (shown in the screenshot below):

• Cluster ID 1: 3 Galera nodes (PXC) + 1 HAProxy + 1 ProxySQL (5 nodes)
• Cluster ID 2: 1 MySQL master + 2 MySQL slaves + 1 ProxySQL (4 nodes)

Introduction

ClusterControl CLI (s9s) is a command line interface tool to interact, control and manage database clusters using the ClusterControl Platform. Starting from version 1.4.1, the installer script will automatically install this package on the ClusterControl node.

There are basically 4 new options introduced under "s9s backup" command, which can be used to achieve our objective:

This blog post will cover example use cases on how to utilize those options. At the moment, they are in release candidate stage and only available via ClusterControl CLI tool.

Backing Up ClusterControl

In order to do this, the ClusterControl server must be at least on v1.7.1 and later. To backup ClusterControl controller, simply run the following command on the ClusterControl node as root user (or with sudo):

$ s9s backup \
--save-controller \
--backup-directory=$HOME/ccbackup \
--output-file=controller.tar.gz \
--log

The --output-file must be a filename or physical path (if you want to omit --backup-directory flag), and the file must not exist beforehand. ClusterControl won't replace the output file if it already exists. By specifying --log, it will wait until the job is executed and the job logs will be shown in the terminal. The same logs can be accessed via ClusterControl UI under Activity -> Jobs -> Save Controller:

The 'Save Controller' job basically performs the following procedures:

1. Retrieve the controller configuration and export it to JSON
2. Export CMON database as MySQL dump file
3. For every database cluster:
   1. Retrieve the cluster configuration and export it to JSON

In the output, you may notice the job found is N + 1 cluster, for example "Found 3 cluster(s) to save" even though we only have two database clusters. This includes cluster ID 0, which carries special meaning in ClusterControl as the global initialized cluster. However, it does not belong to CmonCluster component, which is the database cluster under ClusterControl management.

Restoring ClusterControl to a new ClusterControl server

Supposed ClusterControl is already installed on the new server, we would like to migrate the database clusters to be managed by the new server. The following diagram illustrates our migration exercise:

Firstly, transfer the backup from the old server into the new server:

$ scp $HOME/ccbackup/controller.tar.gz 192.168.0.190:~

Before we perform the restoration, we have to set up passwordless SSH to all nodes from the new ClusterControl server:

$ ssh-copy-id 192.168.0.11 #proxysql cluster 1
$ ssh-copy-id 192.168.0.12 #proxysql cluster 1
$ ssh-copy-id 192.168.0.21 #pxc cluster 1
$ ssh-copy-id 192.168.0.22 #pxc cluster 1
$ ssh-copy-id 192.168.0.23 #pxc cluster 1
$ ssh-copy-id 192.168.0.30 #proxysql cluster 2
$ ssh-copy-id 192.168.0.31 #mysql cluster 2
$ ssh-copy-id 192.168.0.32 #mysql cluster 2
$ ssh-copy-id 192.168.0.33 #mysql cluster 2

Then, on the new server, perform the restoration:

$ s9s backup \
--restore-controller \
--input-file=$HOME/controller.tar.gz \
--debug \
--log

Then we have to sync the cluster in the UI by going to Global Settings -> Cluster Registrations -> Synchronize Cluster. Then if you go back to the ClusterControl main dashboard, you would see the following:

Don't panic. The new ClusterControl UI is not able to retrieve the monitoring and management data because of incorrect RPC API token. We just need to update it accordingly. First, retrieve the rpc_key value for the respective clusters:

$ cat /etc/cmon.d/cmon_*.cnf | egrep 'cluster_id|rpc_key'
cluster_id=1
rpc_key=8fgkzdW8gAm2pL4L
cluster_id=2
rpc_key=tAnvKME53N1n8vCC

In the UI, click the link "here" at the then of the "Change the RPC API token here" line. It will pop up the following dialog:

Paste the respective rpc_key value in the text field and click Save. Repeat for the next cluster. Wait for a moment and the cluster list should be refreshed automatically.

The last step is to fix the MySQL cmon user privileges for the new ClusterControl IP address changes, 192.168.0.190. Login to one of the PXC node and run the following:

$ mysql -uroot -p -e 'GRANT ALL PRIVILEGES ON *.* TO cmon@"192.168.0.190" IDENTIFIED BY "<password>" WITH GRANT OPTION';

** Replace <password> with identical cmon MySQL password as in mysql_password value inside /etc/cmon.cnf. Repeat the same step on the second cluster, MySQL replication but only execute it once on the master node.

Once the privilege is set up, you should see the cluster list is in green, similar to the old one:

It's worth to mention that by default, ClusterControl will disable the cluster automatic recovery (as you can see the red icon next to the word 'Cluster') to avoid race condition with another ClusterControl instance. It's recommended to enable this feature (by clicking the icon to green) once the old server has been decommissioned.

Our migration is now completed. All the configurations and settings from the old server are preserved and transferred to the new server.

Migrating the Management of a Cluster to another ClusterControl server

Backing Up Cluster Information

This is about backing up cluster metadata and information so we can transfer it to another ClusterControl server, also known as partial backup. Otherwise, we have to perform "Import Existing Server/Cluster" to re-import them into the new ClusterControl which means you would lose the monitoring data from the old server. If you have load balancers or asynchronous slave instances, this would have to be imported once the cluster is imported, one node at a time. So it is a bit hassle if you have a complete set of production setup.

The cluster "manager" migration exercise is illustrated in the following diagram:

Basically, we want to migrate out our MySQL Replication (cluster ID: 2) to be managed by another ClusterControl instance. We are going to use --save-cluster-info and --restore-cluster-info options for this one. The --save-cluster-info option will export the corresponding cluster information to be saved somewhere else. Let's export our MySQL Replication Cluster (cluster ID: 2). On the current ClusterControl server, do:

$ s9s backup \
--save-cluster-info \
--cluster-id=2 \
--backup-directory=$HOME/ccbackup \
--output-file=cc-replication-2.tar.gz \
--log

You will see a bunch of new lines printed in the terminal, indicating the backup job is running (the output is also accessible via ClusterControl -> Activity -> Jobs):

If you look at the job logs closely, you would notice the job was trying to export all the related information and metadata for cluster ID 2. The output is stored as a compressed file and located under path that we have specified under using --backup-directory flag. If this flag is ignored, ClusterControl will save the output to the default backup directory which is the home directory of the SSH user, under $HOME/backups.

Restoring Cluster Information

The steps explained here are similar with the restoration steps for ClusterControl full backup. Transfer the backup from the current server to the other ClusterControl server:

$ scp $HOME/ccbackup/cc-replication-2.tar.gz 192.168.0.190:~

Before we perform the restoration, we have to set up passwordless SSH to all nodes from the new ClusterControl server:

$ ssh-copy-id 192.168.0.30 #proxysql cluster 2
$ ssh-copy-id 192.168.0.31 #mysql cluster 2
$ ssh-copy-id 192.168.0.32 #mysql cluster 2
$ ssh-copy-id 192.168.0.33 #mysql cluster 2
$ ssh-copy-id 192.168.0.19 #prometheus cluster 2

Then, on the new server, perform the cluster information restoration for our MySQL Replication:

$ s9s backup \
--restore-cluster-info \
--input-file=$HOME/cc-replication-2.tar.gz \
--log

You can verify the progress under Activity -> Jobs -> Restore Cluster:

If you look at the job messages closely, you can see that ClusterControl automatically re-assigns cluster ID to 1 on this new instance (it was cluster ID 2 on the old instance).

Then sync the cluster in the UI by going to Global Settings -> Cluster Registrations -> Synchronize Cluster. If you go back to the ClusterControl main dashboard, you would see the following:

The error means the new ClusterControl UI is not able to retrieve the monitoring and management data because of incorrect RPC API token. We just need to update it accordingly. Firstly, retrieve the rpc_key value for our cluster ID 1:

$ cat /etc/cmon.d/cmon_1.cnf | egrep 'cluster_id|rpc_key'
cluster_id=1
rpc_key=tAnvKME53N1n8vCC

In the UI, click the link "here" at the then of the "Change the RPC API token here" line. It will pop up the following dialog:

Paste the respective rpc_key value in the text field and click Save. Wait for a moment and the cluster list should be refreshed automatically.

The last step is to fix the MySQL cmon user privileges for the new ClusterControl IP address changes, 192.168.0.190. Login to the master node (192.168.0.31) and run the following statement:

$ mysql -uroot -p -e 'GRANT ALL PRIVILEGES ON *.* TO cmon@"192.168.0.190" IDENTIFIED BY "<password>" WITH GRANT OPTION';

** Replace <password> with identical cmon MySQL password as in mysql_password value inside /etc/cmon.cnf.

You may also revoke the old user privileges (revoke won't delete the user) or simply drop the old user:

$ mysql -uroot -p -e 'DROP USER cmon@"192.168.0.19"'

Once the privilege is set up, you should see everything is green:

At this point, our architecture is looking something like this:

Our migration exercise is now complete.

Final Thoughts

It's now possible to perform full and partial backup of your ClusterControl instances and the clusters they manage, allowing you to move them freely between hosts with little efforts. Suggestions and feedback are welcome.

While MongoDB has spent nearly a decade achieving maturity (initial release Feb 2009), the technology is a bit of a mystery to those experienced in conventional relational database (RDBMS) environments. Integrating NoSQL into an existing environment without in-depth knowledge can be challenging. It is not uncommon to see MongoDB running along MySQL or another RDBMS database.

The experience of RDBMS may help to understand some of the processes, but you need to know how to translate your expertise into the NoSQL world. Managing production environments involves steps like deployment, monitoring uptime and performance, maintaining system security, managing HA, backups and so on. Both RDBMS and NoSQL are viable options, but there are specific critical differences between the two that users must keep in mind while implementing or managing MongoDB. Technology changes rapidly and we need to adapt fast.

When MongoDB is suddenly your responsibility, management tools guarantees that the MongoDB databases you manage are stable and secure. Using predefined processes and automation can not only save you time but also protect from the common mistakes. A management platform that systematically addresses all the different aspects of the database lifecycle will be more robust than patching together a number of point solutions.

At the heart of ClusterControl is its automation functionality that lets you automate the database tasks you have to perform regularly, like deploying new databases, adding and scaling new nodes, managing backups, high availability and failover, topology changes, upgrades, and more. ClusterControl provides programmed security, keeping the integrity of your database infrastructure. Moreover, with ClusterControl, MongoDB users are no longer subject to vendor lock-in; something that was questioned by many recently. You can deploy and import a variety of MongoDB versions and vendors from a single console for free. Users of MongoDB often have to use a mixture of tools and homegrown scripts to achieve their requirements, and it's good to know you can find them combined in the one product.

In this article, we will show you how to deploy and manage MongoDB 4.0 in an automated way. You will find here how to do:

• ClusterControl installation
• MongoDB deployment process
  • Deploy a new cluster
  • Import existing cluster
• Scaling MongoDB
  • Read scaling (replicaSet)
  • Write scaling (sharding)
• Securing MongoDB
• Monitoring and Trending
• Backup and Recovery

ClusterControl installation

To start with ClusterControl you need a dedicated virtual machine or host. The VM and supported systems requirements are described here. The base VM can start from 2 GB, 2 cores and Disk space 20 GB storage space, either on-prem or in the cloud.

The installation is well described in the documentation but basically, it comes to download of the installation script which will walk you through the wizard. The wizard script set up the internal database, installs necessary packages, repositories, and do other necessary tweaks. For the internet lock environments, you can use the offline installation process.

ClusterControl requires SSH access to the database hosts, and monitoring can be agent-based or agentless. Management is agentless.

Setup passwordless SSH to all target nodes (ClusterControl and all database hosts) involves running following commands on the ClusterControl server:

$ ssh-keygen -t rsa # press enter on all prompts
$ ssh-copy-id -i ~/.ssh/id_rsa [ClusterControl IP address]
$ ssh-copy-id -i ~/.ssh/id_rsa [Database nodes IP address] # repeat this to all target database nodes

MongoDB Deployment and Scaling

Deploy a New MongoDB 4.0 Cluster

Once we enter the ClusterControl interface, the first thing to do is deploy a new cluster or import an existing one. The new version 1.7.1 introduces support for version 4.0. You can now deploy/import and manage MongoDB v4.0 with support for SSL connections.

Select the option “Deploy Database Cluster” and follow the instructions that appear.

ClusterControl Deploy Database Cluster

When choosing MongoDB, we must specify User, Key or Password and port to connect by SSH to our servers. We also need the name for our new cluster and if we want ClusterControl to install the corresponding software and configurations for us.

After setting up the SSH access information, we must enter the data to access our database. We can also specify which repository to use. Repository configuration is an important aspect for database servers and clusters. You can have three types of the repository when deploying database server/cluster using ClusterControl:

• Use Vendor Repository
  Provision software by setting up and using the database vendor’s preferred software repository. ClusterControl will install the latest version of what is provided by the database vendor repository.
• Do Not Setup Vendor Repositories
  Provision software by using the pre-existing software repository already set up on the nodes. The user has to set up the software repository manually on each database node and ClusterControl will use this repository for deployment. This is good if the database nodes are running without internet connections.
• Use Mirrored Repositories (Create new repository)
  Create and mirror the current database vendor’s repository and then deploy using the local mirrored repository. This allows you to “freeze” the current versions of the software packages.

In the next step, we need to add our servers to the cluster that we are going to create. When adding our servers, we can enter IP or hostname. For the latter, we must have a DNS server or have added our MongoDB servers to the local resolution file (/etc/hosts) of our ClusterControl, so it can resolve the corresponding name that you want to add. For our example, we will deploy a ReplicaSet with three servers, one primary and two secondaries. It is possible to deploy only 2 MongoDB nodes (without arbiter). The caveat of this approach is no automatic failover, since a 2-node setup is vulnerable to split brain. If the primary node goes down then manual failover is required to make the other server as primary. Automatic failover works fine with 3 nodes and more. It is recommended that a replica set has an odd number of voting members. Fault tolerance for a replica set is the number of members that can become unavailable and still leave enough members in the set to elect a primary. The fault tolerance for three members is one, for five it is two etc.

On the same page you can choose from different MongoDB versions:

ClusteControl Deploy MongoDB version 4.0

When all is set hit the deploy button. You can monitor the status of the creation of our new cluster from the ClusterControl activity monitor. Once the task is finished, we can see our cluster in the main ClusterControl screen and on the topology view.

ClusterControl Topology view

As we can see in the image, once we have our cluster created, we can perform several tasks on it, like converting replica set to shard or adding nodes to the cluster.

ClusterControl Scaling

Import a New Cluster

We also have the option to manage an existing cluster by importing it into ClusterControl. Such environment can be created by ClusterControl or other methods like docker installation.

ClusterControl import MongoDB

First, we must enter the SSH access credentials to our servers. Then we enter the access credentials to our database, the server data directory, and the version. We add the nodes by IP or hostname, in the same way as when we deploy, and press on Import. Once the task is finished, we are ready to manage our cluster from ClusterControl.

Scaling MongoDB

One of the cornerstones of MongoDB is that it is built with high availability and scaling in mind. Scaling can be done either vertically by adding more resources to the server or horizontally with more nodes. Horizontal scaling is what MongoDB is good at, and it is not much more than spreading the workload to multiple machines. In effect, we’re making use of multiple low-cost commodity hardware boxes, rather than upgrading to a more expensive high-performance server. MongoDB offers both read- and write scaling, and we will uncover the differences between these two strategies for you. Whether to choose read- or write scaling all depends on the workload of your application: if your application tends to read more often than it writes data you will probably want to make use of the read scaling capabilities of MongoDB.

With ClusterControl adding more servers to the cluster is an easy step. You can do that from the GUI or CLI. For more advanced users you can use ClusterControl Developer Studio and write an resource base condition to expand your cluster horizontally.

MongoDB ReplicaSet

Sharding

The MongoDB sharding solution is similar to existing sharding frameworks for other major database solutions. It makes use of a typical lookup solution, where the sharding is defined in a shard-key and the ranges are stored inside a configuration database. MongoDB works with three components to find the correct shard for your data. A typical sharded MongoDB environment looks like this:

MongoDB Sharding

The first component used is the shard router called mongos. All read and write operations must be sent to the shard router, making all shards act as a single database for the client application. The shard router will route the queries to the appropriate shards by consulting the Configserver.

ClusterControl Convert to Shard

Shard management is really easy in MongoDB. You can add and remove shards online and the MongoDB shard router will automatically adjust to what you tell it to. If you wish to know more in-depth about how best to manage shards, please read our blog post about managing MongoDB shards.

Securing MongoDB

MongoDB comes with very little security out of the box: for instance, authentication is disabled by default. In other words: by default, anyone has root rights over any database. One of the changes MongoDB applied to mitigate risks was to change its default binding to 127.0.0.1. This prevents it from being bound to the external IP address, but naturally, this will be reverted by most people who install it. ClusterControl removes human error and provides access to a suite of security features, to automatically protect your databases from hacks and other threats. We previously published a short video with security tips.

The new version of ClusterControl offers SSL support for MongoDB connections. Enabling SSL adds another level of security for communication between the applications (including ClusterControl) and database. MongoDB clients open encrypted connections to the database servers and verify the identity of those servers before transferring any sensitive information.

To enable SSL connection you need to use the latest s9s client. You can install it with

wget http://repo.severalnines.com/s9s-tools/install-s9s-tools.sh
chmod 755 install-s9s-tools.sh
./install-s9s-tools.sh

Or follow other possible installation methods described here.

Once you have s9s tools installed (min version 1.7-93.1) you can use --enable-ssl flag to enable SSL connection.

Example below:

[root@cmon ~]# s9s cluster --cluster-id=3 --enable-ssl --log
This is an RPC V2 job (a job created through RPC V2).
The job owner is 'admin'.
Accessing '/.runtime/jobs/jobExecutor' to execute...
Access ok.
Stopping the cluster
node1:27017: Node is already stopped by the user.
node2:27017: Node is already stopped by the user.
node3:27017: Node is already stopped by the user.
Checking/generating (expire 1000 days) server and CA certificate.
node1:27017: setting up SSL as required way of connection.
Using certificate 'mongodb/cluster_3/server'
node1:27017: installed /etc/ssl/mongodb/cluster_3/server.crt, /etc/ssl/mongodb/cluster_3/server.key and /etc/ssl/mongodb/cluster_3/server_ca.crt
node1:27017: Deploying client certificate 'mongodb/cluster_3/client'
Writing file 'node1:/etc/mongod.conf'.
node1:27017: /etc/mongod.conf [mongod] set: ssl_cert, ssl_key and ssl_ca values.
node2:27017: setting up SSL as required way of connection.
Using certificate 'mongodb/cluster_3/server'
node2:27017: installed /etc/ssl/mongodb/cluster_3/server.crt, /etc/ssl/mongodb/cluster_3/server.key and /etc/ssl/mongodb/cluster_3/server_ca.crt
node2:27017: Deploying client certificate 'mongodb/cluster_3/client'
Writing file 'node2:/etc/mongod.conf'.
node2:27017: /etc/mongod.conf [mongod] set: ssl_cert, ssl_key and ssl_ca values.
node3:27017: setting up SSL as required way of connection.
Using certificate 'mongodb/cluster_3/server'
node3:27017: installed /etc/ssl/mongodb/cluster_3/server.crt, /etc/ssl/mongodb/cluster_3/server.key and /etc/ssl/mongodb/cluster_3/server_ca.crt
node3:27017: Deploying client certificate 'mongodb/cluster_3/client'
Writing file 'node3:/etc/mongod.conf'.
node3:27017: /etc/mongod.conf [mongod] set: ssl_cert, ssl_key and ssl_ca values.
Starting the cluster
node3:27017: Doing some preparation for starting the node.
node3:27017: Disable transparent huge page and its defrag according to mongo suggestions.
node3:27017: Checking file permissions and ownership.
node3:27017: Starting mongod MongoDb server with command:
ulimit -u 32000 -n 32000 &&  runuser -s /bin/bash mongod '-c mongod -f /etc/mongod.conf'
node3:27017: Verifing that 'mongod' process is started.
SSL setup done.

ClusterControl will execute all necessary steps including certification creation on all cluster nodes. Such certificates can be maintained later on in the Key Management tab.

ClusterControl Key Management

Monitoring

When working with database systems, you should be able to monitor them. That will enable you to identify trends, plan for upgrades or improvements or react effectively to any problems or errors that may arise.

ClusterControl MongoDB overview

The new ClusterControl 1.7.1 adds high-resolution monitoring for MongoDB based. It's using Prometheus as the data store with PromQL query language. The list of dashboards includes MongoDB Server, MongoDB ReplicaSet, System Overview, and Cluster Overview Dashboards. ClusterControl installs Prometheus agents, configures metrics and maintains access to Prometheus exporters configuration via its GUI, so you can better manage parameter configuration like collector flags for the exporters (Prometheus). We described in details what can be monitored recently in the article How to Monitor MongoDB with Prometheus & ClusterControl.

ClusterControl MongoDB SCUMM Dashboards

Alerting

As a database operator, we need to be informed whenever something critical occurs on our database. The three main methods in ClusterControl to get an alert includes:

• email notifications
• integrations
• advisors

You can set the email notifications on a user level. Go to Settings > Email Notifications. Where you can choose between criticality and type of alert to be sent.

The next method is to use Integration services. This is to pass the specific category of events to the other service like ServiceNow tickets, Slack, PagerDuty etc. so you can create an advanced notification methods and integrations within your organization.

ClusterControl Integration Services

The last one is to involve sophisticated metrics analysis in Advisor section, where you can build intelligent checks and triggers. An example here could be an disk space usage prediction or cluster scaling by adding nodes when the workload reach preset level.

ClusterControl Advisors for MongoDB

Backup and Recovery

Now that you have your MongoDB replicaSet up and running, and have your monitoring in place, it is time for the next step: ensure you have a backup of your data.

ClusterControl Create Backup Policy

ClusterControl provides an interface for MongoDB backup management with support for scheduling and creative reports. It gives you two options for backup methods.

• Mongodump
• Mongodb consistent backup

Mongodump dumps all the data in Binary JSON (BSON) format to the specified location. Mongorestore can later on use the BSON files to restore your database. ClusterControl MongoDB consistent backup includes the transactions from the oplog that were executing while making the backup.

ClusterControl Backup Encryption

A good backup strategy is a critical part of any database management system. ClusterControl offers many options for backups and recovery/restore.

ClusterControl Backup Schedule Control

ClusterControl backup retention is configurable; you can choose to retain your backup for any time period or to never delete backups. AES256 encryption is employed to secure your backups against rogue elements. For rapid recovery, backups can be restored directly into the backup cluster - ClusterControl handles the full restore process from launch to cluster recovery, removing error-prone manual steps from the process.

Thanks to everyone who joined our first webinar of 2019!

Our colleague Sebastian Insausti walked us through all the ins and outs of how to automate and manage PostgreSQL with ClusterControl!

You can now watch the replay and read through the slides of this webinar to see how much time and effort can be saved, as well as risks mitigated, with the help of a unified management platform over the more traditional, manual methods

Running PostgreSQL in production comes with the responsibility for a business critical environment, which includes high availability, disaster recovery, and performance.

Ops staff worry whether …

• Databases are up and running
• Backups are taken and tested for integrity
• There are performance problems that might affect end user experience
• Failover will work properly in case of server failure without breaking applications
• And the list goes on ...

ClusterControl can be used to operationalize your PostgreSQL footprint across your enterprise.

It offers a standard way of deploying high-availability replication setups with auto-failover, integrated with load balancers offering a single endpoint to applications. It provides constant health and performance monitoring through rich dashboards, as well as backup management and point-in-time recovery.

We’ve seen a 152% increase in ClusterControl installations by PostgreSQL users last year, so make sure you don’t miss out on the trend - and watch the replay of this webinar today.

Agenda

• Managing PostgreSQL “the old way”:
  • Common challenges
  • Important tasks to perform
  • Tools that are available to help
• PostgreSQL automation and management with ClusterControl:
  • Deployment
  • Backup and recovery
  • HA setups
  • Failover
  • Monitoring
  • Live Demo

Speaker

Did you know that apart from the ClusterControl web UI, you can also use a command line interface to manage your PostgreSQL instances?

ClusterControl supports PostgreSQL streaming replication (both asynchronous and synchronous replication) as well as standalone PostgreSQL instance. We have put our best effort to make the command line interface to be close to the UI in terms of functionality available.

Why would you want to use the CLI?

It allows you to deploy an entire replication setup in one command, or perform a failover, or add new nodes to the setup. This integrates very well with your existing infrastructure automation code written in Ansible, Chef or Puppet.

This blog post provides a walkthrough on how to manage a PostgreSQL streaming replication cluster using ClusterControl CLI, or s9s.

Take note that most of the functionalities shown in this blog post requires you to have ClusterControl installed and running with a valid subscription, either commercial license or free trial license (valid up to 30-day after ClusterControl installation).

Cluster Deployment and Import

Deploying a new cluster

Before deploying a new cluster, or importing an existing PostgreSQL cluster into ClusterControl, ensure passwordless SSH from ClusterControl node to all database nodes is configured beforehand. Supposed we would want to deploy a new three-node PostgreSQL streaming replication, run the following commands on ClusterControl node:

$ whoami
root
$ ssh-keygen -t rsa # if you haven't generated SSH key
$ ssh-copy-id 192.168.0.91 # PostgreSQL1
$ ssh-copy-id 192.168.0.92 # PostgreSQL2
$ ssh-copy-id 192.168.0.93 # PostgreSQL3

On ClusterControl node, verify if you can the following command without password:

$ ssh 192.168.0.91 "ls /root"

If you can see the directory content, you are in a good shape. Next, use ClusterControl CLI with --create flag to deploy the cluster:

$ s9s cluster \
--create \
--cluster-type=postgresql \
--nodes="192.168.0.91?master;192.168.0.92?slave;192.168.0.93?slave" \
--provider-version='11' \
--db-admin='postgres' \
--db-admin-passwd='s3cr3tP455' \
--os-user=root \
--os-key-file=/root/.ssh/id_rsa \
--cluster-name='PostgreSQL 11 Streaming Replication' \
--wait
Creating PostgreSQL Cluster
\ Job 259 RUNNING    [█▋        ]  15% Installing helper packages

We specified the first node, 192.168.0.91 as the master and the rest are slaves. Since we already configured passwordless SSH via root user, we specified the OS user as "root" together with the corresponding SSH key file using --os-key-file flag. The --wait flag means the job will wait and report the progress until the it finishes.

You can also monitor the deployment progress from ClusterControl UI under Activity > Jobs > Creating PostgreSQL Cluster:

Once the deployment completes, we can see the summary of the running cluster by using the --stat flag:

$ s9s cluster --stat

Importing an existing cluster

If let's say you already have a PostgreSQL streaming replication cluster deployed manually, you can import it into ClusterControl using the --register flag as shown in the following command:

$ s9s cluster \
--register \
--cluster-type=postgresql \
--nodes="192.168.0.91;192.168.0.92;192.168.0.93" \
--provider-version='11' \
--db-admin='postgres' \
--db-admin-passwd='s3cr3tP455' \
--os-user=root \
--os-key-file=/root/.ssh/id_rsa \
--cluster-name="PostgreSQL 11" \
--wait
Register PostgreSQL
- Job 263 RUNNING    [        █ ] ---% Importing Cluster

ClusterControl will then connect to the specified nodes, discover the topology and register the cluster into ClusterControl. You can verify with 's9s cluster --stat' command as shown above.

Node and Cluster Management

Service Control

To perform a rolling restart of a cluster, specify the cluster ID and use --rolling-restart flag:

$ s9s cluster --rolling-restart --cluster-id=8 --wait
Rolling Restart
- Job 264 RUNNING    [██▊       ]  27% Waiting for 192.168.0.91

Use the --stop flag for component "cluster" to stop a cluster. To see the job output instead of progress bar, we can use --log flag instead:

$ s9s cluster --stop --cluster-id=8 --log
This is an RPC V2 job (a job created through RPC V2).
The job owner is 'admin'.
Accessing '/.runtime/jobs/jobExecutor' to execute...
Access ok.
Setting cluster to 'SHUTTING_DOWN' state.
192.168.0.91:5432: Stopping PostgreSQL service.
192.168.0.91:5432: Waiting PostgreSQL top stop.
192.168.0.92:5432: Stopping PostgreSQL service.
192.168.0.92:5432: Waiting PostgreSQL top stop.
192.168.0.93:5432: Stopping PostgreSQL service.
192.168.0.93:5432: Waiting PostgreSQL top stop.
Setting cluster to 'STOPPED' state.

It will report the same job messages as the web UI. Similar to the above, to start a cluster, simply use the --start flag (we use --wait flag instead to see the progress instead of the job logs):

$ s9s cluster --start --cluster-id=8 --wait
Starting Cluster
\ Job 272 RUNNING    [     █    ] ---% Start Cluster

To restart the PostgreSQL service on a database node, we use the "node" component and --restart flag:

$ s9s node \
--restart \
--cluster-id=8 \
--nodes=192.168.0.92 \
--log
Preparing to restart host.
192.168.0.92:5432: Stopping PostgreSQL service.
192.168.0.92:5432: Waiting to stop.
192.168.0.92:5432: Starting PostgreSQL.
192.168.0.92:5432: The postgresql service was started.
192.168.0.92:5432: Waiting to start.

To stop and start a PostgreSQL node, simply apply the same command with --stop or --start flag, as shown below:

$ s9s node --stop --cluster-id=8 --nodes=192.168.0.92
$ s9s node --start --cluster-id=8 --nodes=192.168.0.92

Take note that those actions will not reboot the system.

Scaling Node

To remove a node from a cluster, use the --remove-node flag:

$ s9s cluster \
--remove-node \
--nodes=192.168.0.93 \
--cluster-id=8 \
--log
Removing node 192.168.0.93: checking job parameters.
192.168.0.93:5432: Stopping PostgreSQL service.
192.168.0.93:5432: Waiting to stop.
192.168.0.93:5432: removed PostgreSQL Server
Updating load balancers.

Adding a new node works similarly, but you must ensure the node is accessible via passwordless SSH beforehand. Configure it first and then add the node using the --add-node flag:

$ s9s cluster \
--add-node \
--nodes=192.168.0.93 \
--cluster-id=8 \
--log
addNode: Verifying job parameters.
Found a master candidate: 192.168.0.91:5432, adding 192.168.0.93:5432 as a slave.
Verifying job parameters.
192.168.0.93:5432: Disabling SELinux/Apparmor.
192.168.0.93: Checking firewall.
192.168.0.93: Disabling firewalld.
192.168.0.93: Flushing iptables.
192.168.0.93:5432: Installing new node.
192.168.0.93:5432: Using the master's data directory '/var/lib/pgsql/11/data'.
192.168.0.91: Checking size of '/var/lib/pgsql/11/data'.
192.168.0.91: /var/lib/pgsql/11/data size is 103.00 MiB.
192.168.0.93: Checking free space in '/var/lib/pgsql/11/data'.
192.168.0.93: /var/lib/pgsql/11/data has 34.19 GiB free space.
192.168.0.93:5432: Setting SELinux in permissive mode.
192.168.0.93:5432: Disabling firewall.
192.168.0.93:5432: Tuning OS parameters.
192.168.0.93:5432: Setting vm.swappiness = 1.
192.168.0.93:5432: Installing helper packages.
192.168.0.93: Upgrading nss.
192.168.0.93: Upgrading ca-certificates.
192.168.0.93: Installing net-tools.
192.168.0.93: Installing netcat.
192.168.0.93: Installing nc.
192.168.0.93: Installing socat.
192.168.0.93: Installing perl-Data-Dumper.
192.168.0.93: Installing which.
192.168.0.93: Installing perl-Data-Dumper-Names.
192.168.0.93: Installing psmisc.
192.168.0.93: Installing rsync.
192.168.0.93: Installing libaio.
192.168.0.93: Installing libevent.
192.168.0.93: Installing wget.
192.168.0.93: Installing curl.
192.168.0.93: Installing gnupg2.
192.168.0.93: Installing pigz.
192.168.0.93: Installing bzip2.
192.168.0.93: Installing iproute2.
192.168.0.93: Installing tar.
192.168.0.93: Installing openssl.
192.168.0.93: Upgrading openssl openssl-libs.
192.168.0.93: Finished with helper packages.
192.168.0.93:5432: Using External repositories.
192.168.0.93:5432: Setting up PostgreSQL repositories.
192.168.0.93:5432: Uninstalling old PostgreSQL packages.
192.168.0.93:5432: Installing PostgreSQL 11 packages (centos-7).
192.168.0.93:5432: PostgreSQL installed, init-name: postgresql-11.
192.168.0.93: Updating PostgreSQL port (5432) and directory.
192.168.0.93:5432: Granting remote access to PostgreSQL server.
192.168.0.93:5432: Granting controller (10.0.2.15,192.168.0.19).
192.168.0.93:5432: Updating configuration.
192.168.0.93:5432: Enabling stat_statements plugin.
192.168.0.93:5432: Setting wal options.
192.168.0.93:5432: Performance tuning.
192.168.0.93:5432: Selected workload type: mixed
Detected system memory: 991.18 MiB
Using the following fine-tuning options:
  checkpoint_completion_target: 0.9
  effective_cache_size: 761229kB
  maintenance_work_mem: 63435kB
  max_connections: 100
  shared_buffers: 253743kB
  wal_keep_segments: 32
  work_mem: 5074kB
Writing file '192.168.0.93:/var/lib/pgsql/11/data/postgresql.conf'.
192.168.0.93:5432: Restarting PostgreSQL service
192.168.0.93:5432: Testing connection (attempt #1).
192.168.0.93:5432: Connected ok.
192.168.0.93:5432: Using the master's data directory '/var/lib/pgsql/11/data'.
192.168.0.91:5432(master): Verifying PostgreSQL version.
Setting up replication 192.168.0.91:5432->192.168.0.93:5432
Collecting server variables.
192.168.0.91:5432: Using the pg_hba.conf contents for the slave.
192.168.0.93:5432: Updating slave configuration.
Writing file '192.168.0.93:/var/lib/pgsql/11/data/postgresql.conf'.
192.168.0.93:5432: GRANT new node on members to do pg_basebackup.
192.168.0.91:5432: granting 192.168.0.93:5432.
192.168.0.93:5432: Stopping slave.
192.168.0.93:5432: Cleaning up slave data directory: /var/lib/pgsql/11/data
192.168.0.93:5432: detected version: 11.1
192.168.0.93:5432: Doing initial sync (pg_basebackup) from 192.168.0.91:5432.
192.168.0.93:5432: Synchronizing pg_hba.conf from master.
Writing file '192.168.0.93:/var/lib/pgsql/11/data/postgresql.conf'.
192.168.0.93:5432: Creating '/var/lib/pgsql/11/data/recovery.conf': Setting 192.168.0.91:5432 as master.
192.168.0.93:5432: Successfully created '/var/lib/pgsql/11/data/recovery.conf'.
192.168.0.93:5432: Restarting PostgreSQL
192.168.0.93:5432: Grant cluster members on the new node (for failover).
Grant connect access for new host in cluster.
Adding grant on 192.168.0.91:5432.
Adding grant on 192.168.0.92:5432.
192.168.0.93:5432: Waiting until service starts.
192.168.0.93:5432: Registering node.
192.168.0.93:5432: Verifying configuration.
192.168.0.93:5432: Checking 'listen_addresses'.
192.168.0.93:5432: Checking variables.
192.168.0.93:5432: Detected PostgreSQL 11.1.
192.168.0.93:5432: Registering host with host manager.
192.168.0.93:5432: Added host to cluster.
Replication slave job finished.
192.168.0.93: Installing cronie.
192.168.0.91:5432: [postgres] Pulling '/var/lib/pgsql/11/data/postgresql.conf'.
192.168.0.92:5432: [postgres] Pulling '/var/lib/pgsql/11/data/postgresql.conf'.
192.168.0.93:5432: [postgres] Pulling '/var/lib/pgsql/11/data/postgresql.conf'.

From the job logs, we can see that since the cluster already has a master running (192.168.0.91), the new node will be deployed as a slave to the master. ClusterControl will then perform all necessary actions and prepare the new node as the given role accordingly.

Switchover to a new master

To perform the switchover, pick one of the slave to become the new master with --promote-slave flag:

$ s9s cluster \
--promote-slave \
--nodes=192.168.0.92 \
--cluster-id=8 \
--log
192.168.0.92:5432: Promoting server to master.
192.168.0.92:5432: Current master is 192.168.0.91:5432.

SERVER           HOST_STATUS            STATUS            ROLE RECEIVE/REPLAY
192.168.0.91     CmonHostOnline   NODE_CONNECTED         master 0/9000EF0; 0/9000EF0
192.168.0.92     CmonHostOnline   NODE_CONNECTED         slave  0/9000EF0; 0/9000EF0
192.168.0.93     CmonHostOnline   NODE_CONNECTED         slave  0/9000EF0; 0/9000EF0

Switching over to 192.168.0.92:5432 (previous master is 192.168.0.91:5432)
192.168.0.91:5432: Stopping the current master.
192.168.0.91:5432: Stopping PostgreSQL service.
192.168.0.91:5432: Waiting to stop.
192.168.0.92:5432: Failover, using file.
192.168.0.92:5432: Waiting to become a master.
192.168.0.92:5432: Became master, ok.
Switching slaves to the new master.
192.168.0.93:5432: Stopping PostgreSQL service.
192.168.0.93:5432: Waiting to stop.
192.168.0.92:5432: Granting host (192.168.0.93:5432).
Running /usr/pgsql-11/bin/pg_rewind --target-pgdata=/var/lib/pgsql/11/data --source-server="host=192.168.0.92 port=5432 user=cmon password=***** dbname=postgres"
192.168.0.93: servers diverged at WAL location 0/9000F60 on timeline 1
no rewind required
192.168.0.93:5432: Creating '/var/lib/pgsql/11/data/recovery.conf': Setting 192.168.0.92:5432 as master.
192.168.0.93:5432: Successfully created '/var/lib/pgsql/11/data/recovery.conf'.
192.168.0.93:5432: Starting PostgreSQL.
192.168.0.93:5432: The postgresql service was started.
192.168.0.93:5432: Waiting to start.
192.168.0.93:5432: Restarted with new master.
192.168.0.91:5432: Stopping PostgreSQL service.
192.168.0.91:5432: Waiting to stop.
192.168.0.92:5432: Granting host (192.168.0.91:5432).
Running /usr/pgsql-11/bin/pg_rewind --target-pgdata=/var/lib/pgsql/11/data --source-server="host=192.168.0.92 port=5432 user=cmon password=***** dbname=postgres"
192.168.0.91: servers diverged at WAL location 0/9000F60 on timeline 1
no rewind required
192.168.0.91:5432: Creating '/var/lib/pgsql/11/data/recovery.conf': Setting 192.168.0.92:5432 as master.
192.168.0.91:5432: Successfully created '/var/lib/pgsql/11/data/recovery.conf'.
192.168.0.91:5432: Starting PostgreSQL.
192.168.0.91:5432: The postgresql service was started.
192.168.0.91:5432: Waiting to start.
192.168.0.91:5432: Restarted with new master.
Servers after promote:
SERVER           HOST_STATUS            STATUS            ROLE RECEIVE/REPLAY
192.168.0.91     CmonHostOnline   NODE_CONNECTED         slave  0/9001F90; 0/9001F90
192.168.0.92     CmonHostOnline   NODE_CONNECTED         master 0/9001F90; 0/9001F90
192.168.0.93     CmonHostOnline   NODE_CONNECTED         slave  0/9001F90; 0/9001F90

192.168.0.92:5432: promote finished (this is the new master).
Successfully promoted a new master.

The job messages show that ClusterControl will first discover the current topology and stop all nodes in the cluster. Then, it configures the new master and gets the other nodes to replicate from it. It will also attempt to run pg_rewind to re-synchronise the PGDATA of the downgraded master with a new base backup. At the end of the job, ClusterControl reports the current topology and the state of the promotion.

We can then verify by listing out all nodes for cluster ID 8:

$ s9s node --list --cluster-id=8 --long
STAT VERSION    CID CLUSTER       HOST         PORT COMMENT
coC- 1.7.1.2985   8 PostgreSQL 11 192.168.0.19 9500 Up and running.
poS- 11.1         8 PostgreSQL 11 192.168.0.91 5432 Up and running.
poM- 11.1         8 PostgreSQL 11 192.168.0.92 5432 Up and running.
poS- 11.1         8 PostgreSQL 11 192.168.0.93 5432 Up and running.

The state "poM-" on the leftmost column carries the following meaning:

• p - PostgreSQL node
• o - online
• M - master

Database Management

To list out all databases found on the cluster, use the --list-database flag on component cluster:

$ s9s cluster \
--list-database \
--long \
--cluster-id=8
SIZE      #TBL #ROWS   OWNER  GROUP  CLUSTER                          DATABASE
  7340032    0       0 system admins PostgreSQL Streaming Replication postgres
  7340032    0       0 system admins PostgreSQL Streaming Replication template1
  7340032    0       0 system admins PostgreSQL Streaming Replication template0
382730240   12 1156642 system admins PostgreSQL Streaming Replication sbtest

Take note that if the cluster has a lot of databases, this option might not show some of them. Sampling a huge number of databases would generate high load and so the controller has an upper limit built into it.

If you want to create a new database for the cluster, simply do:

$ s9s cluster \
--create-database \
--cluster-id=8 \
--db-name=my_shopping_db

To create a new database user, together with a database associated with it (using the same database name), use the --create-account with --with-database flag:

$ s9s cluster \
--create-account \
--cluster-id=1 \
--account=mysystem:passwd@10.10.1.100 \
--with-database
Account 'mysystem' created.
192.168.0.91:5432: Allowing connections from 192.168.0.15.
192.168.0.92:5432: Allowing connections from 192.168.0.15.
192.168.0.93:5432: Allowing connections from 192.168.0.15.
Database 'mysystem' created.
Access for 'mysystem' to 'mysystem' granted.

ClusterControl will perform the necessary actions to create the database and user account with proper privileges and allow it on all database nodes.

Backup Management

ClusterControl supports two backup methods for PostgreSQL:

• pgdump - Alias to pg_dumpall, a utility for writing out all PostgreSQL databases of a cluster into one script file.
• pg_basebackup - A utility to create a full, file system-level backup of a PostgreSQL database.

Creating a backup

To create a new backup using pg_dumpall, pick one database node and specify "pgdump" in the --backup-method flag:

$ s9s backup \
--create \
--backup-method=pgdump \
--cluster-id=8 \
--nodes=192.168.0.92 \
--backup-directory=/storage/backups \
    --on-controller

The --on-controller flag indicates that we would want the created backup to be stored under /storage/backups directory on the ClusterControl node. Omit the flag if you want to store it on the database node itself. The same command can be applied to create pg_basebackup backup. Just replace "pgdump" with "pg_basebackup" would do.

To list out the backup, simply use the --list and --cluster-id flags:

$ s9s backup --list --long --cluster-id=8
ID PI CID V I STATE     OWNER          HOSTNAME     CREATED  SIZE    TITLE
 8  -   8 - F COMPLETED admin          192.168.0.92 08:42:47    1204 Untitled Backup Record
 9  -   8 - F COMPLETED admin          192.168.0.92 08:45:52 3865462 Untitled Backup Record

Scheduling a backup

Scheduling a backup is similar to the command we used to create a backup, with additional --recurrence flag:

$ s9s backup \
--create \
--backup-method=pg_basebackup \
--cluster-id=8 \
--nodes=192.168.0.92 \
--backup-directory=/storage/backups \
--on-controller \
--recurrence='30 0 * * *'

The value of recurrence must be enclosed with quote and in crontab format.

Restoring a backup

To restore a backup to a cluster, use the --restore flag and point out the backup ID that you want to use:

$ s9s backup \
--restore \
--cluster-id=8 \
--backup-id=9 \
--log
192.168.0.19: Checking 'socat' availability.
Stop slaves as restoring offline backup to master.
192.168.0.91:5432: Stopping PostgreSQL service.
192.168.0.91:5432: Waiting to stop.
192.168.0.93:5432: Stopping PostgreSQL service.
192.168.0.93:5432: Waiting to stop.
192.168.0.92:5432: Stopping node for restoring a base-backup.
192.168.0.92:5432: Stopping PostgreSQL service.
192.168.0.92:5432: Waiting to stop.
192.168.0.92:5432: Backing up the current datadir.
192.168.0.92: Mount point of '/var/lib/pgsql/11/data': '/'
192.168.0.92: Creating copy of datadir (using 'mv'): /var/lib/pgsql/11/data_bak
192.168.0.92: Checking 'socat' availability.
192.168.0.92: Starting: su - postgres -c 'socat -u tcp-listen:9999,reuseaddr stdout | tar -C/var/lib/pgsql/11/data -xzf-' 2>&1 > /tmp/netcat.pg.log
192.168.0.92: socat/nc is started.
192.168.0.92: Restoring from '192.168.0.19':'/storage/backups/BACKUP-9/base.tar.gz'
192.168.0.92:5432: Starting node after restored a base-backup.
192.168.0.92:5432: Starting PostgreSQL.
192.168.0.92:5432: The postgresql service was started.
192.168.0.92:5432: Waiting to start.
You may now rebuild your slaves.
Finished restoring.
Checking the cluster.
Setting cluster to 'STARTING' state.
192.168.0.91:5432: Starting PostgreSQL.
192.168.0.91:5432: The postgresql service was started.
192.168.0.93:5432: Starting PostgreSQL.
192.168.0.93:5432: The postgresql service was started.
Cluster is successfully started.
Cluster status is STARTED.

Take note that for pg_basebackup's backup, the restore operation requires database downtime. All PostgreSQL nodes will be stopped prior to the restoration and the restoration takes place on the last known master. This master will be brought up first (followed by all slaves) after the restoration completes.

Verifying a backup

To restore and verify the backup, use --verify flag and specify the destination server using the --test-server flag:

$ s9s backup \
--verify \
--cluster-id=8 \
--backup-id=9 \
--test-server=192.168.0.99 \
--log

The test server must not be part of the cluster and must be accessible via passwordless SSH from ClusterControl node. ClusterControl will first install the target server with the same PostgreSQL version, stream and restore the backup on that node. The backup verification looks for the last exit code after restoration. If the backup is restorable, ClusterControl will then stop the test server and remove it from ClusterControl (but ClusterControl won't shut it down). You should see the following once the job completes:

Backup 9 was successfully verified.

Create Cluster from Backup

ClusterControl introduced a new feature in v1.7.1, where one can create a new cluster based on a backup taken by an existing cluster. This can be very useful to test out your database on a different platform or database version. In this example, we would like to deploy a two-node PostgreSQL 9.6 cluster based on backup ID 214:

$ s9s cluster \
--create \
--cluster-type=postgresql \
--nodes="192.168.0.101?master;192.168.0.102?slave" \
--provider-version=9.6 \
--db-admin=postgres \
--db-admin-passwd='s3cr3tP455' \
--os-user=root \
--os-key-file=/root/.ssh/id_rsa \
--cluster-name="PostgreSQL 9.6 - Test"
--backup-id=214 \
--wait

Take note that the the admin user password for the new cluster must be identical with the PostgreSQL admin password as included in the backup. Basically, ClusterControl performs the deployment job based on the following order:

1. Install necessary softwares and dependencies on all PostgreSQL nodes.
2. Start the first node.
3. Stream and restore backup on the first node (with auto-restart flag).
4. Configure and add the rest of the nodes.

You can then verify the cluster list by using the following command:

$ s9s cluster --stat

Configuration Management

To list out the PostgreSQL configuration of a node, use the --list-config flag:

$ s9s node --list-config --cluster-id=8 --nodes=192.168.0.92
GROUP OPTION NAME                  VALUE
-     data_directory               '/var/lib/pgsql/11/data'
-     listen_addresses             '*'
-     port                         5432
-     max_connections              100
-     shared_buffers               253743kB
-     work_mem                     5074kB
-     maintenance_work_mem         63435kB
-     dynamic_shared_memory_type   posix
-     wal_level                    hot_standby
-     full_page_writes             on
-     wal_log_hints                on
-     max_wal_size                 1GB
-     min_wal_size                 80MB
-     checkpoint_completion_target 0.9
-     max_wal_senders              16
-     wal_keep_segments            32
-     hot_standby                  on
-     effective_cache_size         761229kB
-     log_destination              'stderr'
-     logging_collector            on
-     log_directory                'log'
-     log_filename                 'postgresql-%a.log'
-     log_truncate_on_rotation     on
-     log_rotation_age             1d
-     log_rotation_size            0
-     log_line_prefix              '%m [%p] '
-     log_timezone                 'UTC'
-     track_activity_query_size    2048
-     datestyle                    'iso, mdy'
-     timezone                     'UTC'
-     lc_messages                  'en_US.UTF-8'
-     lc_monetary                  'en_US.UTF-8'
-     lc_numeric                   'en_US.UTF-8'
-     lc_time                      'en_US.UTF-8'
-     default_text_search_config   'pg_catalog.english'
-     shared_preload_libraries     'pg_stat_statements'
-     pg_stat_statements.track     all

ClusterControl returns the output of OPTION NAME and VALUE accordingly. The GROUP column is not applicable in PostgreSQL thus you should see '-' value.

To change a configuration option, use --change-config flag and specify the parameter and value using --opt-name and --opt-value respectively:

$ s9s node \
--change-config \
--nodes=192.168.0.92 \
--opt-name=min_wal_size \
--opt-value='100MB'
192.168.0.92:5432: Changed a read-only parameter. Node restart is required for change to take effect.

You should see ClusterControl return the configuration modification status and advise the follow-up procedure to make sure the configuration change takes affect. You may then use the "s9s node --restart" command to restart the particular node.

Final Thoughts

ClusterControl offers great flexibility when it comes to managing and monitoring your PostgreSQL database cluster. You have the choice of web UI, which is fairly simple and straightforward plus command line interface, which empowers you to achieve full database automation via scripting. Happy managing!

ClusterControl 1.7.1 introduces a new feature called Create Cluster from Backup, which allows you to deploy a new MySQL or Postgres-based cluster and restore data on it from a backup. This blog post shows how this new feature works, and how this type of automation can deliver improvements to your infrastructure operations.

Introducing: Create Cluster from Backup

Backup management is the most loved feature by our users, and we just lifted it to the next level. This new functionality sounds simple - ClusterControl 1.7.1 is able to deploy a new cluster from an existing backup. However, there are multiple procedures and checks involved in order to deploy a production-grade cluster directly from a backup. Restoration itself comes with its own challenges, such as:

• Full or partial restore consequences
• Base backup and its incremental backups ordering (for incremental backup)
• Backup decryption (if encrypted)
• Restoration tool options
• Decompression (if compressed)
• Backup streaming from the source to the destination server
• Disk space utilization during and after the restoration
• Restoration progress reporting

Combine the above with the complexity and repetitiveness of database cluster deployment tasks, you can save time and reduce risk in running error-prone procedures. The hardest part from the user's perspective is to pick which backup to restore from. ClusterControl will handle all the heavy lifting behind the scene, and report the end result once finishes.

The steps are basically simple:

1. Setup passwordless SSH from the ClusterControl node to the new servers.
2. Pick one logical backup from the backup list, or create one under Backups -> Create Backup.
3. Click Restore -> Create Cluster from Backup and follow the deployment wizard.

This feature is specifically built for MySQL Galera Cluster and PostgreSQL at the time being. Here is what you would see in the UI after clicking "Restore" on an existing backup:

The bottom option is what we are looking for. Next, is the summary dialog on the chosen backup before the deployment configuration:

Next, the same database cluster deployment wizard for the respective cluster (MySQL Galera Cluster or PostgreSQL) will be shown to configure a new cluster:

Note that you must specify the same database root/admin username and password as the one that you have in the backup. Otherwise, the deployment would fail half-way when starting up the first node. In general, restoration and deployment procedures will happen in the following order:

1. Install necessary softwares and dependencies on all database nodes.
2. Start the first node.
3. Stream and restore backup on the first node (with auto-restart flag).
4. Configure and add the rest of the nodes.

A new database cluster will be listed under ClusterControl cluster dashboard once the job completes.

What can you gain from it?

There are a number of things you could benefit from this feature, as explained in the following sections.

Test your dataset in various conditions

Sometimes, you might be wondering a the new database version would work or perform for your database workload and testing it out is the only way to know. This is where this feature comes in handy. It allows you to perform tests and benchmark on many variables involved that would affect the database stability or performance, for instance, the underlying hardware, software version, vendor and database or application workloads.

For a simple example, there is a big improvement on the DDL execution between MySQL 5.6 and MySQL 5.7. The following DROP operation on a 10 million rows table proves it all:

mysql-5.7> ALTER TABLE sbtest1 DROP COLUMN xx;
Query OK, 0 rows affected (1 min 58.12 sec)

mysql-5.6> ALTER TABLE sbtest1 DROP COLUMN xx;
Query OK, 0 rows affected (2 min 23.74 sec)

Having another cluster to compare with actually allows us to measure the improvement and justify a migration.

Database migration with logical backup

Logical backup like mysqldump and pg_dumpall is the safest way to upgrade, downgrade or migrate your data from one version or vendor to another. All logical backups can be used to perform database migration. The database upgrade steps are basically simple:

1. Create (or schedule) a logical backup - mysqldump for MySQL or pg_dumpall for PostgreSQL
2. Setup passwordless SSH from ClusterControl node to the new servers.
3. Pick one created logical backup from the backup list.
4. Click Restore -> Create Cluster from Backup and follow the deployment wizard.
5. Verify the data restoration on the new cluster.
6. Point your application to the new cluster.

Faster total cluster recovery time

Imagine a catastrophic failure which prevents your cluster to run, like for example a centralized storage failure which affected all the virtual machines that connected to it, you could get a replacement cluster almost immediately (provided the backup files are stored outside of the failed database nodes, stating the obvious). This feature can be automated via s9s client, where you can trigger a job via the command line interface, for example:

$ s9s cluster \
--create \
--cluster-type=postgresql \
--nodes="192.168.0.101?master;192.168.0.102?slave;192.168.0.103?slave" \
--provider-version=11 \
--db-admin=postgres \
--db-admin-passwd='s3cr3tP455' \
--os-user=root \
--os-key-file=/root/.ssh/id_rsa \
--cluster-name="PostgreSQL 9.6 - Test"
--backup-id=214 \
--log

One thing to note when using this feature is to use the same admin username and password as what's stored in the backup. Also, the passwordless SSH to all database nodes must be configured beforehand. Otherwise, if you prefer to configure it interactively, just use the web UI interface.

Scale Out via Asynchronous Replication

For MySQL Galera Cluster, the newly created cluster has the possibility to be scaled out via MySQL asynchronous replication. Let's say we already restored a new cluster in the office based on the latest backup from the production cluster in the data center, and we would like the office cluster to continue replicating from the production cluster, as illustrated in the following diagram:

You may then set up the asynchronous replication link by using the following way:

1. Pick one node in the production and enable binary logging (if disabled). Go to Nodes -> pick a node -> Node Actions -> Enable Binary Logging.

2. Enable binary logging on all nodes for office cluster. This action requires a rolling restart which will be performed automatically if you choose "Yes" under "Auto-restart node" dropdown:

   Otherwise, you can perform this operation without downtime by using Manage -> Upgrade -> Rolling Restart (or manually restart one node at a time).

3. Create a replication user on the production cluster by using Manage -> Schemas and Users -> Users -> Create New User:

4. Then, pick one node to replicate to the master node in production cluster and setup the replication link:

   mysql> CHANGE MASTER master_host = 'prod-mysql1', master_user = 'slave', master_password = 'slavepassw0rd', master_auto_position = 1;
   mysql> START SLAVE;

5. Verify if the replication is running:

   mysql> SHOW SLAVE STATUS\G

   Make sure Slave_IO_Thread and Slave_SQL_thread are reporting 'Yes'. The office's cluster should start to catch up with the master node if it's lagging behind.

That’s it for now folks!

With the 1.7 release of our flagship product ClusterControl, we introduced our new agent-based monitoring infrastructure: SCUMM - which this blog discusses in more detail.

As a core element of our product, ClusterControl provides a complete monitoring system with real time data to know what is happening now, with high resolution metrics for better accuracy, pre-configured dashboards, and a wide range of third-party notification services for alerting.

On-premises and cloud systems can be monitored and managed from one single point.

Intelligent health-checks are implemented for distributed topologies, for instance detection of network partitioning by leveraging the load balancer’s view of the database nodes.

And ... monitoring can be agentless via SSH or agent-based … which is where SCUMM comes in!

ClusterControl’s new SCUMM system is agent-based, with a server pulling metrics from agents that run on the same hosts as the monitored databases and uses Prometheus agents for greater accuracy and customization options while monitoring your database clusters.

But why SCUMM and what is it all about?

Introduction to SCUMM

SCUMM - Severalnines CMON Unified Monitoring and Management - is our new agent-based monitoring infrastructure.

This monitoring infrastructure consists of two main components:

The first component is the Prometheus server which acts as the time series database and stores the collected metrics.

The second component is the exporter. There can be one or more exporters responsible for collecting metrics from a node or a service. The Prometheus server collects these metrics (this is called scraping) from the exporters over HTTP. On top of this, we have created a set of dashboards to visualise the collected metrics.

The main benefits are:

1. Collect metrics with community supported Prometheus exporters
   1. For example data from MySQL Performance Schema or ProxySQL
2. A number of specialized dashboards showing the most important metrics and historical trending for each monitored service
3. High frequency monitoring makes it possible to scrape the targets with a one second interval
4. An architecture that scales with the number of database servers and clusters. A single Prometheus instance can ingest thousands of samples per second.
5. No reliance on SSH connectivity for collecting host and process metrics, which means a more scalable system compared to an agentless monitoring solution
6. The ability to create custom dashboards with custom rules (watch out for our upcoming releases)

The SCUMM Agents/Exporters that are installed on the monitored nodes are called Prometheus Exporters. The exporters collect metrics from the node (e.g CPU, RAM, Disk, and Network) and from services such as MySQL or PostgreSQL servers. The Prometheus server is installed on a server and scrapes (samples) the exporters with a custom interval.

Why Prometheus ?

Prometheus is a very popular time-series databases that has gained a large adoption with an active ecosystem. It offers a rich data model and a query language with a http based poll system. It is easy to install, maintain and configure in HA setup as well.

Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts.

Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic, service-oriented architectures. In a world of microservices, its support for multi-dimensional data collection and querying is a particular strength.

Prometheus is designed for reliability, to be the system you go to during an outage to allow you to quickly diagnose problems. Each Prometheus server is standalone, not depending on network storage or other remote services. You can rely on it when other parts of your infrastructure are broken, and you do not need to set up extensive infrastructure to use it. Thus for high-availability it is possible to simply install a second Prometheus server scraping the same data as the first Prometheus server.

Moreover, Prometheus is a very popular time series database and its adoption has grown very fast. It is possible for another Prometheus server that’s higher up in the organization to scrape the Prometheus servers closer to the database tier. This allows for a scalable monitoring infrastructure where on the database tier the data resolution is higher than further up in an organization.

Exporters

One or more exporters are installed on the monitored server and are responsible for collecting metrics about a specific part of the infrastructure. E.g, there may be one exporter to capture host specific information, an exporter to capture MySQL metrics, and ProxySQL metrics.

We have also created a specific process exporter that monitors the running processes of the server. This exporter is critical to the high availability features in ClusterControl, and allows ClusterControl to quickly react on process failures and process states. Using the process exporter (which is installed by default when Agent Based Monitoring is enabled) reduces the system load on the monitored servers.

Enabling Agent Based Monitoring In ClusterControl

Enabling Agent Based Monitoring is as simple as clicking on the Dashboard, and then click on "Enable Agent Based Monitoring." Select a host where the Prometheus server will be installed. This Prometheus server can then be shared with other clusters.

With its 1.7.1 release, ClusterControl comes with the following dashboards:

• System Overview
• Cluster Overview
• MySQL Server - General
• MySQL Server - Caches
• MySQL InnoDB Metrics
• Galera Cluster Overview
• Galera Server Overview
• PostgreSQL Overview
• ProxySQL Overview
• HAProxy Overview
• MongoDB Cluster Overview
• MongoDB ReplicaSet
• MongoDB Server

To summarise …

Whether one wants to use a monitoring agent or go the agentless route is completely based on organizational policy requirements and custom needs. And although we love the simplicity of not having to install or manage agents on the monitored database hosts, an agent-based approach can provide higher resolution of monitoring data and has certain advantages in terms of security.

ClusterControl’s new SCUMM system uses Prometheus agents for greater accuracy and customization options while monitoring your database clusters.

Why not give it a try and see for yourself!

Install ClusterControl today (it’s free with our Community Edition) or download our new ClusterControl Guide if you’d like to read about our product more first.

ClusterControl consists of multiple components and one of them is the user interface, a web application running on PHP platform. Some customers had concerns since PHP 5.x was deprecated early this year. This blog post shows you how you would run ClusterControl on PHP 7.x.

What's up with PHP 5?

PHP 5 has reached the end of life on January 1st, 2019. Each release branch of PHP is fully supported for two years from its initial stable release. During this period, bugs and security issues that have been reported are fixed and are released in regular point releases. End of life (EOL) simply means the release is no longer supported. Users of this release should upgrade as soon as possible, as they may be exposed to unpatched security vulnerabilities.

While ClusterControl already supports PHP 7 since version 1.6.1 (release date - May 25th, 2018), some OS distribution still offers PHP 5 in its default repository. The following list shows the supported OS alongside its PHP packages in the repository:

• Ubuntu 18.04 (Bionic) - PHP 7.2
• Ubuntu 16.04 (Xenial) - PHP 7.0
• Ubuntu 14.04 (Trusty) - PHP 5.5
• Debian 9 (Stretch) - PHP 7.0
• Debian 8 (Jessie) - PHP 5.6
• Debian 7 (Wheezy) - PHP 5.4
• CentOS/RHEL 6 - PHP 5.3
• CentOS/RHEL 7 - PHP 5.4

Up until today, the active PHP versions are 7.2 and 7.3, as shown in the following Gantt chart (taken from http://php.net/supported-versions.php):

Note that even PHP 7.1 has entered passive support with security fixes only until Dec 1st, 2019. Considering all the options we have here, the following operating systems offers an outdated PHP version:

• Debian 9 (Stretch)
• Debian 8 (Jessie)
• Debian 7 (Wheezy)
• Ubuntu 16.04 (Xenial)
• Ubuntu 14.04 (Trusty)
• CentOS/RHEL 6
• CentOS/RHEL 7

It is probably the right time to upgrade the PHP at least up to version 7.1 to avoid security concerns and get peace of mind when running web applications.

Installing ClusterControl with PHP 7

CentOS/RHEL

For new installation, the best way for now is to use third-party repository that offers PHP 7 packages. For RHEL/CentOS system, you can enable Remi repository which uses the same generic name for PHP packages. First, enable the Remi repository:

For CentOS/RHEL 7:

$ yum install -y epel-release
$ wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
$ rpm -Uvh remi-release-7.rpm
$ sed -i s/enabled=0/enabled=1/g /etc/yum.repos.d/remi-php73.repo

For CentOS/RHEL 6:

$ yum install -y epel-release
$ wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
$ rpm -Uvh remi-release-6.rpm
$ sed -i s/enabled=0/enabled=1/g /etc/yum.repos.d/remi-php73.repo

Then, proceed with the ClusterControl installation as usual:

$ wget severalnines.com/downloads/cmon/install-cc
$ chmod 755 install-cc
$ ./install-cc

ClusterControl will be installed on top of PHP 7.3. There are a number of third-party repositories that offer PHP 7 installation like Webstatic, RPMFusion and so on. However, using the Remi repository is the best option we got considering it requires zero modification to the ClusterControl installer script.

Ubuntu/Debian

The following is applicable for Ubuntu and Debian OS platform. On ClusterControl run, configure "ondrej/php" PPA repository. The following example is what you should have executed on an Ubuntu 16.04 (Xenial) server:

$ apt-get install -y software-properties-common language-pack-en-base
$ LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php

Download the installer script:

$ wget severalnines.com/downloads/cmon/install-cc

For the following operating systems, we have to make a small modification to instruct the script to install the latest PHP 7:

• Debian 8 (Jessie)
• Debian 7 (Wheezy)
• Ubuntu 14.04 (Trusty)

The installer script is configured to use php5-* packages for the above OSes. You can see the line that corresponds to this check starting at line 312, as shown in the following code snippet:

    case "${os_codename}" in
        'bionic' | 'xenial' | 'stretch')
            # prefixes packages simply with 'php'
            install_packages="${install_packages} php-xml"
            install_packages="`echo $install_packages | sed 's/php5/php/g'`"
            ;;
    esac

Thus, if the operating system is not listed there (right below the "case" line), we have to add the OS code name into the list. In this example, we are adding Ubuntu 14.04 (Trusty) into the list so the installer will use the php-* packages instead of php5-* packages (as shown in the "sed" command). Add the OS codename into line 313, similar to the example below:

'bionic' | 'xenial' | 'stretch' | 'trusty')

You can use "lsb_release -a" command to get the OS codename. Finally, proceed with the ClusterControl installation as usual:

$ chmod 755 install-cc
$ ./install-cc

You will end up with PHP 7.2 installed, which is good enough to run the ClusterControl web application.

Upgrading to PHP 7

For those who already installed ClusterControl with PHP 5.x, it's possible to upgrade the PHP version to the latest one. Basically, the same principle applies to what we described in the previous section.

CentOS/RHEL

Install and enable the third-party package repository. In this case, we are using Remi repository:

For CentOS/RHEL 7:

$ yum install -y epel-release
$ wget http://rpms.famillecollet.com/enterprise/remi-release-7.rpm
$ rpm -Uvh remi-release-7.rpm
$ sed -i s/enabled=0/enabled=1/g /etc/yum.repos.d/remi-php73.repo

For CentOS/RHEL 6:

$ yum install -y epel-release
$ wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
$ rpm -Uvh remi-release-6.rpm
$ sed -i s/enabled=0/enabled=1/g /etc/yum.repos.d/remi-php73.repo

Then, perform the upgrade accordingly:

$ yum upgrade -y php-*

Once done, restart Apache web server to load the new PHP version:

$ systemctl restart httpd # CentOS/RHEL 7
$ service httpd restart # CentOS/RHEL 6

Re-login into the ClusterControl UI and you should be good.

Ubuntu/Debian

The following is applicable for Ubuntu and Debian OS platform. First of all, verify if the ClusterControl server is installed with php5-* or php-* package name:

$ dpkg --list | grep -i php5-
ii  php5-cli                        5.5.9+dfsg-1ubuntu4.26                     amd64        command-line interpreter for the php5 scripting language
ii  php5-common                     5.5.9+dfsg-1ubuntu4.26                     amd64        Common files for packages built from the php5 source
ii  php5-curl                       5.5.9+dfsg-1ubuntu4.26                     amd64        CURL module for php5
ii  php5-gd                         5.5.9+dfsg-1ubuntu4.26                     amd64        GD module for php5
ii  php5-json                       1.3.2-2build1                              amd64        JSON module for php5
ii  php5-ldap                       5.5.9+dfsg-1ubuntu4.26                     amd64        LDAP module for php5
ii  php5-mysql                      5.5.9+dfsg-1ubuntu4.26                     amd64        MySQL module for php5
ii  php5-readline                   5.5.9+dfsg-1ubuntu4.26                     amd64        Readline module for php5

If you got something like the above, remove all PHP 5 related packages first:

$ apt-get remove php5-* libapache2-mod-php5

Otherwise, you may skip the above step. On ClusterControl run, configure "ondrej/php" PPA repository. The following example is what you should have executed on an Ubuntu server:

$ apt-get install -y software-properties-common language-pack-en-base
$ LC_ALL=en_US.UTF-8 add-apt-repository ppa:ondrej/php

Then, update the repolist followed by upgrading the PHP packages:

$ apt-get update
$ apt-get install libapache2-mod-php php-cli php-common php-curl php-gd php-json php-ldap php-mysql php-readline php-xml

Restart the Apache web server to load the new changes:

$ systemctl restart apache2 # systemd
$ service apache2 restart # sysvinit or upstart

Re-login ClusterControl UI. It should be running on PHP 7.2 at this point. You may verify with phpinfo() page or using the "php --version" command.

Final Thoughts

It's highly recommended to keep up with the active supported version for all dependencies of your applications. This ensures that you run the overall infrastructure with minimal security risks, and better performance.

In this blog we want to highlight three global eCommerce companies who have leveraged ClusterControl to manage and monitor the databases behind their powerful and innovative applications.

Each eCommerce platform & application has unique needs when it comes to database management. For some applications the need to protect their users data is paramount, for others speed and performance is the key requirement. Because of this uniqueness it is important to choose the right technology from the start as well as develop strategies to ensure that the databases consistently perform at optimal levels.

Below we will highlight three ClusterControl eCommerce users and what challenges they faced in building a highly-available, eCommerce database infrastructure. If you want to learn more, you can read their individual case studies which goes into greater detail.

Instant Gaming

The migration from physical game boxes and discs has been rapidly moving online with platforms like Steam, uPlay, Xbox & PlayStation Network growing in popularity. This trend, being fueled by the rapid increase in internet bandwidth, lets users buy and immediately download the games.

Instant-Gaming.com has positioned themselves as a leader in this space selling games that can be installed on a variety of platforms.

Due to rapid growth it was not uncommon for the company to have surges of activity that resulted in a strain on their infrastructure. The downtime from these performance issues caused an impact to their business and revenue. As they put it: “Downtime is critical for e-commerce as your revenue stream depends on your uptime.”

Built using a custom-made eCommerce framework, Instant Gaming’s front-end applications used a MySQL database deployed in Master - Slave mode, with manual failover. The same database was used for back office applications like order management, billing, inventory and customer data. The applications ran in a Docker environment, while the database itself ran on barebone hardware.

The team began using ClusterControl and its pre-configured MySQL HA deployment features and the automatic failover function. This resulted in improved reliability and stability for their platform.

You can read more about Instant Gaming here.

OxyShop

In the Czech Republic, there aren’t many companies who understand the challenges of eCommerce like OxyShop. They design custom eCommerce solutions for a variety of regional customers using a high-quality, custom-built web platform.

OxyShop also offers their clients managed options as well. In these scenarios they are not only responsible for designing and developing the application but also for providing SLAs of performance of the site.

When a particularly demanding customer approached them about building a new site, one that was predicted to have a large amount of traffic, they knew they needed a better way of handling the database traffic in order to guarantee uptime.

The OxyShop team decided that it was neither cost effective nor would it fit their aggressive project schedule to build a custom solution. ClusterControl, however, allowed them to manage a highly available PostgreSQL setup. It would detect any primary node failures, promote a secondary node with the most recent data and failover to it.

You can read more about OxyShop here.

SportPesa

Online gambling and offshore betting are booming businesses. Based in Kenya, SportPesa is Africa’s largest betting platform.

The system behind SportPesa handles an enormous amount of traffic. From tracking bets to collecting money, making payments to mobile transactions, or simply keeping up with the latest betting statistics, data is at the core of the online gambling business. Distributed, in-memory clusters are employed to handle this massive amount of traffic, but the explosive growth and high traffic during peak times was a cause of concern as it was a source of instability.

The company, while searching for solutions to configuration and performance challenges, decided to give ClusterControl a shot. Impressed by the knowledge of MySQL Cluster, the SportPesa team then decided to download and give ClusterControl a try.

What did they find? With ClusterControl’s automation, advanced monitoring and performance management capabilities they had this to say… “Severalnines has deep database competence, and can be a trusted partner when building high performance, distributed database systems.”

You can read more about SportPesa here.

Conclusion

If you have an eCommerce application that needs highly-available, high-performance databases that can automatically recover when disaster strikes, then why not give ClusterControl a try?

We’re happy to announce that our newly updated whitepaper MySQL Replication for High Availability is now available to download for free!

MySQL Replication enables data from one MySQL database server to be copied automatically to one or more MySQL database servers.

Unfortunately database downtime is often caused by sub-optimal HA setups, manual/prolonged failover times, and manual failover of applications. This technology is common knowledge for DBAs worldwide, but maintaining those high availability setups can sometimes be a challenge.

In this whitepaper, we discuss the latest features in MySQL 5.6, 5.7 & 8.0 as well as show you how to deploy and manage a replication setup. We also show how ClusterControl gives you all the tools you need to ensure your database infrastructure performs at peak proficiency.

Topics included in this whitepaper are …

• What is MySQL Replication?
  • Replication Scheme
    • Asynchronous Replication
    • Semi-Synchronous Replication
  • Global Transaction Identifier (GTID)
    • Replication in MySQL 5.5 and Earlier
    • How GTID Solves the Problem
    • MariaDB GTID vs MySQL GTID
  • Multi-Threaded Slave
  • Crash-Safe Slave
  • Group Commit
• Topology for MySQL Replication
  • Master with Slaves (Single Replication)
  • Master with Relay Slaves (Chain Replication)
  • Master with Active Master (Circular Replication)
  • Master with Backup Master (Multiple Replication)
  • Multiple Masters to Single Slave (Multi-Source Replication)
  • Galera with Replication Slave (Hybrid Replication)
• Deploying a MySQL Replication Setup
  • General and SSH Settings
  • Define the MySQL Servers
  • Define Topology
  • Scaling Out
• Connecting Application to the Replication Setup
  • Application Connector
  • Fabric-Aware Connector
  • Reverse Proxy/Load Balancer
    • MariaDB MaxScale
    • ProxySQL
    • HAProxy (Master-Slave Replication)
• Failover with ClusterControl
  • Automatic Failover of Master
    • Whitelists and Blacklists
  • Manual Failover of Master
  • Failure of a Slave
  • Pre and Post-Failover Scripts
    • When Hooks Can Be Useful?
      • Service Discovery
      • Proxy Reconfiguration
      • Additional Logging
• Operations - Managing Your MySQL Replication Setup
  • Show Replication Status
  • Start/Stop Replication
  • Promote Slave
  • Rebuild Replication Slave
  • Backup
  • Restore
  • Software Upgrade
  • Configuration Changes
  • Schema Changes
  • Topology Changes
• Issues and Troubleshooting
  • Replication Status
  • Replication Lag
  • Data Drifting
  • Errant Transaction
  • Corrupted Slave
  • Recommendations

Download the whitepaper today!

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that lets you automate many of the database tasks you have to perform regularly like deploying new databases, adding and scaling new nodes, running backups and upgrades, and more.

To learn more about ClusterControl click here.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 32,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States. To see who is using Severalnines today visit, https://www.severalnines.com/company.

In this blog we wanted to highlight three telecommunications providers who have used ClusterControl to power their database management.

As technology gets smaller, easier and more powerful, existing Telecommunications providers must continue to innovate to stay relevant while newcomers to the block must come out strong and stable to earn a spot in the crowded marketplace.

When it comes to databases, however, telecommunications challenges are BIG. Large amount of data both pass through and are created by their systems & applications. As the growth of data continues to skyrocket, having systems in place to handle that growth (and ensure performance) is even more important than before.

In the sections below we highlight three ClusterControl Telecom users, what challenges they faced in building highly-available, dependable and secure database environments. Two of these companies leveraged ClusterControl to fix issues and one from the start. If you want to learn more, you can read their individual case studies which go into greater detail.

CAN’L

Running lean-and-mean is a requirement in the Telecommunications industry if you want to show a good profit in spite of being in a commodity industry.

Can’l, an internet service provider located in New Caledonia, struggled with managing their databases. Their single DBA was overworked but the requirements on the database were still increasing. The decision needed to be made… hire more people or find a more efficient way.

In the end the Can’l team began leveraging ClusterControl to allow them to re-build their MySQL NDB Cluster database setup. So how have things been going since they implemented? Just ask CTO Cocconi Alain who said “We will renew as long as possible with Severalnines: great experience with three years without issues in production. It gives us the assurance of security and quality.”

You can read more about Can’l here.

Net-SOL

If you live in Austria and are in need of an advanced VOIP phone system then Net-Sol is the company to call.

Behind the curtains, however, the company was experiencing database performance issues caused by frequent node outages requiring failover; each time resulting in slow downs. They were barely able to keep everything running smoothly and needed a better way to monitor and manage the performance of their MySQL Galera Cluster database clusters.

The Net-Sol team turned to ClusterControl and, remarkably, in two days had their existing infrastructure imported, reconfigured, and a load balancer added (HAProxy). From there it was as simple as setting up monitoring, system alerts, and enabling ClusterControl’s auto-repair and recovery features.

You can read more about Net-Sol here.

Sielte

Innovation in the telecommunications space is no more apparent than when talking about digital identity systems. These systems allow for an individual to take their government sponsored digital identification securely throughout the web.

Sielte, a premier telecommunications provider in Italy, has been developing SielteID in conjunction with SPID, Italy’s digital identity. public system.

Unlike the other two companies listed above, Sielte started off trying to find a way to achieve high-availability and performance. After evaluating several vendors and some free open-source tools then decided to use ClusterControl to build their database infrastructure.

Sielte quickly deployed a MySQL-based database cluster with automatic failover & recovery to ensure their application met uptime SLAs for their demanding clientele.

You can read more about Sielte & SielteID here.

Conclusion

If you have a Telecommunications application that needs highly-available, high-performance databases that can automatically recover when disaster strikes, then why not give ClusterControl a try?

We’re happy to announce that our new whitepaper How to Deploy Open Source Databases is now available to download for free!

Choosing which DB engine to use between all the options we have today is not an easy task. An that is just the beginning. After deciding which engine to use, you need to learn about it and actually deploy it to play with it. We plan to help you on that second step, and show you how to install, configure and secure some of the most popular open source DB engines.

In this whitepaper we are going to explore the top open source databases and how to deploy each technology using proven methodologies that are battle-tested.

Topics included in this whitepaper are …

• An Overview of Popular Open Source Databases
  • Percona
  • MariaDB
  • Oracle MySQL
  • MongoDB
  • PostgreSQL
• How to Deploy Open Source Databases
  • Percona Server for MySQL
  • Oracle MySQL Community Server
    • Group Replication
  • MariaDB
    • MariaDB Cluster Configuration
  • Percona XtraDB Cluster
  • NDB Cluster
  • MongoDB
  • Percona Server for MongoDB
  • PostgreSQL
• How to Deploy Open Source Databases by Using ClusterControl
  • Deploy
  • Scaling
  • Load Balancing
  • Management   

Download the whitepaper today!

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that lets you automate many of the database tasks you have to perform regularly like deploying new databases, adding and scaling new nodes, running backups and upgrades, and more.

To learn more about ClusterControl click here.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skill levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 32,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States. To see who is using Severalnines today visit, https://www.severalnines.com/company.