Streaming replication setups with Postgres are inevitably related to failovers. The sole writer in the setup would be the primary, it generates XLOG records and continuously ships them to one or more standby servers. If the primary fails, one of the standby servers will have to take its role through the process of failover. This process can be tricky and potentially, it may cause data loss. It may happen, for example, if a standby is not up to date while it is promoted. The master may also die before it is able to transfer all binlog events to at least one of its standby’s.

PostgreSQL has been supported by ClusterControl for a while, it provided an easy way to deploy a streaming replication setup, with a primary server and one or more standby servers. ClusterControl 1.4.2 brings major improvements related to how ClustertControl handles failures in a PostreSQL replication setup. In this blog, we’ll have a look at these features.

Deploying entire replication topologies

Starting from ClusterControl 1.4.2, you can deploy an entire PostgreSQL replication setup in the same way as you would deploy MySQL and MongoDB: you can use “Deploy Cluster” menu to deploy a primary and one or more PostgreSQL standby servers. Let’s see what it looks like.

First, you need to define access details, as usual when deploying new clusters using ClusterControl. It requires root or sudo password access to all nodes on which PostgreSQL will be deployed.

Next, we need to define user and password for the PostgreSQL user.

Finally, you want to define topology - which host should be the primary and which hosts should be configured as standby. While you define hosts in the topology, ClusterControl will check if the ssh access works as expected - this lets you catch any connectivity issues early on. That’s pretty much it, it is then a matter of starting the deployment. A job is created in ClusterControl, and you will be able to follow the progress.

Failover handling

Once the replication setup is deployed, ClusterControl is able to monitor the setup and automatically recover failed servers. It can also orchestrate changes in topology.

First, ClusterControl now supports full failover handling - when a master failure is detected, a list of master candidates is created and one of them is chosen to be the new master. It is possible to have a whitelist of servers to promote to primary, as well as a blacklist of servers that cannot be promoted to primary. The remaining slaves are now slaved off the new primary, and the old primary is not restarted.

ClusterControl intentionally keeps the old primary offline because it may happen that some of the data has not been transferred to the standby servers. In such case, the primary is the only host containing this data and you may want to recover the missing data manually. For those who want to have the failed primary automatically rebuilt, there is an option in the cmon configuration file: replication_auto_rebuild_slave. By default, it’s disabled but when the user enables it, the failed primary will be rebuilt as a slave of the new primary. Of course, if there is any missing data which exists only on the failed primary, that data will be lost.

Rebuilding standby servers

Another feature is “Rebuild Replication Slave” job which is available for all slaves (or standby servers) in the replication setup. This is to be used for instance when you want to wipe out the data on the standby, and rebuild it again with a fresh copy of data from the primary. It can be useful if a standby server is not able to connect and replicate from the primary for some reason.

Monitoring Queries

Last of the new PostgreSQL features we’d like to discuss is Query Monitor.

You can now easily check which queries are responsible for the load on your PostgreSQL setup. You’ll see here some basic performance data - how many queries of a given type have been executed? What was their maximum and average execution time? How the total execution time for that query looks like? This should be enough to give you insight into what queries cause the highest load on the system, and on which of them to focus.

So, all in all, ClusterControl 1.4.2 has some good stuff for PostgreSQL users. Do give it a try, and let us know how we’re doing.

The video below details the new ChatOps Integrations available in the latest version of ClusterControl.  Included in the video are…

• Overview of ClusterControl Integrations
• Installing new integrations
• Configuring which clusters to monitor
• Selecting what items to communicate alerts
• Adding multiple chatops integrations
• Using webhooks to set up custom alerts

ClusterControl ChatOps Integrations

Included in the latest release of ClusterControl are direct connections for many popular incident communication services that allow to customize how you are alerted from ClusterControl when something goes wrong with your database environments.

ClusterControl integrates with...

• Slack
• Flowdock
• Hipchat
• Campfire
• PagerDuty
• IRC
• Telegram
• OpsGenie
• VictorOps

Learn more about the new ClusterControl chatops integrations here.

The video below details the features and functions in ClusterControl for PostgreSQL. Included in this demonstration video are...

• Overview of PostgreSQL features
• Deploying a PostgreSQL cluster
• Configuring master and slave nodes
• Monitoring your cluster
• Adding additional nodes to an existing cluster
• Rebuilding slaves

ClusterControl for PostgreSQL

As most of you will know, PostgreSQL is a powerful, open-source relational database system powered by a strong community that is not controlled by any corporation or other private entity. And we’re excited about the new features we recently announced for it with the latest version of ClusterControl (1.4.2) such as the ability to deploy entire replication topologies with our new deployment wizard, advanced failover handling, and new monitoring queries.

ClusterControl provides the basic functionality you need to get PostgreSQL up-and-running using our deployment wizard, monitoring and basic management abilities like automatic failover, backups, and restores.

ClusterControl for PostgreSQL includes...

• Point and Click Replication Deployments - ClusterControl allows you to easily deploy and configure master/slave replication PostgreSQL instances.
• Advanced Performance Monitoring - ClusterControl monitors queries and detects anomalies with built-in alerts.
• Automated Failover Handling - ClusterControl detects master failures and automatically promotes a new master
• Database Automation - ClusterControl lets you manage configurations, schedule, and restore backups.

Learn more about ClusterControl and PostgreSQL here.

With the latest release of ClusterControl we’ve introduced a range of key new features to automate and manage PostgreSQL (alongside MySQL and MongoDB).

PostgreSQL is not only the world’s most advanced open source database (in its own words), but it is also currently the second most popular one after MySQL and before MongoDB (according to DB-Engines).

We’ve been working closely with a partner from the Netherlands, OptimaData, in the past few months to bring users the good news about open source database automation and management.

They happen to have a particular affection for PostgreSQL, so we’ve asked them to contribute to today’s blog post, which is all about singing the praises of PostgreSQL and recommendations on which systems or tools to use to automate and manage it.

We’ll start with the basics though and with a bit of an introduction to PostgreSQL, which though it is highly popular, somehow seems to be less talked about than its most direct “competitors” MongoDB and MySQL.

Introduction to PostgreSQL and recommendations - by Martijn Wallet, OptimaData

PostgreSQL is an advanced SQL database server, which is available on a wide range of platforms. It has an enviable reputation for performance, stability, and a wide range of advanced features, which have been developed over the course of more than 20 years. With that, PostgreSQL is one of the oldest open source projects in existence, completely free to use, and developed by a very diverse, worldwide community. And best of all, it just works!

One of its key benefits is that it is open source, meaning that customers and users have a very permissive license to install, use, and distribute PostgreSQL without paying anyone any fees or royalties. On top of that, PostgreSQL is well-known as a database that stays up for long periods, and requires little or no maintenance in many cases. As a result, PostgreSQL provides a very low total cost of ownership.

Originally developed by the Database Research Group at the University of California, Berkeley, PostgreSQL is now developed and maintained by a huge army of developers and contributors. Many of those contributors have full-time jobs related to PostgreSQL, working as designers, developers, database administrators, and trainers. Some, but not many, of those contributors work for companies that specialize in support for PostgreSQL. No single company owns PostgreSQL, nor are you required (or even encouraged) to register your usage. In other words, it is truly an open source project, in the best sense of the term.In the early days, when PostgreSQL was still a research database, the focus was solely on the cool new features. Over the last 15 years, enormous amounts of code have been rewritten and improved, giving us one of the most stable and largest software servers available for operational use.

PostgreSQL is a general-purpose database management system, which offers users many ways to work. You define the database that you would like to manage with it. You can use a normalized database model, augmented with features such as arrays and record subtypes, or use a fully dynamic schema with the help of JSONB and an extension named hstore. PostgreSQL also allows you to create your own server-side functions in any of a dozen different languages.

PostgreSQL is highly extensible, so you can add your own data types, operators, index types, and functional languages. You can even override different parts of the system using plugins to alter the execution of commands or add a new optimizer. All of these features offer a huge range of implementation options to software architects and many ways to keep out of trouble when building applications and maintaining them over long periods of time.

In my opinion for PostgreSQL the following features are key:

• Excellent SQL standards compliance
• Client-server architecture
• Highly concurrent design where readers and writers don't block each other
• Highly configurable and extensible for many types of applications
• Excellent scalability and performance with extensive tuning features
• Support for many kinds of data models: relational, document (JSON and XML), and key/value

What makes PostgreSQL different from other databases?

The PostgreSQL project focuses on the following objectives according to its website:

• Robust, high-quality software with maintainable, well-commented code
• Low maintenance administration for both embedded and enterprise use
• Standards-compliant SQL, interoperability, and compatibility
• Performance, security, and high availability

About PostgreSQL and MySQL

What surprises many people is that PostgreSQL's feature set is more comparable with Oracle or SQL Server than it is with MySQL. The only connection between MySQL and PostgreSQL is that these two projects are open source; apart from that, the features and philosophies are almost totally different.

One of the key features of Oracle, since Oracle 7, has been snapshot isolation, where readers don't block writers and writers don't block readers. You may be equally surprised to learn that PostgreSQL was the first database to be designed with this feature, and it offers a complete implementation. In PostgreSQL, this feature is called Multiversion Concurrency Control (MVCC).

Who is using PostgreSQL?

Prominent users include Adyen, Apple, BASF, Genentech, Heroku, IMDB. com, Skype, McAfee, NTT, The UK Met Office, and The U. S. National Weather Service. 5 years ago, PostgreSQL received well in excess of 1 million downloads per year, according to data submitted to the European Commission, which concluded that, "PostgreSQL is considered by many database users to be a credible alternative."

Main advantages of PostgreSQL

PostgreSQL offers many advantages for users and businesses over other database systems.

Immunity to over-deployment

Over-deployment is what some proprietary database vendors regard as their #1 licence compliance problem. With PostgreSQL, no-one can sue our customers for breaking licensing agreements, as there is no associated licensing cost for the software.

This has several additional advantages:

• More profitable business models with wide-scale deployment
• No possibility of being audited for license compliance at any stage
• Flexibility to do concept research and trial deployments without needing to include additional licensing costs

Better support than the proprietary vendors

There is a vibrant community of PostgreSQL professionals and enthusiasts that users can interact with and get support from. OptimaData is also a good option :-)

Legendary reliability and stability

Unlike many proprietary databases, it is extremely common for companies to report that PostgreSQL has never, ever crashed for them in several years of high activity operation. Not even once. It just works.

Extensible

The source code is available to all at no charge. If users have a need to customise or extend PostgreSQL in any way then they are able to do so with a minimum of effort, and with no attached costs. This is complemented by the community of PostgreSQL professionals and enthusiasts around the globe that also actively extend PostgreSQL on a daily basis.

Cross platform

PostgreSQL is available for almost every brand of Unix (34 platforms with the latest stable release), and Windows compatibility is available via the Cygwin framework. Native Windows compatibility is also available with version 8.0 and above.

Designed for high volume environments

It is possible to make use of a multiple row data storage strategy called MVCC to make PostgreSQL extremely responsive in high volume environments. The leading proprietary database vendors use this technology as well, for the same reasons.

Why Severalnines and ClusterControl for PostgreSQL?

I’ve been following Severalnines a couple of years and along that way I discovered the maturity of their product, ClusterControl. All kinds of companies find their way to OptimaData, for example companies which are operating several database types that all need attending to.

For such companies ClusterControl is a very interesting solution to consider as it automates a lot of important tasks. But also at larger companies like VidaXL with a few dozen database nodes the automatic recovery features for clusters are a great enrichment for their database operations.

The newest additions in release 1.4.2 of ClusterControl for PostgreSQL are a great step forward. The automated failover handling is from my point of view a key feature and for me an important reason to advise ClusterControl to our customers. But also the deployment of an entire replication topology through the new wizard is good consideration. I have been using it a few times already and it’s a real advantage compared to other deployment methods. Also the monitoring has improved compared to earlier versions of ClusterControl.

Of course, there is always something more to wish for and I can’t wait to see what’s in the next release of ClusterControl.

For automation and management therefore, we recommend ClusterControl for PostgreSQL to customers and users for the following features:

• ClusterControl for PostgreSQL
  • Manage All Your Servers From a Single Console
  • Automate Failover Handling
  • Stream Replication
  • Monitor performance
• Streaming Replication
  • Easily deploy and setup master/slave replication setups
• Performance Monitoring
  • Monitor queries and detect anomalies, built-in and custom advisors
• Automated Day to Day Tasks
  • Manage configurations, schedule backups, encrypt client/server traffic
• Configuration Management
  • Automatically manage and provision configurations for your servers
• Automated Failover Handling
  • Detect master failures and automatically promote new master

And in addition to ClusterControl, we also like to recommend the following tools:

• pgAdmin, a database management tool for database administrators;
• pgAgent, job scheduling agent for the planning of complex tasks;
• Repmgr, replication and failover manager for PostgreSQL server clusters;
• pgBouncer, connection pooler for setting up high redundant environments;
• Barman, Backup and Recovery Manager for the planning of backup and disaster recovery;
• pgAudit, detailed session and/or object audit logging via the standard PostgreSQL logging facility;
• PostGIS, provides spatial objects for the PostgreSQL database, allowing storage and query of information about location and mapping;
• PgBadger, a fast PostgreSQL log analysis report;
• PgWatch2, flexible self-contained PostgreSQL metrics monitoring/dashboarding solution.

Additional useful links

GUI database design and administration tools

There are many high-quality GUI Tools available for PostgreSQL from both open source developers and commercial providers. A list is available on the wiki that functions as a community guide to PostgreSQL GUI Tools.

Technical Features

Please see the Feature Matrix for a summary of PostgreSQL's features

One last tidbit of information I’d like to share: when PostgreSQL was first developed, it was named Postgres, and therefore many aspects of the project still refer to the word "postgres"; for example, the default database is named postgres, and the software is frequently installed using the postgres user ID. As a result, people shorten the name PostgreSQL to simply Postgres, and in many cases use the two names interchangeably. And for those you hesitate:

PostgreSQL is pronounced as "post-grez-q-l". Postgres is pronounced as "post-grez."

We hope you enjoyed the read! If you have any questions or comments, do share them in the comments box below. We’ve referred to postgresql.org (thank you) for some of the content in this blog.

Severalnines is excited to announce it’s newest customer Black Hills Corp, an energy provider operating in the United States.  

In the case study, you can learn how Black Hills utilized ClusterControl to migrate their open source databases to Galera Cluster to operate alongside their Oracle and Microsoft SQL Server database servers.

Requiring enterprise-grade management and monitoring, Black Hills selected Severalnines due to our expertise with the technology, top-level support, and feature-rich product to help with the ongoing management.

Read the case study to learn more.

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that let’s you automate many of the database tasks you have to perform regularly, like deploying new databases, detecting anomalies, recovering nodes from failures, adding and scaling new nodes, running backups and upgrades, and more.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

Open source database technology has matured to the point where it is now being used widely in financial, eCommerce, and payment processing applications.  

Furthermore, FinTech applications are held to very high standards when it comes to security, governance, compliance and data integrity.

With ClusterControl you benefit from the innovation, performance and cost savings that come with open source technology, while maintaining the enterprise-grade quality and dependability you expect from a comprehensive database management system.

ClusterControl offers a wide array of features to ensure your data management needs are met, including:

• Deployment Automation - ClusterControl allows you to deploy the top 3 open source databases in the world (MySQL/MariaDB, MongoDB and PostgreSQL), with setups ranging from single-instance to master-slave replication, and all the way to shared-nothing clusters running on commodity hardware.
• Advanced Security - ClusterControl supports secure encrypted connections between nodes using SSL protocol. Both the connections between database clients and servers as well as replication traffic particular to the database itself are supported through standard SSL protocol
• User Management - ClusterControl provides you with advanced user management so you can know and report on who in your organisation uses your databases and how.
• Automated Failover and Recovery - Advanced automated failover and verified backup technology in ClusterControl ensure your mission critical applications achieve high availability with zero downtime.
Related resources
 ClusterControl for Financial Technology
 Press Release: iyzoco Payment Processing
 Press Release: Veritrans eCommerce Payment Processing
 Whitepaper: Management & Automation of Open Source Databases
• Operational Reporting - When you need to show you are meeting your SLAs comprehensive  operational reports help you keep track of the historical data of your database.
• Monitoring & Alerts -  Advanced monitoring and reporting let you easily keep a close eye on the performance of your environment, by providing complex metrics to analyze challenges and predict future needs.
• Multi-Data & Cross-Regional Support - ClusterControl supports environments across multiple datacenters and regions, keeping even the most complex environments highly available.
• Load Balancing Technology - Achieve high availability using the most advanced load balancing technologies supported by ClusterControl.

ClusterControl also offers you everything you need to ensure you are meeting your industry’s compliance standards to secure your database operations. Segregation of duties, comprehensive reporting and automated functionality afford you the ability to rest assured your deployments are secure, internally accessible and easily managed from a single interface.

Learn more about ClusterControl for Financial Technology and read what our customers are saying about their experience using ClusterControl.

We were very excited to release a new wave of ChatOps and Notification System integrations in the last version of ClusterControl.  But these systems are not the only ones that you can integrate with ClusterControl.

This video details the different type of integrations that are possible when you use ClusterControl to deploy, manage, monitor and scale your open source databases.

ClusterControl Integrations

ClusterControl offers direct connections for many popular incident communication services that allow to customize how you are alerted from ClusterControl when something goes wrong with your database environments. Until recently it was only possible to propagate (critical) alerts and warnings to other systems, using our Notification Services framework.

In ClusterControl 1.4.2 we have overhauled this functionality and replaced it with ClusterControl Integrations. ClusterControl Integrations do not require any modifying of external files. In effect the integration with applications and services has now become so easy, that on average you can add a new integration in less than one minute!

ClusterControl can integrate with many different types of services like...

• Notification Services - PagerDuty, VictorOps and OpsGenie
• ChatOps - Slack, Telegram, Slack, Flowdock, Hipchat and Campfire
• Orchestration - Chef, Puppet, Ansible
• Monitoring - Nagios, Zabbix

Learn more about how ClusterControl integrates with your favorite tools you use here.

We’re happy to announce that our new whitepaper An Executive’s Guide to Database Management ROI is now available to download for free!

This guide discusses the options available to IT leaders when bringing in open source databases into their environments as well as general information on the open source database market. Also included in this whitepaper is an analysis of the costs of performing and omitting essential tasks typically associated with managing open source databases.

Topics included in this whitepaper are…

• The value of community innovation
• Understanding the open source database world
• Understanding the needs of open source databases
• The downsides of open source
• Understanding the costs of free
• Understanding the costs of downtime
• The cost of new knowledge
• Management open source databases - a comparison
• Database total cost of ownership (TCO)
• Where ClusterControl fits
Related resources
 ClusterControl for IT Leaders
 Whitepaper: Management and Automation of Open Source Databases
 Whitepaper: A Guide to Efficient Database Infrastructure Operations
• How many tools to do what ClusterControl does
• The value of ClusterControl
• Introduction to Severalnines & ClusterControl

The whitepaper also discusses how ClusterControl fits into the open source equation and how, in many cases, is more cost-effective than cobbling together multiple point solutions. Throughout the document are quotes from actual Severalnines clients who have implemented open source database technology with ClusterControl.

In the end, the key to maintaining high performance and achieving high availability must be both cost-effective and technical capable of delivering the results need for the success of the application.

If your organization has or is exploring incorporating open source database technology, this whitepaper will help you better understand the options.

Severalnines is excited to announce its newest customer Root Level Technology, a dedicated and co-location hosting provider in the United States.

Root Level Tech is a fast growing company with multiple data centers across the country.  The company builds custom solutions with geographic redundancy, high availability and protection from Distributed Denial of Service (DDoS) attacks.

In this case study, you’ll learn how, after experiencing issues with one its customers, Root Level Tech contacted Severalnines to partner in developing an open source database management solution for MySQL that would provide them with deployments they could count on coupled with monitoring and management features to prevent downtime, ensure high-availability, and help protect their customers data.

Read the case study to learn more.

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that let’s you automate many of the database tasks you have to perform regularly, like deploying new databases, detecting anomalies, recovering nodes from failures, adding and scaling new nodes, running backups and upgrades, and more.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

Severalnines is excited to announce it’s newest customer Mediacloud, a Spain-based, cloud digital infrastructure provider.

Mediacloud is a sub-division of Mediapro, one of the largest multimedia communications groups in Europe, and manages all of its data center operations. As part of these duties it manages cloud operations across six worldwide data centers.

In the case study, you can learn how Mediacloud overcame database consistency issues to achieve a multi-datacenter, highly-available solution using Galera Cluster.  This allowed MediaCloud to provide more than 150 of its clients several nines uptime :-) for their mission-critical applications.

Read the case study to learn more.

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that let’s you automate many of the database tasks you have to perform regularly, like deploying new databases, detecting anomalies, recovering nodes from failures, adding and scaling new nodes, running backups and upgrades, and more.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

Galera Cluster by Codership is a synchronous multi-master replication technology which can be utilized to build highly available MySQL or MariaDB clusters.

It has been downloaded over one million times since last year, establishing itself as one of the most popular high availability and scalability technologies for MySQL, MariaDB and Percona Server with database users worldwide.

And while Galera Cluster is easy enough to deploy, it is complex to operate. To properly automate and manage it does require a sound understanding of how it works and how it behaves in production. For instance, once it’s deployed, how does it behave under a real-life workload, scale, and during long term operations?

This is where monitoring performance and optimizing it, understanding anomalies, recovering from failures, managing schema and configuration changes and pushing them in production, version upgrades and performing backups come into play.

There are a number of things you’d want to have thought through and be in control of before going in production with Galera Cluster for MySQL or MariaDB:

• Hardware and network requirements
• OS tuning
• Sane configuration settings for the database
• Production-grade deployment
• Security
• Monitoring and alerting
• Query performance
• Anomaly detection and troubleshooting
• Recovering from failures
• Schema changes
• Backup strategies and disaster recovery
• Disaster recovery
• Reporting and analytics
• Capacity planning

And the list goes on ...

We saw great potential in Galera Cluster early on, and started building a deployment and management product for it even before the first 1.0 version was released. We are happy to see that the technology has delivered on its promises - high availability of MySQL with good write scalability. Over the years, we have been able to build out comprehensive management procedures in ClusterControl and battle-test these across thousands of installations.

Not everyone has the knowledge, skills, time or resources to manage a high availability database. It is hard enough to find a production DBA, or a DevOps person with strong database knowledge. So imagine that most of the relevant steps in that process could be automated and managed from one central system?

This is where ClusterControl comes in.

ClusterControl is our all-inclusive database management system that lets you easily deploy, monitor, manage and scale highly available open source databases on-premise or in the cloud.

So Why Use ClusterControl for Galera Cluster?

Deploying a production ready Galera Cluster has become a matter of a few clicks for ClusterControl users worldwide. And with tens of thousands of deployments to date, it’s safe to say that ClusterControl is truly ‘Galera battle-tested’. We’ve included years of industry best practices into the product to help companies automate and manage their database operations as smoothly as possible.

Some of the key benefits of using ClusterControl with Galera Cluster include:

• Maximum efficiency: automated failure detection, failover, and automatic recovery of individual nodes or even entire clusters to achieve
• Pro-active intelligence: gain access to advanced monitoring features that give you insights into your database performance and alert you to any problems right away
• Advanced security: ClusterControl provides an array of advanced security features that you can depend on to keep your data safe

One of our most trusted users put it this way:

Customers include Cisco, British Telecom, Orange, Ping Identity, Cisco, Liberty Global, AVG and many others.

The following are some of the key features to be found in ClusterControl for Galera Cluster:

• Deploy Database Clusters
• Configuration Management
• Full stack monitoring (DB/LB/Host)
• Query Monitoring
• Anomaly detection
• Failure detection and automatic recovery/repair
• Add Node, Load Balancer (HAProxy, ProxySQL, MaxScale) or asynchronous replication slave
• Backup Management
• Encryption of data in transit
• Online rolling upgrades
• Developer Studio with Advisors

For a general introduction to ClusterControl, view the following video:

And for a demonstration of the ClusterControl features for Galera Cluster, view the following demo video:

To summarise, working seamlessly with your Galera setup, ClusterControl provides an integrated monitoring and troubleshooting approach, speeding up problem resolutions. A single interface saves you time by not having to cobble together configuration management tools, monitoring tools, scripts, etc. to operate your databases. And you can maximize efficiency and reduce database downtime with battle-tested automated recovery features.

Finally, ClusterControl fully supports all three Galera Cluster flavours, so you can easily deploy different clusters and compare them yourself with your own workload, on your own hardware. Do give it a try.

While many of our customers utilize ClusterControl on-premise to automate and manage their open source databases, several are deploying ClusterControl alongside their applications in the cloud. Utilizing the cloud allows your business and applications to benefit from the cost-savings and flexibility that come with cloud computing. In addition you don’t have to worry about purchasing, maintaining and upgrading equipment.

Along the same lines, ClusterControl offers a suite of database automation and management functions to give you full control of your database infrastructure. With it you can deploy, manage, monitor and scale your databases, securely and with ease through our point-and-click interface.

As the load on your application increases… Your cloud environment can be expanded to provide more computing power to handle that load. In much the same way ClusterControl utilizes state-of-the-art database, caching, and load balancing technologies that enable you to scale-out the load on your databases and spread that load evenly across nodes.

These performance benefits are just some of the many reasons to leverage ClusterControl to manage your open source database instances in the cloud. From advanced monitoring to backups and automatic failover, ClusterControl is your true end-to-end database management solution.

Below you will find some of our top resources to help you get your databases up-and-running in the cloud…

AWS Marketplace

ClusterControl on the AWS Marketplace

Want to install ClusterControl directly onto your AWS EC2 instance? Check us out on the Amazon Marketplace.
(New version coming soon!)

Install Today

Top Blogs

Migrating MySQL database from Amazon RDS to DigitalOcean

This blog post describes the migration process from EC2 instance to a DigitalOcean droplet

Read More

MySQL in the Cloud - Online Migration from Amazon RDS to EC2 Instance (PART ONE)

RDS for MySQL is easy to get started. It's a convenient way to deploy and use MySQL, without having to worry about any operational overhead. The tradeoff though is reduced control.

Read More

MySQL in the Cloud - Online Migration from Amazon RDS to Your Own Server (PART TWO)

It's challenging to move data out of RDS for MySQL. We will show you how to do the actual migration of data to your own server, and redirect your applications to the new database without downtime.

Read More

MySQL in the Cloud - Pros and Cons of Amazon RDS

Moving your data into a public cloud service is a big decision. All the major cloud vendors offer cloud database services, with Amazon RDS for MySQL being probably the most popular. In this blog, we’ll have a close look at what it is, how it works, and compare its pros and cons.

Read More

About Cloud Lock-in and Open Source Databases

Severalnines CEO Vinay Joosery discusses key considerations to take when choosing cloud providers to host and manage mission critical data; and thus avoid cloud lock-in.

Read More

Infrastructure Automation - Deploying ClusterControl and MySQL-based systems on AWS using Ansible

This blog post has the latest updates to our ClusterControl Ansible Role. It now supports automatic deployment of MySQL-based systems (MySQL Replication, Galera Cluster, NDB Cluster).

Read More

Leveraging AWS tools to speed up management of Galera Cluster on Amazon Cloud

We previously covered basic tuning and configuration best practices for MyQL Galera Cluster on AWS. In this blog post, we’ll go over some AWS features/tools that you may find useful when managing Galera on Amazon Cloud. This won’t be a detailed how-to guide as each tool described below would warrant its own blog post. But this should be a good overview of how you can use the AWS tools at your disposal.

Read More

5 Performance tips for running Galera Cluster for MySQL or MariaDB on AWS Cloud

Amazon Web Services is one of the most popular cloud environments. Galera Cluster is one of the most popular MySQL clustering solutions. This is exactly why you’ll see many Galera clusters running on EC2 instances. In this blog post, we’ll go over five performance tips that you need to take under consideration while deploying and running Galera Cluster on EC2.

Read More

How to change AWS instance sizes for your Galera Cluster and optimize performance

Running your database cluster on AWS is a great way to adapt to changing workloads by adding/removing instances, or by scaling up/down each instance. At Severalnines, we talk much more about scale-out than scale up, but there are cases where you might want to scale up an instance instead of scaling out.

Read More

ClusterControl is infrastructure-agnostic - it can be used in your own datacenter on physical hosts, as well as in virtualized cloud environments. All you need is ssh access from the ClusterControl host to the database nodes, and you can then deploy standalone/replicated/clustered MySQL/MariaDB, MongoDB (replica sets or sharded clusters) or PostgreSQL (streaming replication). In this blog post, we will walk you through the steps to deploy ClusterControl on EC2.

Setting up instances in EC2

The hardware requirements for ClusterControl are described here. Those are meant to create a performant platform for the ClusterControl server. Having said that, we will use a small instance for our testing purposes (t2.micro) - it should be enough for us.

First, we need to pick an AMI. ClusterControl supports:

• Redhat/CentOS/Oracle Linux 6 and later
• Ubuntu 12.04/14.04/16.04 LTS
• Debian 7.0 and later

We are going to use Ubuntu 16.04.

Next step - instance type. We will pick t2.micro for now, although you will want to use larger instances for production setups. For other cloud providers, pick instances with at least 1 GB of memory.

We are going to deploy four instances at once, one for ClusterControl and three for Percona XtraDB Cluster. You need to decide where those instances should be deployed (VPC or not, which subnet etc). For our testing purposes, we are going to use a VPC and a single subnet. Of course, deploying nodes across the subnets (Availability Zones) makes your setup more likely to survive if one of the AZ’s would become unavailable.

For storage we’ll use 100GB of general purpose SSD volume (GP2). This should be enough to perform some tests with a reasonable volume of data.

Next - security groups. SSH access is a requirement. Other than that, you need to open ports required by the database you plan to deploy. You can find more information on which ports are required in our support portal.

Finally, you need to either pick one of the existing key pairs or you can create a new one. After this step your instances will be launched.

Once the instances are up and running, it’s time to install ClusterControl. For that, log into one of the instances and download the ClusterControl installation script, install-cc:

ubuntu@ip-172-30-4-20:~$ wget http://www.severalnines.com/downloads/cmon/install-cc
--2017-09-06 11:13:10--  http://www.severalnines.com/downloads/cmon/install-cc
Resolving www.severalnines.com (www.severalnines.com)... 107.154.146.155
Connecting to www.severalnines.com (www.severalnines.com)|107.154.146.155|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.severalnines.com/downloads/cmon/install-cc [following]
--2017-09-06 11:13:10--  https://www.severalnines.com/downloads/cmon/install-cc
Connecting to www.severalnines.com (www.severalnines.com)|107.154.146.155|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://severalnines.com/downloads/cmon/install-cc [following]
--2017-09-06 11:13:11--  https://severalnines.com/downloads/cmon/install-cc
Resolving severalnines.com (severalnines.com)... 107.154.238.155, 107.154.148.155
Connecting to severalnines.com (severalnines.com)|107.154.238.155|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 56913 (56K) [text/plain]
Saving to: ‘install-cc’

install-cc                                           100%[=====================================================================================================================>]  55.58K   289KB/s    in 0.2s

2017-09-06 11:13:12 (289 KB/s) - ‘install-cc’ saved [56913/56913]

Then, make sure it can be executed before running it:

ubuntu@ip-172-30-4-20:~$ chmod +x install-cc
ubuntu@ip-172-30-4-20:~$ sudo ./install-cc

At the beginning, you’ll get some information about the requirements on supported Linux distributions:

!!
Only RHEL/Centos 6.x|7.x, Debian 7.x|8.x, Ubuntu 12.04.x|14.04.x|16.04.x LTS versions are supported
Minimum system requirements: 2GB+ RAM, 2+ CPU cores
Server Memory: 990M total, 622M free
MySQL innodb_buffer_pool_size set to 512M

Severalnines would like your help improving our installation process.
Information such as OS, memory and install success helps us improve how we onboard our users.
None of the collected information identifies you personally.
!!
=> Would you like to help us by sending diagnostics data for the installation? (Y/n):

This script will add Severalnines repository server for deb and rpm packages and
install the ClusterControl Web Applicaiton and Controller.
An Apache and MySQL server will also be installed. An existing MySQL Server on this host can be used.

At some point you will have to answer some questions about hostnames, ports and passwords.

=> The Controller hostname will be set to 172.30.4.20. Do you want to change it? (y/N):
=> Creating temporary staging dir s9s_tmp

=> Setting up the ClusterControl Web Application ...
=> Using web document root /var/www/html
=> No running MySQL server detected
=> Installing the default distro MySQL Server ...
=> Assuming default MySQL port is 3306. Do you want to change it? (y/N):

=> Enter the MySQL root password:
=> Enter the MySQL root password again:
=> Importing the Web Application DB schema and creating the cmon user.

=> Importing /var/www/html/clustercontrol/sql/dc-schema.sql
mysql: [Warning] Using a password on the command line interface can be insecure.
=> Set a password for ClusterControl's MySQL user (cmon) [cmon]
=> Enter a CMON user password:
=> Enter the CMON user password again: => Creating the MySQL cmon user ...
mysql: [Warning] Using a password on the command line interface can be insecure.
=> Creating UI configuration ...

Finally, you’ll get the confirmation that ClusterControl has been installed. Install script will also attempt to detect your public IP and print out a link that can be used in your browser to access ClusterControl.

=> ClusterControl installation completed!
Open your web browser to http://172.30.4.20/clustercontrol and
enter an email address and new password for the default Admin User.

Determining network interfaces. This may take a couple of minutes. Do NOT press any key.
Public/external IP => http://34.230.71.40/clustercontrol
Installation successful. If you want to uninstall ClusterControl then run install-cc --uninstall.

Once the installation is done, there’s still one thing to take care of - SSH access from ClusterControl to the remaining hosts. Unless you already have SSH access between nodes (and you can use ssh-copy-id), this will be a manual process. First and the foremost, we need to generate a new ssh key:

root@ip-172-30-4-20:~# ssh-keygen -C 'galera_cluster' -f id_rsa_galera -t rsa -b 4096
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_galera.
Your public key has been saved in id_rsa_galera.pub.
The key fingerprint is:
SHA256:2tWOGXbrtc0Qh45NhNPzUSVdDzE9ANV1TJ0QBE5QrQY galera_cluster
The key's randomart image is:
+---[RSA 4096]----+
|         .o=**X*&|
|         Eo  + BO|
|          ..+ +.o|
|           + o +.|
|        S = o + o|
|       o o * * o |
|      . . o + =  |
|           . . = |
|            . . o|
+----[SHA256]-----+
root@ip-172-30-4-20:~#

We can verify it has been created correctly. You will also want to copy the contents of the public key - we will use it to create its copies on remaining nodes.

root@ip-172-30-4-20:~# cat id_rsa_galera.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDYKil17MzTrNc70GIQlVoK6xLop9acfT3W6kBUGO4ybsvIA5Fss+WvT/DLsYMtukq2Eih93eO4amLRYQIeyWSjJ/bBwIF/LXL4v04GF8+wbDgCyiV/t9dSuXna9qFeawkUVcPjnmWnZqUoaP5QeovXTluxl54xEwbFm1oLT+fgWbaim5w9vVUK4+hAHaZ7wVvTPVsIt1B3nJgWJF0Sz+TJN87vSUg7xdshgzhapUotXlguFGVzmNKWLnEFDCK7RT41oh4y4rkKP7YLc+wFfRHYTnKyMIcf0/0VMyL+2AdwQp8RThbBommf2HGimF1hSyA9/fc+tLi7FVTg1bKKeXj4hwexeFAJZwoy3HyD3wQ/NwadpDVk5Pg7YYzdN2aCZfvo27qp3gdQQ2H+LF6LvDyQEkgRpFN+pHoWQvPjJJasjfIcfdaC9WmDiL4s5fXyCTQz/x0NaTXVkLBS9ibfOUw8AGdd36FvdqnNOFOlMLKLa359JhdpqXnH7ksiThcotQuFmV5Dc8M66vTDz9rvVZhNC0nME478RNBP0Bgj1BM26XdQlzozeaRmHGoZXcSQVJTXBC93+QN4+bRmWmxhhj5G5M7bFiQyal1VtugoUt8ZV4NiiG+KDd6yj5um8+CffD/BASGrv3vffH+AK7xtjchIv5su40+unecfSOtO98TiQw== galera_cluster

Now, on every remaining node, you need to add this public key into the authorized_keys file. For ubuntu, you may want to clean its contents first if you want to use a root login. By default, only ubuntu user can be used to connect through SSH. Such setup (regular user and sudo) is also possible to use with ClusterControl but here we’ll go for a root user.

root@ip-172-30-4-198:~# vim ~/.ssh/authorized_keys

Once authorized_keys files on all nodes contain our public key, we will copy our public key to .ssh directory and make necessary changes in access rights:

root@ip-172-30-4-20:~# cp id_rsa_galera /root/.ssh/
root@ip-172-30-4-20:~# chmod 600 /root/.ssh/id_rsa_galera

Now we can test if SSH access works as expected:

root@ip-172-30-4-20:~# ssh -i /root/.ssh/id_rsa_galera 172.30.4.46
Welcome to Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-1022-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.


root@ip-172-30-4-46:~# logout
Connection to 172.30.4.46 closed.

All’s good. It’s time to configure ClusterControl.

Fill in some registration details.

Once you have logged in, a wizard will appear with an option to either deploy new cluster or import an existing one.

We want to deploy Percona XtraDB Cluster so we’ll go for “Deploy Database Cluster” and pick the “MySQL Galera” tab. Here we have to fill in access details required for SSH connectivity. We’ll set SSH User to root and we will fill in the path to our SSH key.

Next, we’ll define a vendor, version, password and IP addresses for our database hosts. Please keep in mind that ClusterControl will check SSH connectivity to the target database hosts. If everything works ok, you’ll see green ticks. If you see that SSH authentication failed, then you will need to investigate as the ClusterControl server is not able to access your database hosts.

Then, click on Deploy to start the deployment process.

You can track the deployment progress in the activity monitor.

Remember that deployment is only the first step. Operating a database requires you to monitor performance of your hosts, database instances, queries, and manage backups, fix failures and other anomalies, manage proxies, upgrades, etc. ClusterControl can manage all these aspects for you, so do give it a try and let us know how you get on.

Severalnines is excited to announce it’s newest customer VentraIP, an Australian-based web hosting, domain names, and SSL Certificate provider.

VentraIP Australia is the largest privately owned web host and domain name registrar in Australia, backed by a team of industry veterans and local technical professionals who ensure their 150,000 customers always get the best customer service and technical support 24 hours a day, 7 days a week.

In the case study, you can learn how VentraIP went from using a ill-performing MySQL standalone instance to power their front end and replaced it with Galera Cluster using ClusterControl to deliver a high-performance and redundant system that met their customers needs and the needs of the future.  

Read the case study to learn more.

About ClusterControl

ClusterControl is the all-inclusive open source database management system for users with mixed environments that removes the need for multiple management tools. ClusterControl provides advanced deployment, management, monitoring, and scaling functionality to get your MySQL, MongoDB, and PostgreSQL databases up-and-running using proven methodologies that you can depend on to work. At the core of ClusterControl is it’s automation functionality that let’s you automate many of the database tasks you have to perform regularly, like deploying new databases, detecting anomalies, recovering nodes from failures, adding and scaling new nodes, running backups and upgrades, and more.

About Severalnines

Severalnines provides automation and management software for database clusters. We help companies deploy their databases in any environment, and manage all operational aspects to achieve high-scale availability.

Severalnines' products are used by developers and administrators of all skills levels to provide the full 'deploy, manage, monitor, scale' database cycle, thus freeing them from the complexity and learning curves that are typically associated with highly available database clusters. Severalnines is often called the “anti-startup” as it is entirely self-funded by its founders. The company has enabled over 12,000 deployments to date via its popular product ClusterControl. Currently counting BT, Orange, Cisco, CNRS, Technicolor, AVG, Ping Identity and Paytrail as customers. Severalnines is a private company headquartered in Stockholm, Sweden with offices in Singapore, Japan and the United States.

Cloud computing is booming and the cost to rent a server, or even an entire datacenter has been going down in the last couple of years. There are many cloud providers to choose from, and most of them provide pay-per-use business models. These benefits are tempting enterprises to move away from traditional on-premise bare-metal infrastructure to a more flexible cloud infrastructure as the latter supports their business growth more effectively.

ClusterControl can be used to automate the management of databases in the cloud.In this blog post, we are going to look into a number of considerations that users tend to overlook when running ClusterControl in the cloud. Some of them can be seen as limitations of ClusterControl, due to the nature of the cloud environment itself.

IP Address and Networking

There a couple of things to look out for:

• ClusterControl have limited capabilities in provisioning IPv6 hosts. When deploying a database server or cluster, please use IPv4 whenever possible.
• ClusterControl must be able to reach the database host directly, either by using the FQDN, public IP address or internal IP address.
• ClusterControl does not support the bastion host approach to access the monitored hosts. Bastion host is a common setup in the cloud, whose purpose is to provide access to a private network from an external network. This reduces the risk of penetration from the external network. You can think of it as a gateway between an inside network and an outside network.
• ClusterControl relies on sticky hostname or IP address. Changing IP address of the monitored hosts is not straightforward in ClusterControl. Try to use a dedicated IP address whenever possible. Otherwise, use a dedicated FQDN.

Since version 1.4.1, ClusterControl is able to manage database instances that runs with multiple network interfaces per host. This is a pretty common setup in the cloud, where a cloud instance is usually assigned with an internal IP address which maps to the outside world via Network Address Translation (NAT) with a public IP address. In the pre-1.4.1 version, ClusterControl could only use the public IP address to monitor the database instances from outside, and it would deploy the database cluster based on the public IP address. ClusterControl now tackles this limitation by separating IP address connectivity per host into two types - management and data address.

When deploying a new cluster or importing an existing cluster, ClusterControl will perform sanity checks once you have entered the IP address. During this check, ClusterControl will detect the operating system, collect host stats and network interfaces of the host. You can then choose which IP address that you would like to use as the management and data IP address. The management interface is used by ClusterControl operations, while the other interface is solely for database cluster communications.

The following is an example on how you can choose the data IP address during deployment:

Once the database cluster is deployed, you should see both IP addresses represented under the host, in the Nodes section. Keep in mind that having the ClusterControl host close to the monitored database server, e.g. in the same network, is always a better option.

Identity Management

ClusterControl only runs on Linux (see here for supported OS platforms). It uses SSH as the main communication channel to provision hosts. SSH offers simple implementation, strong authentication with secure encrypted data communication and is the number one choice used by administrators to control systems remotely. ClusterControl uses key-based SSH to connect to each of the monitored hosts, to perform management and monitoring tasks using a sudo user or root. In fact, the first and foremost step that you need to do after installing ClusterControl is to set up passwordless SSH authentication to all target database hosts (see the Getting Started page).

When launching a new Linux-based cloud instance, the instance is usually installed with an SSH server. The cloud provider usually generates a new private key, or associates an existing private key into the created instance for the sudo user (or root user) so the person who holds the private key can authenticate and access the server using an SSH client. Some cloud providers offer password-based SSH and will generate a random password for the same purpose.

If you already have a private key associated with the instance, you can use the same key as the SSH identity in ClusterControl to provision other cloud instances directly. It’s not necessary for you to setup a passwordless SSH key for the new nodes, since all of them are associated with the very same private key. If you’re on AWS EC2 for example, upload the SSH key to the ClusterControl host:

$ scp mykey.pem ubuntu@54.128.10.104:~

Then assign the correct ownership or permission on ClusterControl host:

$ chown ubuntu.ubuntu /home/ubuntu/mykey.pem
$ chmod 400 /home/ubuntu/mykey.pem

Then, when being asked by ClusterControl to specify the SSH user and its associated SSH key file, specify the following:

Otherwise, if you don’t have an SSH key, generate a new one and copy them to all database nodes that you would like to manage under ClusterControl:

$ whoami
ubuntu
$ ssh-keygen -t rsa # press Enter for all prompts
$ ssh-copy-id ubuntu@{target host} # repeat this for each target host

ClusterControl known limitations in this area:

• SSH key with passphrase is not supported.
• When using a user other than root, ClusterControl will default to connect through SSH with pseudo-terminal (tty). This would usually cause wtmp log file to grow quickly. See Setup wtmp Log Rotation for details.

Firewall and Security Groups

When running in the public cloud, you will be exposed to threats if your instances are open to the internet. It’s recommended for you to use a firewall (or security groups in cloud terminology) to only open the required ports. If ClusterControl is monitoring the cloud instances externally (i.e. from e.g. outside the security group that the databases are in), do open the relevant ports as listed in this documentation page. Otherwise, consider running ClusterControl in the same network as the databases. We recommend users to isolate their database infrastructure from the public Internet and just whitelist the known hosts or networks to connect to the database.

If you use ClusterControl command line client called s9s, you have to expose port 9501 as well. This is the ClusterControl backend (CMON) RPC interface that runs on TLS for the client to communicate. This service is configured to listen to localhost interface only by default. To configure this, take a look at this documentation. The s9s CLI tool allows you to control your database cluster through command lines remotely from anywhere, including your local workstation.

Instance Types and Specifications

As mentioned in the Hardware Requirements page, the minimum server requirement for ClusterControl is 2GB of RAM and 2 CPU cores. However, you can start with the smallest, lowest cost instance and resize it when the host requires more resources.

ClusterControl relies on a MySQL database to store and retrieve its monitoring data. During the ClusterControl installation, the installer script (install-cc) will configure the MySQL configuration options according to the resource it sees at this stage, especially the innodb_buffer_pool_size. If you upgrade the instance to a higher spec, do not forget to tweak this option accordingly inside the my.cnf and restart the MySQL service. A simple rule of thumb would be 50% of the total available memory in the host.

Take note that the following will affect the performance of the ClusterControl instance:

• The number of hosts it monitors
• MySQL server where ‘cmon’ and ‘dcps’ databases reside
• Granularity of the sampling

Migrating a cloud instance from one site to another, either across availability zones or geographical regions is pretty straightforward. You can simply take a snapshot of the instance and fire up a new instance based on the taken snapshot in another region. ClusterControl relies on proper IP address configuration, it will not work correctly if the database is started as a new instance with a different IP address. To manage the new instance, it is advisable to re-import the database node to ClusterControl.

With the global video streaming market expected to grow to US$70+ Billion and the video streaming software market to US$7+ Billion by 2022, it’s no surprise that video streaming is getting a lot of attention these days.

Who hasn’t sat in front of their laptop or held their tablet in their hands cursing the screen as the video or live show you’d been looking forward to watch painfully stops and goes due to lags in the video streaming application?

It’s also no surprise then that behind the scenes of this massive new market, the biggest database challenges with streaming video are maintaining uptime and scaling to ensure consistent delivery, no matter how many users are accessing the content. And no matter how much data that usage generates.

At Severalnines, we’re lucky to count a number of video streaming providers amongst our trusted customers, who rely on ClusterControl to automate and manage the open source databases that power their video streaming platforms. And they identified the following as their main database challenges:

• Automation
• Redundancy
• Scaling
• Performance

Companies or organisations who stream online media need advanced options for handling and distributing the database load required by the application. And our all-inclusive database automation and management system, ClusterControl, does exactly that and more for them.

We recently sat down with one these customers, StreamAMG, to discuss what their database challenges are. StreamAMG is Europe’s largest player in online video solutions, helping football teams such as Liverpool FC, Aston Villa, Sunderland AFC and the BBC keep fans watching from across the world.

Andrew de Bono, Platform Manager, explains how they manage these challenges with the help of ClusterControl. Watch the video interview below.

ClusterControl provides advanced scaling, management and fault tolerance features to allow you to maintain uptime and operate at peak performance.

To find out more about how ClusterControl helps overcome database challenges and get started with it yourself, visit our website on: https://severalnines.com/product/clustercontrol

Reference points: Video Streaming Market 70.05 Billion USD by 2021

Severalnines was pleased to yet again sponsor Percona Live Europe, the Open Source Database Conference which was held this year in Dublin, Ireland.

At the Conference

Severalnines team members flew in from around the world to partner up with our two local Dubliners to demo ClusterControl in the exhibit hall and present three sessions (see below).

On our Twitter feed we live tweeted both of the keynote sessions to help keep those who weren’t able to attend up-to-speed on the latest happenings in the database world.

We were also able to sit down with René Cannaò, creator of ProxySQL to talk about what’s new with the exciting load balancing technology.

Our Sessions

Members of the Severalnines team presented three technical sessions, all of which were widely attended… some with standing room only!

MySQL Load Balancers - MaxScale, ProxySQL, HAProxy, MySQL Router & nginx - A Close Up Look

Session Details: Load balancing MySQL connections and queries using HAProxy has been popular in the past years. Recently however, we have seen the arrival of MaxScale, MySQL Router, ProxySQL and now also Nginx as a reverse proxy.

For which use cases do you use them and how well do they integrate in your environment? This session aims to give a solid grounding in load balancer technologies for MySQL and MariaDB.

We review the main open-source options available: from application connectors (php-mysqlnd, jdbc), TCP reverse proxies (HAproxy, Keepalived, Nginx) and SQL-aware load balancers (MaxScale, ProxySQL, MySQL Router).

We also look into the best practices for backend health checks to ensure load balanced connections are routed to the correct nodes in several MySQL clustering topologies. You'll gain a good understanding of how the different options compare, and enough knowledge to decide which ones to explore further.

MySQL on Docker - Containerizing the Dolphin

Session Details: Docker is becoming more mainstream and adopted by users as a method to package and deploy self-sufficient applications in primarily stateless Linux containers. It's a great toolset on top of OS-level virtualization (LXC, a.k.a containers) and plays well in the world of micro services.

However, Docker containers are transient by default. If a container is destroyed, all data created is also lost. For a stateful service like a database, this is a major headache to say the least.

There are a number ways to provide persistent storage in Docker containers. In this presentation, we will talk about how to setup a persistence data service with Docker that can be torn down and brought up across hosts and containers.

We touch upon orchestration tools, shared volumes, data-only-containers, security and configuration management, multi-host networking, service discovery and implications on monitoring when we move from host-centric to role-centric services with shorter life cycles.

Automating and Managing MongoDB: An Analysis of Ops Manager vs. ClusterControl

Session Details: In any busy operations environment, there are countless tasks to perform - some monthly, or weekly, some daily or more frequently, and some on an ad-hoc basis. And automation is key to performing fast, efficient and consistently repeatable software deployments and recovery.

There are many generic tools available, both commercial and open source, to aid with the automation of operational tasks. Some of these tools are even deployed in the database world. However, there are a small number of specialist domain-specific automation tools available also, and we are going to compare two of these products: MongoDB?s own Ops Manager, and ClusterControl from Severalnines.

We cover Installation and maintenance, Complexity of architecture, Options for redundancy, Comparative functionality, Monitoring, Dashboard, Alerting, Backing up and restoring, Automated deployment of advanced configurations, and Upgrading existing deployments

Thanks to the Percona Team for organising another great conference and to everyone who participated from near and afar! We hope to see you again soon!

Join us on October 24th for a different perspective on how to automate and manage your MongoDB or Percona Server for MongoDB databases.

In today’s business busy operations environments automation is key to performing fast, efficient and consistently repeatable software deployments and recovery. And there are many generic tools available, both commercial and open source, to aid with the automation of operational tasks.

However, there are a small number of specialist domain-specific automation tools available, and we are going to compare the MongoDB-relevant functionality of two of these products: MongoDB’s Ops Manager, and ClusterControl from Severalnines.

Sign up below to hear all about the differences between these tools, and how they help automate and manage MongoDB operations. We’ll be covering key aspects from installation and maintenance via backing up your deployments through to doing upgrades.

Date, Time & Registration

Europe/MEA/APAC

Tuesday, October 24th at 09:00 BST / 10:00 CEST (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, October 24th at 09:00 PDT (US) / 12:00 EDT (US)

Register Now

Agenda

• Installation and maintenance
• Complexity of architecture
• Options for redundancy
• Comparative functionality
• Monitoring, Dashboard, Alerting
• Backing up your deployments
• Automated deployment of advanced configurations
• Upgrading existing deployments

Speaker

Ruairí Newman is passionate about all things cloud and automation and has worked for MongoDB, VMware and Amazon Web Services among others. He has a background in Operational Support Systems and Professional Services.

Prior to joining Severalnines, Ruairí worked for Huawei Ireland as Senior Cloud Solutions Architect on their Web Services project, where he advised on commodity cloud architecture and Monitoring technologies, and deployed and administered a Research & Development Openstack lab.

Database management has traditionally been complex and time-consuming. Deployment, with the headaches of security, complex networking, backup planning and implementation, and monitoring, has been a headache. Scaling out your database cluster has been a major undertaking. And in a world where 24/7 availability and rapid disaster recovery is expected, managing even a single database cluster can be a full-time job.

Severalnines’ ClusterControl is a database deployment and management system that addresses the above, facilitating rapid deployment of redundant, secure database clusters or nodes, including advanced backup and monitoring functionality - whether on premise or in the cloud. With plugins supporting Nagios, PagerDuty, and Zabbix, among others, ClusterControl integrates well with existing infrastructure and tools to help you manage your database servers with confidence.

MongoDB is the leading NoSQL database server in the world today. Using ClusterControl, with which you can deploy and manage either official MongoDB or Percona Server for MongoDB, Percona’s competing offering incorporating MongoDB Enterprise features, we are going to walk through deploying a MongoDB Replica Set with three data nodes, and look at some of the features of the ClusterControl application.

We’re going to run through some key features of ClusterControl, especially as they pertain to MongoDB, using Amazon Web Services. Amazon Web Services (or AWS) is the largest Infrastructure as a Service cloud provider globally, hosting millions of users all over the world.It comprises many services for all use cases from virtually unlimited object storage with S3 and highly scalable virtual machine infrastructure using EC2 all the way to enterprise database warehousing with Redshift and even Machine Learning.

Once you’ve read this blog, you may also wish to read our DIY Cloud Database on Amazon Web Services Whitepaper, which discusses configuration and performance considerations for database servers in the AWS Cloud in more detail. In addition, we have Become a MongoDB DBA, a whitepaper with more in depth MongoDB-specific detail.

To begin, first you will need to deploy four AWS instances. For a production platform, the instance type should be carefully chosen based on the guidelines we have previously discussed, but for our purposes instances with 2 virtual CPUs and 4GB RAM will be sufficient. One of these nodes will host ClusterControl, the others will be used to deploy the three database nodes.

Begin by creating your database nodes’ security group, allowing inbound traffic on port 27017. There is no need to restrict outbound traffic, but should you wish to do so, allow outbound traffic on ports 1024-65535 to facilitate outbound communication from the database servers.

Next create the security group for your ClusterControl node. Allow inbound traffic on ports 22, and 80. Add this security group ID to your database nodes security group, and allow unrestricted TCP communication. This will facilitate communication between the two security groups, without allowing ssh access to the database nodes from external clients.

Launch the instances into their respective security groups, choosing for each instance a KeyPair for which you have the ssh key. For the purposes of this task, use the same KeyPair for all instances. If you have lost the ssh key for your KeyPair, you will have to create a new KeyPair. When launching the instances, do not choose the default Amazon Linux image, instead choose an AMI based on a supported operating system listed here. As I am using AWS region EU-CENTRAL-1, I will use community AMI ami-fa2df395, a CentOS 7.3 image, for this purpose.

If you have the AWS command line tools installed, use the aws ec2 describe-instances command detailed previously to confirm that your instances are running--otherwise view your instances in the AWS web console--and when confirmed, log in to the ClusterControl instance via ssh.

Copy the public key file you downloaded when creating your KeyPair to the ClusterControl instance. You can use the scp command for this purpose. For now, let’s leave it in the default /home/centos directory, the home directory of the centos user. I have called mine s9s.pem. You will need the wget tool installed; install it using the following command:

$ sudo yum -y install wget

To install ClusterControl, run the following commands:

$ wget http://www.severalnines.com/downloads/cmon/install-cc
$ chmod +x install-cc
$ ./install-cc # as root or sudo user

The installation will walk you through some initial questions, after which it will take a few minutes to retrieve and install dependencies using your operating system’s package manager.

When installation is complete, point your web browser to http://<address of your ClusterControl instance>. You can find the external facing address of the instance using the describe-instances command, or via the AWS web console.

Once you have successfully logged in, you will see the following screen, and can continue to deploy your MongoDB Replica Set.

Figure 1: Welcome to ClusterControl!

As you can see, ClusterControl can also import existing database clusters, allowing it to manage your existing infrastructure as easily as new deployments.

For our purposes, you are going to click Deploy Database Cluster. On the next screen you will see the selection of database servers and cluster types that ClusterControl supports. Click the tab labelled MongoDB ReplicaSet. Here the values with which you are concerned are SSH User, SSH Key Path, and Cluster Name. The port should already be 22, the default ssh port, and the AMI we are using does not require a Sudo Password.

Figure 2: Deploying a MongoDB Replica Set

The ssh user for the CentOS 7 AMI is centos, and the SSH Key Path is /home/centos/s9s.pem, or the appropriate path depending on your own Key file name. Let’s use MongoDB-RS0 as the Cluster Name. Accepting the default options, we click Continue.

Figure 3: Configuring your deployment

Here we can choose between the MongoDB official build, and a Percona build. Select whichever you prefer, and supply an admin user and password with which to configure MongoDB securely. Note that ClusterControl will not let you proceed unless you provide these details. Make a note of the credentials you have supplied, you will need them to log in to the deployed MongoDB database, if you wish to later use it. Now choose a Replica Set name, or accept the default. We are going to use the vendor repositories, but be aware that you can configure ClusterControl to use your own repositories or those of a third party, if you prefer.

Add your database nodes, one at a time. You can choose to use the external IP address, but if you provide the hostname, which is generally recommended, ClusterControl will record all network interfaces in the hosts, and you will be able to choose the interface on which you would like to deploy. Once you have added your three database nodes, click Deploy. ClusterControl will now deploy your MongoDB Replica Set. Click Full Job Details to observe as it carries out the configuration of your cluster. When the job is complete, go to the Database Clusters screen and see your cluster.

Figure 4: Auto Recovery

Taking a closer look, you can see that Auto Recovery is enabled at both a cluster and a node level; in the case of failures, ClusterControl will attempt to recover your cluster or the individual node having an issue. The green tick beside each node also displays the cluster’s health status at a glance.

Figure 5: Scheduling Backups

The last feature we will cover here is Backups. ClusterControl provides a backup feature that allows a full cluster consistent backup, or simply a standard mongodump backup if you prefer. It also provides the facility to create scheduled backups to run periodically to a schedule of your choosing. Backup retention is also handled, with the option to retain backups for a limited period, avoiding storage issues.

In this blog I’ve attempted to give you a brief overview of using ClusterControl with MongoDB, but there are many more features supported by ClusterControl. Deployment of Sharded Clusters, with hidden and/or delayed slaves, arbiters and other features are all available. More information is available on our website, where you can also find webinars, whitepapers, tutorials, and training, and try out ClusterControl free.

We’re excited to announce that Tyler Duzan, Product Manager at Percona, will join our colleague Ruairí Newman, Senior Support Engineer, on October 24th to present a different perspective on how to automate and manage your MongoDB or Percona Server for MongoDB databases.

Tyler will walk us through some of the key features of the drop-in compatible Percona Server for MongoDB as it compares to MongoDB, and how it is “Community ++”.

We had the chance to catch up with Tyler to ask him about what he’s going to be discussing during this webinar. Watch the interview here.

Ruairí will talk about the tools that are available, both commercial and open source, to aid with the automation of operational MongoDB tasks. There are a small number of specialist domain-specific automation tools available, and Ruairí will compare the MongoDB-relevant functionality of two of these products: MongoDB’s Ops Manager, and ClusterControl from Severalnines.

Sign up below to hear all about Percona Server for MongoDB and the differences between these two tools, and how they help automate and manage MongoDB operations. We’ll be covering key aspects from installation and maintenance via backing up your deployments through to doing upgrades.

“See” you there!

Date, Time & Registration

Europe/MEA/APAC

Tuesday, October 24th at 09:00 BST / 10:00 CEST (Germany, France, Sweden)

Register Now

North America/LatAm

Tuesday, October 24th at 09:00 PDT (US) / 12:00 EDT (US)

Register Now

Agenda

• Introduction to Percona Server for MongoDB
• How to automate and manage MongoDB
  • Installation and maintenance
  • Complexity of architecture
  • Options for redundancy
  • Comparative functionality
  • Monitoring, Dashboard, Alerting
  • Backing up your deployments
  • Automated deployment of advanced configurations
  • Upgrading existing deployments

Speakers

Ruairí Newman, Senior Support Engineer at Severalnines, is passionate about all things cloud and automation and has worked for MongoDB, VMware and Amazon Web Services among others. He has a background in Operational Support Systems and Professional Services.

Prior to joining Severalnines, Ruairí worked for Huawei Ireland as Senior Cloud Solutions Architect on their Web Services project, where he advised on commodity cloud architecture and Monitoring technologies, and deployed and administered a Research & Development Openstack lab.

Tyler Duzan: Prior to joining Percona as a Product Manager, Tyler spent almost 13 years as an operations and security engineer in a variety of different industries. Deciding to take his analytical mindset and strategic focus into new territory, Tyler is applying his knowledge to solving business problems for Percona customers with inventive solutions combining technology and services.