ClusterControl’s Vault Integration Adds Dynamic Data Security Protections

The security is paramount nowadays, especially for the organizations that deal with the sensitive data - financial, health, personal data. In those cases you are obliged to comply with numerous security and data protection standards like HIPAA, PCI DSS or similar. With the most recent release, ClusterControl makes it even easier to meet the strict standards. ClusterControl 1.8.0 comes with the support for integration with HashiCorp Vault. Let’s take a look at the improvements it brings.

ClusterControl integration with HashiCorp Vault

ClusterControl is a management platform therefore it requires full access to the systems it manages - databases. It has to store passwords for management users. Currently it’s done in the cluster configuration files - all the required access details for the cluster are stored in one of the configuration files located in /etc/cmon.d/cmon_X.cnf. Here is the example for the PostgreSQL cluster:

postgresql_password=pass

postgresql_server_addresses=10.0.0.105:5432,10.0.0.106:5432

postgresql_user=myuser

repl_password=!ROos}Shf9Z,6fWdF2Iq83V0wXom!P03

repl_user=cmon_replication

rpc_key=yx9i2r8olpou9lgi

As you can see, whoever can access this file will be able to access the PostgreSQL nodes. It also stores rpc_key that can be used to manage the cluster using ClusterControl. Of course, those files are accessible only to the root user and one can argue that if someone has gained access to the root account, you are pretty much in a bad spot anyway, but ensuring that there are no passwords in the configuration file is still a good practice.

root@vagrant:~# ls -alh /etc/cmon.d/

total 28K

drwx------   2 root root 4.0K Sep 23 10:02 .

drwxr-xr-x 103 root root 4.0K Sep 23 13:34 ..

-rw-------   1 root root  668 Sep 23 13:34 cmon_1.cnf

-rw-------   1 root root  695 Sep 23 13:34 cmon_2.cnf

-rw-------   1 root root  600 Sep 23 13:34 cmon_3.cnf

-rw-------   1 root root  585 Sep 23 13:34 cmon_4.cnf

-rw-r--r--   1 root root   81 Sep  1  2016 README.txt

Here enters HashiCorp Vault - it is a product dedicated to storing sensitive data, just like the passwords we want to protect. It made sense to use it to integrate with ClusterControl. As of now, in ClusterControl 1.8.0, the only way to integrate is to use a pre-deployed Vault. Vault can be deployed in a number of ways, including clustered deployment. Covering this is out of the scope of this blog, we will still show you how to test the integration by deploying a standalone node of HashiCorp Vault and configure it with ClusterControl. Let’s take a look at the steps required to accomplish this.

Deploying HashiCorp Vault

The deployment process for HashiCorp Vault is pretty straightforward. At first, we have to create directory for the data:

mkdir -p /root/vault/data

cd /root/vault

Then we should download and decompress the Vault:

wget https://releases.hashicorp.com/vault/1.5.3/vault_1.5.3_linux_amd64.zip

unzip vault_1.5.3_linux_amd64.zip

Next, we should create SSL certificate:

# create an SSL certificate

if [ ! -f cert.crt ]; then

    openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout cert.key -out cert.crt

fi

Then we should create Valut’s configuration file:

# create a Vault configuration

cat - > vault.hcl <<CFGEND

ui = true



storage "file" {

  path = "/root/vault/data"

}



listener "tcp" {

  address     = "0.0.0.0:8200"

  tls_cert_file = "/root/vault/cert.crt"

  tls_key_file = "/root/vault/cert.key"

}



# replace with you computer's IP

api_addr = "https://10.0.0.100:8200"

CFGEND

Once this is done, you can start the Vault. It will have to be initialized and later unsealed before ClusterControl will be able to actually use it.

# Start the vault server:

$ ./vault server -config=vault.hcl

Starting from then the UI at https://IP:8200/ui/ is available and you can use it for management actions regarding the Vault. It is not necessary to use it though. We have to initialize the Vault now:

root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator init -key-shares=1 -key-threshold=1

Unseal Key 1: HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=



Initial Root Token: s.f1kl9XUBuFjdyV048QIgHoJh



Vault initialized with 1 key shares and a key threshold of 1. Please securely

distribute the key shares printed above. When the Vault is re-sealed,

restarted, or stopped, you must supply at least 1 of these keys to unseal it

before it can start servicing requests.



Vault does not store the generated master key. Without at least 1 key to

reconstruct the master key, Vault will remain permanently sealed!



It is possible to generate new unseal keys, provided you have a quorum of

existing unseal keys shares. See "vault operator rekey" for more information.

You can see here unseal key and root token. Unseal key is required in order to unseal the Vault, which we will do in the next step. Root token will be used by ClusterControl to connect to the Vault.

Let’s unseal the Vault:

root@vagrant:~/vault# VAULT_SKIP_VERIFY=1 ./vault operator unseal HfovvS/3HdBEN8XwnrnNKd65eB7I1j8WHSbvUhjG4Xk=

Key             Value

---             -----

Seal Type       shamir

Initialized     true

Sealed          false

Total Shares    1

Threshold       1

Version         1.5.3

Cluster Name    vault-cluster-2bf2e433

Cluster ID      350fd80a-67c8-bda0-6d6a-e77354dacfa2

HA Enabled      false

Now we have to setup ClusterControl to use the Vault to store passwords. This can be accomplished by creating following entries in the /etc/cmon.cnf:

vault_token=s.f1kl9XUBuFjdyV048QIgHoJh

vault_auto_migrate=true

Vault_token should contain a root token for the Vault and vault_auto_migrate=true means that ClusterControl will automatically migrate data from configuration files into the Vault. This is a good moment to make a copy of your existing configuration files located in /etc/cmon.d/ as you will need them should you decide to revert this process and do not use Vault at some point in the future.

Once you have the configuration set up, you have to restart cmon process to migrate data into the Vault:

service cmon restart

This is it, ClusterControl, from now on, will use the data in the Vault to access the backend databases. We can easily check what exactly is stored there:

root@vagrant:~/vault# export VAULT_TOKEN=s.f1kl9XUBuFjdyV048QIgHoJh

root@vagrant:~/vault# ./vault secrets list

Path               Type         Accessor              Description

----               ----         --------              -----------

clustercontrol/    kv           kv_939a83de           Created by ClusterControl

cubbyhole/         cubbyhole    cubbyhole_42b40ca0    per-token private secret storage

identity/          identity     identity_594e8bfc     identity store

sys/               system       system_9509de58       system endpoints used for control, policy and debugging

Now, knowing the path, we can query the data:

root@vagrant:~/vault# ./vault kv list clustercontrol/

Keys

----

clustercontrol/



root@vagrant:~/vault# ./vault kv list clustercontrol/clustercontrol

Keys

----

cluster_0

cluster_1

cluster_2

cluster_3

cluster_4

Let’s retrieve data for cluster_4:

root@vagrant:~/vault# ./vault kv get clustercontrol/clustercontrol/cluster_4

====== Metadata ======

Key              Value

---              -----

created_time     2020-09-24T10:51:27.242621166Z

deletion_time    n/a

destroyed        false

version          5



============ Data ============

Key                     Value

---                     -----

backup_user_password    Y3rocu-7wWGSJ-56iMK7cd8@yK+70AzH

mysql_password          pass

postgresql_password     pass

repl_password           !ROos}Shf9Z,6fWdF2Iq83V0wXom!P03

rpc_key                 yx9i2r8olpou9lgi

As you can see, the passwords are stored in the Vault. You can access the data using the UI as well, after authenticating with root token:

Please keep in mind that ClusterControl, once it is configured to work with Vault, will require Vault to be available and unsealed before ClusterControl can work properly. If it won’t be available, ClusterControl won’t work. 

Should you want to revert this process, you should follow those steps:

1. Get all the keys and values from the Vault for all of the clusters.
2. Stop cmon process
3. Remove Vault configuration from /etc/cmon.cnf
4. Add the keys and values you extracted from the Vault into respective configuration files in /etc/cmon.d. You can use the backup copy of the configuration from before Vault has been used but please keep in mind that some additional passwords might have been added in the meantime - double check that your old configuration files contain all the data that is stored in the Vault.

We hope this short introduction to ClusterControl integration with HashiCorp Vault will be useful. If you decided to give it a try, let us know how it worked for you.