Keeping backups of your database is one of the most important tasks in any production environment. It is the process of copying your data to some other place to keep it safe. This can be useful in recovery from emergency situations like database corruption or a database crashing beyond repair.

Apart from recovery, a backup can also be used to mimic a production database for testing an application in a different environment, or even to debug something that can not be done on the production database.

There are various methods of database backups that you can implement, from logical backup using tools that are embedded in the database (eg. mysqldump, mongodump, pg_dump) to physical backup using third party tools (eg. xtrabackup, barman, pgbackrest, mongodb consistent backup). 

Which method to use is often determined on how you would like to restore. For instance, assume you dropped a table or a collection by mistake. Unlikely as it might seem, it does happen. So the fastest way to recover would be to restore just that table or collection, instead of having to restore an entire database.

Backup and Restore in Mongodb

Mongodump and mongorestore is the tool for logical backup used in MongoDB, it is kind of mysqldump in MySQL, or pg_dump in PostgreSQL. The mongodump and mongorestore utility will be included when you install MongoDB and it dumps the data in BSON format. Mongodump is used to backup the database logically into dump files, while mongorestore is used for the restore operation.

mongodump and mongorestore commands are easy to use, although there are a lot of options.  

As we can see below, you can backup specific databases or collections. You can even take a point in time snapshot by including the oplog.

root@n2:~# mongodump --help

Usage:

  mongodump <options>



Export the content of a running server into .bson files.



Specify a database with -d and a collection with -c to only dump that database or collection.



See http://docs.mongodb.org/manual/reference/program/mongodump/ for more information.



general options:

      --help                                                print usage

      --version                                             print the tool version and exit



verbosity options:

  -v, --verbose=<level>                                     more detailed log output (include multiple times for more verbosity, e.g. -vvvvv, or specify a numeric value, e.g. --verbose=N)

      --quiet                                               hide all log output



connection options:

  -h, --host=<hostname>                                     mongodb host to connect to (setname/host1,host2 for replica sets)

      --port=<port>                                         server port (can also use --host hostname:port)



kerberos options:

      --gssapiServiceName=<service-name>                    service name to use when authenticating using GSSAPI/Kerberos ('mongodb' by default)

      --gssapiHostName=<host-name>                          hostname to use when authenticating using GSSAPI/Kerberos (remote server's address by default)



ssl options:

      --ssl                                                 connect to a mongod or mongos that has ssl enabled

      --sslCAFile=<filename>                                the .pem file containing the root certificate chain from the certificate authority

      --sslPEMKeyFile=<filename>                            the .pem file containing the certificate and key

      --sslPEMKeyPassword=<password>                        the password to decrypt the sslPEMKeyFile, if necessary

      --sslCRLFile=<filename>                               the .pem file containing the certificate revocation list

      --sslAllowInvalidCertificates                         bypass the validation for server certificates

      --sslAllowInvalidHostnames                            bypass the validation for server name

      --sslFIPSMode                                         use FIPS mode of the installed openssl library



authentication options:

  -u, --username=<username>                                 username for authentication

  -p, --password=<password>                                 password for authentication

      --authenticationDatabase=<database-name>              database that holds the user's credentials

      --authenticationMechanism=<mechanism>                 authentication mechanism to use



namespace options:

  -d, --db=<database-name>                                  database to use

  -c, --collection=<collection-name>                        collection to use



uri options:

      --uri=mongodb-uri                                     mongodb uri connection string



query options:

  -q, --query=                                              query filter, as a JSON string, e.g., '{x:{$gt:1}}'

      --queryFile=                                          path to a file containing a query filter (JSON)

      --readPreference=<string>|<json>                      specify either a preference name or a preference json object

      --forceTableScan                                      force a table scan



output options:

  -o, --out=<directory-path>                                output directory, or '-' for stdout (defaults to 'dump')

      --gzip                                                compress archive our collection output with Gzip

      --repair                                              try to recover documents from damaged data files (not supported by all storage engines)

      --oplog                                               use oplog for taking a point-in-time snapshot

      --archive=<file-path>                                 dump as an archive to the specified path. If flag is specified without a value, archive is written to stdout

      --dumpDbUsersAndRoles                                 dump user and role definitions for the specified database

      --excludeCollection=<collection-name>                 collection to exclude from the dump (may be specified multiple times to exclude additional collections)

      --excludeCollectionsWithPrefix=<collection-prefix>    exclude all collections from the dump that have the given prefix (may be specified multiple times to exclude additional prefixes)

  -j, --numParallelCollections=                             number of collections to dump in parallel (4 by default) (default: 4)

      --viewsAsCollections                                  dump views as normal collections with their produced data, omitting standard collections

There are many options in mongorestore command, mandatory option is related to connection options such as host, port, and authentication. There are other parameters, like -j used to restore collections in parallel, -c or --collection is used for a specific collection, and -d or --db is used to define a specific database. The list of options of the mongorestore parameter can be shown using help : 

root@n2:~# mongorestore --help

Usage:

  mongorestore <options> <directory or file to restore>



Restore backups generated with mongodump to a running server.



Specify a database with -d to restore a single database from the target directory,

or use -d and -c to restore a single collection from a single .bson file.



See http://docs.mongodb.org/manual/reference/program/mongorestore/ for more information.



general options:

      --help                                                print usage

      --version                                             print the tool version and exit



verbosity options:

  -v, --verbose=<level>                                     more detailed log output (include multiple times for more verbosity, e.g. -vvvvv, or specify a numeric value, e.g. --verbose=N)

      --quiet                                               hide all log output



connection options:

  -h, --host=<hostname>                                     mongodb host to connect to (setname/host1,host2 for replica sets)

      --port=<port>                                         server port (can also use --host hostname:port)



kerberos options:

      --gssapiServiceName=<service-name>                    service name to use when authenticating using GSSAPI/Kerberos ('mongodb' by default)

      --gssapiHostName=<host-name>                          hostname to use when authenticating using GSSAPI/Kerberos (remote server's address by default)



ssl options:

      --ssl                                                 connect to a mongod or mongos that has ssl enabled

      --sslCAFile=<filename>                                the .pem file containing the root certificate chain from the certificate authority

      --sslPEMKeyFile=<filename>                            the .pem file containing the certificate and key

      --sslPEMKeyPassword=<password>                        the password to decrypt the sslPEMKeyFile, if necessary

      --sslCRLFile=<filename>                               the .pem file containing the certificate revocation list

      --sslAllowInvalidCertificates                         bypass the validation for server certificates

      --sslAllowInvalidHostnames                            bypass the validation for server name

      --sslFIPSMode                                         use FIPS mode of the installed openssl library



authentication options:

  -u, --username=<username>                                 username for authentication

  -p, --password=<password>                                 password for authentication

      --authenticationDatabase=<database-name>              database that holds the user's credentials

      --authenticationMechanism=<mechanism>                 authentication mechanism to use



uri options:

      --uri=mongodb-uri                                     mongodb uri connection string



namespace options:

  -d, --db=<database-name>                                  database to use when restoring from a BSON file

  -c, --collection=<collection-name>                        collection to use when restoring from a BSON file

      --excludeCollection=<collection-name>                 DEPRECATED; collection to skip over during restore (may be specified multiple times to exclude additional collections)

      --excludeCollectionsWithPrefix=<collection-prefix>    DEPRECATED; collections to skip over during restore that have the given prefix (may be specified multiple times to exclude additional prefixes)

      --nsExclude=<namespace-pattern>                       exclude matching namespaces

      --nsInclude=<namespace-pattern>                       include matching namespaces

      --nsFrom=<namespace-pattern>                          rename matching namespaces, must have matching nsTo

      --nsTo=<namespace-pattern>                            rename matched namespaces, must have matching nsFrom



input options:

      --objcheck                                            validate all objects before inserting

      --oplogReplay                                         replay oplog for point-in-time restore

      --oplogLimit=<seconds>[:ordinal]                      only include oplog entries before the provided Timestamp

      --oplogFile=<filename>                                oplog file to use for replay of oplog

      --archive=<filename>                                  restore dump from the specified archive file. If flag is specified without a value, archive is read from stdin

      --restoreDbUsersAndRoles                              restore user and role definitions for the given database

      --dir=<directory-name>                                input directory, use '-' for stdin

      --gzip                                                decompress gzipped input



restore options:

      --drop                                                drop each collection before import

      --dryRun                                              view summary without importing anything. recommended with verbosity

      --writeConcern=<write-concern>                        write concern options e.g. --writeConcern majority, --writeConcern '{w: 3, wtimeout: 500, fsync: true, j: true}'

      --noIndexRestore                                      don't restore indexes

      --noOptionsRestore                                    don't restore collection options

      --keepIndexVersion                                    don't update index version

      --maintainInsertionOrder                              preserve order of documents during restoration

  -j, --numParallelCollections=                             number of collections to restore in parallel (4 by default) (default: 4)

      --numInsertionWorkersPerCollection=                   number of insert operations to run concurrently per collection (1 by default) (default: 1)

      --stopOnError                                         stop restoring if an error is encountered on insert (off by default)

      --bypassDocumentValidation                            bypass document validation

      --preserveUUID                                        preserve original collection UUIDs (off by default, requires drop)

Restoring specific collections in MongoDB can be done using the parameter collection in mongorestore. Assume you have an orders database, inside the orders database there are some collections as shown below:

my_mongodb_0:PRIMARY> show dbs;

admin   0.000GB

config  0.000GB

local   0.000GB

orders  0.000GB

my_mongodb_0:PRIMARY> use orders;

my_mongodb_0:PRIMARY> show collections;

order_details

orders

stock

We already have scheduled a backup for the orders database, and we want to restore the stock collection into a new database order_new in the same server. If you want to use option --collection, you need to pass the collection name as  parameter of mongorestore or you can use the option --nsInclude={db}.{collection} if you didn’t specify the path to the collection file.

root@n2:~/dump/orders# mongorestore -umongoadmin --authenticationDatabase admin --db order_new --collection stock /root/dump/orders/stock.bson

Enter password:

​2020-03-09T04:06:29.100+0000 checking for collection data in /root/dump/orders/stock.bson

2020-03-09T04:06:29.110+0000 reading metadata for order_new.stock from /root/dump/orders/stock.metadata.json

2020-03-09T04:06:29.134+0000 restoring order_new.stock from /root/dump/orders/stock.bson

2020-03-09T04:06:29.202+0000 no indexes to restore

2020-03-09T04:06:29.203+0000 finished restoring order_new.stock (1 document)

2020-03-09T04:06:29.203+0000 done

You can check the collection in order_new database as shown below :

​my_mongodb_0:PRIMARY> use order_new;

switched to db order_new

my_mongodb_0:PRIMARY> show collections;

stock

How We Can Restore Using mongodump in ClusterControl

Restoring a backup dump through ClusterControl is easy, you just need 2 steps to restore the backup. There will be lots of backup files in the list if you enabled your backup schedule, there is some information about the backups that can be very useful. For example, status of backup which indicates if the backup was completed / failed, method of backup had taken, list of databases, and size of dump. The steps to restore MongoDB data via ClusterControl are as below :

Step One

Follow the prompts to restore backup to a node as shown below...

Step Two

You need to choose which backup that needs to be restored. 

Step Three

Review the summary...

Previously, we wrote about Setting Up a Geo-Distributed Database Cluster Using MySQL Replication. This time, it's about PostgreSQL. Setting up a geo-distributed cluster for PostgreSQL is not a new concept and the topology is quite common. 

To achieve high availability, organizations and companies are dispersing their database nodes so that when a catastrophic event happens in a specific region (which affects your data center) you have your standby nodes available for failover.

This is a very common practice (using this type of topology) as part of your organization's Business Continuity and Disaster Recovery plans. This type of topology removes having a single point of failure (SPOF). A common requirement especially if you have a low RPO and a higher uptime (if possible at 99.999999999%).

In this blog, I'll take a simple implementation on how to do this using ClusterControl. ClusterControl is an agentless management and automation software for database clusters. It helps deploy, monitor, manage, and scale your database server/cluster directly from ClusterControl user interface.

The Desired Architectural Setup

The target outcome here is to deploy efficiently in a secure environment. To do this, it's important that you need to place your established connection in between using VPN and would be more secure if you also setup your database nodes over TLS/SSL connection. For this setup in our blog, we simply deploy a node over a VPN and showcase to you how you can easily do this approach. See below for the diagram of the target setup:

To elaborate the setup, the on-premise network shall communicate over the public cloud using a VPN tunnel and both of these networks shall have a VPN gateway so both can communicate or establish a connection. ClusterControl requires you to oversee all the nodes that have to be registered as it will collect information about your nodes for data metrics. Aside from that, it requires that your on-prem active-writer node can also reach the standby node into the other domain, which is for this blog, hosted in Google Cloud Platform (GCP).

Setting Up Your OpenVPN

OpenVPN setup is very tricky for both network domains. The gist of this is that, it must have the following consideration:

• Nodes from your on-prem shall be able to establish a connection to the target public cloud domain nodes
• Nodes from your on-prem can be able to have internet access to download packages that are required to set up. Unless you have all the repositories stored locally that are required, this can be not the case
• Nodes from your public cloud domain shall be able to establish a connection to the on-premise nodes
• Nodes from your public cloud domain can be able to have internet access to download packages that are required to set up. Unless you have all the repositories stored locally that are required, this can be not the case

OpenVPN Installation and Configuration

Step One

Install the openvpn package (and easy-rsa packages for Ubuntu/Debian distros)

$ sudo apt-get install openvpn easy-rsa

For CentOS/RHEL based OS, 

$ sudo yum install openvpn wget

$ wget -O /tmp/easyrsa https://github.com/OpenVPN/easy-rsa-old/archive/2.3.3.tar.gz

Step Two

Generate your certificates such as certification authority (CA), server, and client certificates. 

For Ubuntu/Debian, you ca do the following actions:

$ /usr/bin/make-cadir CA

Change to CA directory

$ cd CA

At this point, you might likely to edit vars file in accordance to your needs, e.g.

export KEY_COUNTRY="SE"

export KEY_PROVINCE="SMD"

export KEY_CITY="Kalmar"

export KEY_ORG="Severalnines"

export KEY_EMAIL="paul@s9s.io"

export KEY_CN="S9s"

export KEY_NAME="server"

export KEY_OU="Support Unit"

Then execute the vars script to define the required env variables

[ ~/CA ]$ source ./vars

NOTE: If you run ./clean-all, I will be doing a rm -rf on /CA/keys

Run a clean-up

[ ~/CA ]$ ./clean-all

Then build the certificates for your CA, server, and client.

[ ~/CA ]$ ./build-ca

[ ~/CA ]$ ./build-key-server server

 $ ./build-dh 2048

[ ~/CA ]$ ./build-key client

Lastly, generate a Perfect Forward Secrecy key.

$ openvpn --genkey --secret pfs.key

If you're using CentOS/RHEL type distros, you can do the following:

$ tar xfz /tmp/easyrsa

$ sudo mkdir /etc/openvpn/easy-rsa

$ sudo cp -rf easy-rsa-old-2.3.3/easy-rsa/2.0/* /etc/openvpn/easy-rsa

# Ensure your RSA keys are on the right permission for security purposes

$ sudo chown vagrant /etc/openvpn/easy-rsa/

$ sudo cp /usr/share/doc/openvpn-2.4.4/sample/sample-config-files/server.conf /etc/openvpn

$ sudo mkdir /etc/openvpn/easy-rsa/keys

$ sudo nano /etc/openvpn/easy-rsa/vars

$ cd /etc/openvpn/easy-rsa

At this point, you might likely to edit vars file in accordance to your needs, e.g.

export KEY_COUNTRY="SE"

export KEY_PROVINCE="SMD"

export KEY_CITY="Kalmar"

export KEY_ORG="Severalnines"

export KEY_EMAIL="paul@s9s.io"

export KEY_CN="S9s"

export KEY_NAME="server"

export KEY_OU="Support Unit"

Then execute the vars script to define the required env variables

$ source ./vars

NOTE: If you run ./clean-all, I will be doing a rm -rf on /CA/keys

Run a clean-up

$ ./clean-all

Then build the certificates for your CA, server, and client.

$ ./build-ca

$ ./build-key-server server

$ ./build-dh 2048

$ cd /etc/openvpn/easy-rsa

$ ./build-key client

$ cp /etc/openvpn/easy-rsa/openssl-1.0.0.cnf /etc/openvpn/easy-rsa/openssl.cnf

Once you have all setup, you must take into account where your keys and certificates are in place. If you're using systemd or service in Linux to run this, then you might place your certificates and keys to /etc/openvpn. Likely, you might have to run the following command:

sudo cp dh2048.pem ca.crt server.crt server.key /etc/openvpn

Step Three

At this point, I end up with the following server and client configuration. See my configuration files accordingly,

OpenVPN Server Config

$ cat /etc/openvpn/server-ovpn.conf 

port 1194

proto udp

dev tun

ca /etc/openvpn/keys/ca.crt

cert /etc/openvpn/keys/server.crt

key /etc/openvpn/keys/server.key # This file should be kept secret

dh /etc/openvpn/keys/dh2048.pem

cipher AES-256-CBC

auth SHA512

server 10.8.0.0 255.255.255.0

client-to-client

topology subnet

push "route 192.168.30.0 255.255.255.0"

#push "redirect-gateway def1 bypass-dhcp"

#push "redirect-gateway"

push "dhcp-option DNS 8.8.8.8"

push "dhcp-option DNS 8.8.4.4"

ifconfig-pool-persist ipp.txt

keepalive 10 120

comp-lzo

persist-key

persist-tun

#status openvpn-status.log

#log-append  openvpn.log

verb 3

tls-server

tls-auth /etc/openvpn/keys/pfs.key

The most important thing you need to take into account are these following options as noted as follows.

client-to-client - Very important so nodes in the VPN can ping the other nodes in different network domain. Say, ClusterControl is located in on-prem, it can ping the nodes in GCP.

push "route 192.168.30.0 255.255.255.0"- I push the routing tables so that GCP node/s connected to VPN can ping my nodes in the on-premise domain. In my GCP VPN gateway, I have the following routing tables as push "route 10.142.0.0 255.255.255.0"

#push "redirect-gateway def1 bypass-dhcp" ,

#push "redirect-gateway"- Both these sections are not required since I need internet connection for both to setup my repo and dependent packages upon installation. 

push "dhcp-option DNS 8.8.8.8", 

push "dhcp-option DNS 8.8.4.4"-  Both these sections can be changed to your desired DNS if needed. This is for your desired DNS especially when you need internet connection.

OpenVPN Client Config

$ cat openvpn/client-vpn.ovpn 

client

dev tun

proto udp

remote 34.73.238.239  1194  

ca ca.crt

cert client.crt

key client.key

tls-version-min 1.2

tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256

cipher AES-256-CBC

auth SHA512

resolv-retry infinite

auth-retry none

nobind

persist-key

persist-tun

ns-cert-type server

comp-lzo

verb 3

tls-client

tls-auth pfs.key

The most important thing here is you need to be sure of your key paths, and also replace the params in this section,

remote 34.73.238.239  1194  

which can be the hostname/IP address of your VPN server gateway to connect to.

Step Four

Lastly, setup proxy VPN server to have the network packets routed to the network interface on the server and allow the kernel to forward IPV4 traffic

sudo iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

sudo echo 1 > /proc/sys/net/ipv4/ip_forward

For more in-depth installation, I suggest you look at these posts for CentOS and on Ubuntu.

Extending Over the Cloud Efficiently

Suppose you have the following topology in your on-prem domain,

and you now want to extend your availability over another datacenter, which is GCP for this blog. Deploying using ClusterControl is very straightforward. You can do the following procedure stated below,

Step One

Create A Slave Cluster

Step Two

Choose your master for replication,

Step Three

Setup the access to your public cloud environment

Step Four

Specify the hostname/IP of your node to be extended into your PG replication cluster,

Step Five

Lastly, monitor the job activity how does ClusterControl reacts to this type of action

The outcome will show you the linkage between your on-prem and your extended datacenter, which is in this blog, our GCP PostgreSQL standby node. See below for the result

Conclusion

Setting up a geo-location standby node is not difficult,  but the main issue is how secure will this be in your architectural design. Using a VPN can alleviate the main concern of the problem. Using OpenVPN is just a simple way to implement this but for heavy transactional applications, organizations are likely to invest on upscale services or hardware to deal with this setup. Also by adding a TLS/SSL can be easier than done. We'll discuss this on how you can use TLS/SSL with PostgreSQL in our next blogs.

Modern IT needs to have a non-relational, dynamic schema (meaning no requirements for Joins Statements Queries) to provide support for Big Data/real-time applications. NoSQL databases were created with the notion of improving data processing performance and tackling the ability to scale out to overcome distributed database load using the concept of multiple hosts have won the new generation demand for data processing. 

Besides providing the essential support for various data models and scripting languages, MongoDB also allows easy to start with the process for the developers. 

NoSQL database open the doors to... 

• Text-based protocols using a scripting language (REST and, JSON, BSON)
• Indeed minimal cost to generate, store and transport data
• Support huge amounts of data processing. 
• Increased write performance
• Not required to perform object-relational mapping and normalization process
• No rigid controls with referential integrity rules
• Reducing maintenance cost with database administrators 
• Lowering expansion cost
• Fast key-value access
• Advancing the support for machine learning and intelligence 

MongoDB Market Acceptance 

The modern needs for Big Data Analytics and modern applications play a crucial role in the need to improve the lifecycle of data processing, with no expectations for hardware expansion and cost increase. 

If you are planning for a new application, and you want to choose a database, arriving at the right decision with many database options in the market can be a complicated process. 

The DB-engine popularity ranking shows that MongoDB stands at no.1 compared to Oracle NoSQL (which placed at No. 74). The trend, however, is indicating that something is changing. The need for many cost-effective expansions goes hand in hand with much simpler data modelling, and administration is transforming how developers would want to consider the best for their systems. 

According to Datanyze market share information to-date there are about 289 websites that are running on Oracle Nosql with a market share of 11%, where else MongoDB has a complete 12,185 website with a market share of 4.66%. These impressive numbers indicate that there is a bright future for MongoDB. 

NoSQL Data Modeling 

Data modelling requires understanding of...

• The types of your current data. 
• What are the types of data that you are expecting in the future?
• How is your application gaining access to required data from the system?
• How is your application going to fetch required data for processing?

The exciting thing for those who have always followed the Oracle way of creating schemas, then storing the data, MongoDB allows creating the collection along with the document. This means the creation of collections is not a must-have to exist before the document creation takes place, making MongoDB much appreciated for its flexibility. 

In Oracle NoSQL, however,  the table definition has to be created first, after which you can continue to create the rows.  

The next cool thing is that MongoDB does not imply strict rules on schema and relations implementation, which gives you the freedom for continuous improvement of the system without fearing much on the need to ensure tight schematic design. 

Let’s look at some of the comparisons between MongoDB and Oracle NoSQL.

Comparing NoSQL Concepts in MongoDB and Oracle

NoSQL Terminologies

Document Store and Key-Value Store 

Oracle NoSQL provides a storage system that stores values indexed by a key; this concept is viewed as the least complex model as the datasets consist of an indexed key-value. The records organised using major keys and minor keys. 

The major key can be viewed as the object pointer and the minor key as the fields in the record.  Efficient Search for the data is made possible with the use of the key as the mechanism to access the data just like a Primary key. 

MongoDB extends key-value pairs. Each document has a unique key, which serves the purpose to retrieve the document. Documents are known as dynamic schema, as the collections in a document do not need to have the same set of fields. A collection can have a common field with different types of data. These attributes lead the document data model to map directly to support the modern object-oriented languages.

MongoDB	Oracle NoSQL
Document store Example:	Key-value store Example:

BSON and JSON

Oracle NoSQL uses JSON as a standard data format to transmit (data + attribute-value pairs). On the other hand MongoDB uses BSON. 

BSON is not in a human-readable text, unlike JSON. BSON stands for binary-encoded serialization of JSON like data, mainly used for data storage and a transfer format with MongoDB. BSON data format consists of a list of ordered elements containing a field name (string), type, and value. As for the data types BSON supports, all the datatypes commonly found in JSON and includes two additional datatypes (Binary Data and Date). Binary data or known as BinData that is less than 16MB can be stored directly into MongoDB documents. BSON is said to be consuming more space than JSON data documents. 

There are two reasons why MongoDB consumes more space as compared to Oracle NoSQL: 

• MongoDB achieved the objective of being able to traverse fast, enabling the option to traverse fast requires the BSON document to carry additional metadata (length of string and subobjects). 
• BSON design can encode and decode fast. For example, integers are stored as 32 (or 64) bit integers, to eliminate parsing to and from the text. This process uses more space than JSON for small integers but is much faster to parse.

Data Model Definition

MongoDB Collection Statement

db.createCollection("users")

db.users.insert

( {
    User_id: "U1",
    First_name: "Mary"                  
    Last_name : "Winslet",  

    Age       : 15

    Contact   : {

               Phone: "123-456-789"

               Email: "mary@example.com"  

                }

   access  : {

              Level:5,

              Group:"dev"

             }            

})

Create namespace newns1; 

news1:users

News1:users.access

Create table newns1.user (

idValue INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1 MAXVALUE 10000), 

User_id String,

First_name String,

Last_name String, 

Contact Record (Phone string,         

                Email string),

Primary key (idValue));

Create table newns1.user (

idValue INTEGER GENERATED ALWAYS AS IDENTITY (START WITH 1 INCREMENT BY 1 MAXVALUE 10000),

User_profile JSON, 

Primary Key (shard(idValue),User_id));

{

  "id":U1,

  "User_profile" : {

     "First_name":"Mary",

     "Lastname":"Winslet",

     "Age":15,

     "Contact":{"Phone":"123-456-789",                   

     "Email":"mary@example.com"

                   }

}

Oracle NoSQL does have several ways to start defining tables. If you are using the Oracle SQL CLI by default, new table creation will be placed in sysdefault until you decide to create a new namespace to associate a set of new tables with it. The above example demonstrates the new namespace “ns1” created, and the user table is associated with the new namespace. 

Besides identifying the primary key, Oracle NoSQL also uses the IDENTITY column to auto-increment a value each time you add a row. The IDENTITY value is auto-generated and must be an Integer, long or number data type. In Oracle NoSQL, IDENTITY associates with the Sequence Generator similar to the concept of objectid with MongoDB. As Oracle NoSQL allows the IDENTITY key to be used as the primary key. If you are considering IDENTITY key as the primary key, this is where careful consideration is required as it may have an impact over the insertion of data and the update process takes place.  

MongoDB and Oracle NoSQL table/collection level definition show how the ‘contact’ information is embedded into the same single structure without requiring additional schema definition. The benefit of embedding a dataset is that no further queries would be necessary to retrieve the embedded dataset.  

If you are looking to maintain your system in a simple form, MongoDB provides the best option to retain the data documents with less complication. At the same time, MongoDB provides the capabilities to deliver the existing complex data model from relational schema using schema validation tool.

Oracle NoSQL provides the capabilities to use SQL, like query language with DDL and DML, which requires much less effort for users who have some experience with the use of Relation Database systems.  

MongoDB shell uses Javascript, and if you are not comfortable with the language or with the use of mongo shell, then the best fit for the process is to opt to use an IDE tool. The top 5 MongoDB IDEtools in 2020,  like studio 3T, Robo 3T, NoSQLBooster, MongoDB Compass and Nucleon Database Master will be helpful to assist you in creating and managing complex queries with the use of aggregation features. 

Performance and Availability

As the MongoDB data structure model uses documents and collections, using BSON data format for processing a huge amount of data becomes much faster compared to Oracle NoSQL. While some consider querying data with SQL is a more comfortable pathway for many users, the capacity becomes an issue. When we have a huge amount of data to support, the need for increased throughput and followed by the use of SQL to Design complex queries, these processes are asking us to relook at the server capacity and cost increase over time. 

Both MongoDB and Oracle NoSQL provide sharding and replication features. Sharding is a process that allows the dataset and the overall processing load to be distributed across multiple physical partitions to increase processing (read/write) speed. The implementation of shard with oracle requires you to have prior information on how sharding keys work. The reason behind the pre-planning process is due to the need of having to implement the shard key at the schema initiation level.

The implementation of shard with MongoDB gives the room for you to work on your dataset first to identify the potential right shard key based on query patterns before implementation. As the sharding process includes data replication, MongoDB has a reputation for fast data replication as well. Replication takes care of fault tolerance of having to have all data in a single server. 

Conclusion 

What makes MongoDB preferred over Oracle NoSQL is that it is in binary format and its inborn characteristics of lightweight, traversable, and efficient. This allows you to support the advancing modern application in the area of machine learning and artificial intelligence. 

MongoDB characteristics enable the developers to work much more confidently to build modern applications faster. The MongoDB data model allows the processing of huge amounts of unstructured data with an improved speed that is well thought off compared to Oracle NoSQL. Oracle NoSQL wins when it comes to tools it has to offer and possible options to create data models. However, it is essential to make sure developers and designers can learn and adapt to technology fast, which is not the case with Oracle NoSQL.

Backups are the means of protecting from data loss - should something happen, you can easily restore it from the backup. You cannot predict what part of the data will have to be restored - it can be everything or just a subset. Typically you want to have a full backup to ensure you can handle a total data loss scenario but what would happen if only a single table had been dropped? Can we do a partial restore if we used Xtrabackup to create our safety data copy? Let’s explore this scenario in a short blog post.

Partial Restore Using Xtrabackup

The main thing you have to keep in mind before you perform a partial restore with Xtrabackup is that this will break the consistency of the node where you would restore the backup. This is extremely important in replication or Galera setups where the consistency of the cluster is paramount as otherwise replication (standard or Galera) may break. 

How to approach this problem? It all depends on your environment. One of the solutions could be to use a separate host to restore missing data and then proceed with regular logical backup, something that you can restore on the live cluster without introducing data inconsistency. 

Alternatively, if you can afford to stop the whole cluster, you can perform the restore on all of the nodes in the cluster - this as well will result in a consistent state of the data across the whole environment. We won’t go into details how to proceed because, as we stated, this may depend on your business requirements, ability to schedule a downtime and so on. 

For now let’s take a look how to restore a single table, not focusing where you would do that.

We are assuming that a full backup created by Xtrabackup is ready. We have a simple environment of asynchronous replication with one master and one slave. We use Percona Server 8.0 therefore we ensured we have percona-xtrabackup-80 installed.

As can be seen, the backup has been created:

root@vagrant:~# ls -alh /backup/

total 149M

drwxr-xr-x  6 root root 4.0K Mar 13 12:24 .

drwxr-xr-x 25 root root 4.0K Mar 13 12:23 ..

-rw-r-----  1 root root 479 Mar 13 12:24 backup-my.cnf

-rw-r-----  1 root root 195 Mar 13 12:24 binlog.000005

-rw-r-----  1 root root   16 Mar 13 12:24 binlog.index

-rw-r-----  1 root root 5.8K Mar 13 12:24 ib_buffer_pool

-rw-r-----  1 root root 100M Mar 13 12:24 ibdata1

drwxr-x---  2 root root 4.0K Mar 13 12:24 mysql

-rw-r-----  1 root root 24M Mar 13 12:24 mysql.ibd

drwxr-x---  2 root root 4.0K Mar 13 12:24 performance_schema

drwxr-x---  2 root root 4.0K Mar 13 12:24 sbtest

drwxr-x---  2 root root 4.0K Mar 13 12:24 sys

-rw-r-----  1 root root 12M Mar 13 12:24 undo_001

-rw-r-----  1 root root 12M Mar 13 12:24 undo_002

-rw-r-----  1 root root   63 Mar 13 12:24 xtrabackup_binlog_info

-rw-r-----  1 root root   99 Mar 13 12:24 xtrabackup_checkpoints

-rw-r-----  1 root root 540 Mar 13 12:24 xtrabackup_info

-rw-r-----  1 root root 8.5K Mar 13 12:24 xtrabackup_logfile

-rw-r-----  1 root root 248 Mar 13 12:24 xtrabackup_tablespaces

Now, if we want to restore it, we have to prepare the backup - it’s a standard process for Xtrabackup. There is one major difference though in a way we will prepare it. We will use --export flag:

root@vagrant:~# xtrabackup --prepare --export --target-dir=/backup/

Now we can restore a particular table following this process:

1. We have to create the table using exactly the same schema as it used to have when the backup has been taken.
2. We have to discard its tablespace
3. We will copy the tablespace from the backup along with its *.cfg file
4. We will import new tablespace

Let’s assume one of the tables has been accidentally truncated:

mysql> SELECT COUNT(*) FROM sbtest.sbtest11\G

*************************** 1. row ***************************

COUNT(*): 0

1 row in set (0.00 sec)

In this case we already have the table with a proper schema in place and we can proceed to step 2):

mysql> ALTER TABLE sbtest.sbtest11 DISCARD TABLESPACE;

Query OK, 0 rows affected (0.02 sec)

Now we have to copy the data from the backup:

root@vagrant:~# cp /backup/sbtest/sbtest11.* /var/lib/mysql/sbtest/

root@vagrant:~# chown mysql.mysql /var/lib/mysql/sbtest/sbtest11.*

Finally, we can import the restored tablespace:

mysql> ALTER TABLE sbtest.sbtest11 IMPORT TABLESPACE;

Query OK, 0 rows affected (0.48 sec)



mysql> SELECT COUNT(*) FROM sbtest.sbtest11\G

*************************** 1. row ***************************

COUNT(*): 100000

1 row in set (0.05 sec)

As you can see, the contents of the table have been restored. Now, based on how we approached the whole problem, we can either repeat this process on all of the nodes in the cluster or we can use mysqldump or SELECT … INTO OUTFILE to extract this data and then load it on the live cluster.

Please keep in mind that Xtrabackup allows as well to take a backup of a single database or single table. This is another feature, loosely tied to what we have just discussed - it is not required to create a backup of a single table to be able to restore it. What is required though is the schema - you may want to schedule backups of the schema (no data is required) using mysqldump that will go along with your xtrabackup backups. You may find them very handy if your schema changes often.

How to Restore a Single Table Using ClusterControl?

ClusterControl, as of now, does not come with an ability to restore a single table out of full backup. You can schedule partial backups with ClusterControl though. Then you can use those backups and restore them on a separate host and then extract the data and apply it on the live cluster.

As you can see on the screenshot, you can decide which database you want to backup and then list the tables (or decide that you want to include all of them) you would like to backup. You can setup a backup schedule where you would backup individual tables, one at a time. You could as well design the schedule on a schema-per-schema basis. Once you have a backup ready, you can restore it on a standalone host:

Then, we will have to decide what host it is. You also have to make sure this host can be reached from ClusterControl node using SSH.

We want ClusterControl to setup software, provision it with data and then keep the server running after the backup has been restored.

We should review the options we took and then confirm that the backup should be restored.

Job has been started and all we need to do is to wait for it to complete.

Once the job has completed, you can access the backup verification server, dump the missing data and restore it on the live cluster.

Using a Load Balancer is a good idea for any database technology, as you can redirect applications to the available or healthy database nodes and even distribute the traffic across multiple servers to improve performance. This is not only useful on-prem but also in a cloud environment. In this blog, we’ll see how to deploy and configure a new database cluster with HAProxy on the Google Cloud Platform from scratch.

Creating the VM on Google Cloud

For this example, we’ll assume that you have a Google Cloud account created.

You can deploy your virtual machines directly from ClusterControl. Go to the deploy section and select “Deploy in the Cloud”.

Specify vendor and version for your new cluster.

Add the number of nodes, cluster name, and database information.

Choose the cloud credentials, in this case, your Google Cloud account. If you don’t have your account added in ClusterControl, you can follow our documentation for this task.

Now you can specify the virtual machine configuration, like operating system, size, and region.

ClusterControl will create the virtual machines, install the software, and configure it, all in the same job and in an unattended way.

You can monitor the creation process in the ClusterControl activity section. When it finishes, you will see your new cluster in the ClusterControl main screen.

Deploying HAProxy in Google Cloud

Note: To deploy it, first, you need to create the VM in the Google Cloud Platform as the virtual machine creation is not implemented for the ClusterControl load balancer deployment yet (it will be available soon).

Now you have your new cluster up and running, go to ClusterControl -> Select Cluster -> Cluster Actions -> Add Load Balancer.

Here you must add the information that ClusterControl will use to install and configure your HAProxy load balancer.

The information that you need to introduce is:

Action: Deploy or Import.

Server Address: IP Address for your HAProxy server.

Listen Port (Read/Write): Port for read/write mode.

Listen Port (Read-Only): Port for read-only mode.

Policy: It can be:

• leastconn: The server with the lowest number of connections receives the connection.
• roundrobin: Each server is used in turns, according to their weights.
• source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request.

Install for read/write splitting: For master-slave replication.

Build from Source: You can choose Install from a package manager or build from source.

And you need to select which servers you want to add to the HAProxy configuration and some additional information like:

Role: It can be Active or Backup.

Include: Yes or No.

Connection address information.

Also, you can configure Advanced Settings like Admin User, Backend Name, Timeouts, and more.

When you finish the configuration and confirm the deployment, you can follow the progress in the Activity section on the ClusterControl UI.

And when this finishes, you can go to ClusterControl -> Nodes -> HAProxy node, and check the current status.

You can also monitor your HAProxy servers from ClusterControl checking the Dashboard section.

Conclusion

A Load Balancer can help you to handle your database traffic by balancing it between multiple servers. It is also useful to improve your high availability environment by performing failover tasks. ClusterControl can help you too with different features like auto-recovery, monitoring, deployment, and even more, and it can manage on-prem, cloud or mixed environments with different database technologies at the same time.

It is extremely important to install and configure a production MySQL server with the necessary packages and tools to smooth-out the operations in the long run. We have seen many cases where troubleshooting or tuning a production server (especially one without public internet access) is commonly difficult because of the lack of necessary tools installed on the server to help identify and solve the problem. 

In this two-part blog series, we are going to show you 9 tips and tricks on how to prepare a MySQL server for production usage from a system administrator perspective. All examples in this blog post are based on our two-node, master-slave MySQL Replication setup running on CentOS 7.

Install Essential Packages

After the installation of MySQL or MariaDB client and server packages, we need to prepare the MySQL/MariaDB server with all necessary tools to cope with all the administration, management and monitoring operations that are going to happen on the server. If you are planning to lock down the MySQL server in production, it will be a bit harder to install them all manually without the Internet connection. 

Some of the important packages that should be installed on the MySQL/MariaDB server for Linux:

• Percona Xtrabackup/MariaDB Backup - Non-blocking physical backup of the database server.
• ntp/ntpdate - Sync server's time.
• pv - Monitor data through a pipeline, can also be used for throttling.
• socat or netcat- Data streaming tool, good for streaming backup.
• net-tools - A collection of network debugging tools for Linux.
• bind-utils - A collection of DNS debugging tools for Linux.
• sysstat - A collection of performance monitoring tools for Linux.
• telnet - Telnet client to check service reachability.
• mailx/mailutils - MTA client.
• openssl - Toolkit for the Transport Layer Security (TLS) and Secure Sockets Layer (SSL) protocols.
• unzip - Uncompress tool.
• htop - Host monitoring tool.
• innotop - MySQL monitoring tool.
• vim - Text editor with syntax highlighting (or any preferred text editor).
• python-setuptools - Python package manager.
• lm_sensors/ipmitool - To check server component's temperature. Bare-metal server only.

Note that some of the suggested packages are only available in non-default package repositories like EPEL for CentOS. Therefore, for YUM-based installation:

$ yum install epel-release

$ yum install -y wget ntp pv socat htop innotop vim mailx bind-utils net-tools telnet sysstat openssl python-setuptools lm_sensors ipmitool

While for APT-based installation:

$ apt-get install ntp pv socat htop innotop vim easy_install mailx bind-utils sysstat net-tools telnet openssl lm_sensors ipmitool

For MySQL command line interface, we can use another tool other than the standard "mysql" command line client like mycli, with auto-completion and syntax highlighting. To install the package, we can use pip (Python package manager):

$ pip install mycli

With mycli, one can reduce the human-error vector with a better visualization when dealing with production server, as shown in the following screenshot:

Meaningful Shell Prompt

This part looks unnecessary in the first place, but it is probably going to save you from making silly mistakes in production. As a human, we are prone to make errors especially when running destructive commands during an intense moment, for example when the production server is down.

Take a look at the following screenshot. By default, the bash PS1 prompt (primary prompt) looks pretty dull:

A good PS1 prompt should provide distinctful information to make SysAdmins more aware of the environment, server and current path that they are currently dealing with. As a result, one would be more careful and always know whether it's the path/server/ to execute the command.

To achieve this, simply find the line that describing PS1 (primary prompt) configuration, commonly in /etc/bashrc line 41:

  [ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\u@\h \W]\\$ "

And replace it with this line:

  [ "$PS1" = "\\s-\\v\\\$ " ] && PS1="[\[\e[36m\]\u\[\e[m\]@\[\e[32m\]\h\[\e[m\]\[\e[31;47m\]Production\[\e[m\]: \[\e[33m\]\w\[\e[m\]]$ "

Log out from the terminal and re-login again. You should see something like this in the terminal now:

As shown in the screenshot above, the current user (blue), server's hostname (green), Production tier (bold in red colour with white background), together with the full path of the current directory (yellow) provides a better summary of the current session where the important information are easily distinguishable with different colours.

You can use this free online tool to customize your bash prompt, to suit your taste.

MOTD

If you are managing a database cluster with multiple roles like MySQL or MariaDB replication, it's common to always have this anxious feeling when directly administering one of the hosts because we need to perform extra checks to verify that the node that we are in is the one that we really want to administer. Replication topology tends to become more complex as your database cluster scales out and there could be many roles in a cluster like intermediate master, binlog server, backup master with semi-sync replication, read-only slaves and also backup verification server.

It will be way better if we can get a summary of the database state whenever we are in that particular server, just to give us a heads up on what we are going to deal with. We can utilize Linux's Message of the Day (MOTD) to automate this behaviour whenever we log into the server. Using the default /etc/motd is only good for static content, which is not what we really want if we want to report the current state of a MySQL server.

To achieve similar result, we can use a simple Bash script to produce a meaningful MOTD output to summarize our MySQL/MariaDB server, for example:

$ vim ~/.motd.sh

#!/bin/bash

# Auto-generate MOTD for MySQL/MariaDB Replication

# .motd.sh, to be executed under ~/.bash_profile



#####

# Preferred role of the node, pick one



#PREFER_ROLE='Slave'

PREFER_ROLE='Master'

#####



HOSTNAME=$(hostname)

UPTIME=$(uptime -p)

MYSQL_COMMAND='mysql --connect-timeout=2 -A -Bse'

MYSQL_READONLY=$(${MYSQL_COMMAND} 'SHOW GLOBAL VARIABLES LIKE "read_only"' | awk {'print $2'})

TIER='Production'

MAIN_IP=$(hostname -I | awk {'print $1'})

CHECK_MYSQL_REPLICATION=$(${MYSQL_COMMAND} 'SHOW SLAVE STATUS\G' | egrep 'Slave_.*_Running: Yes$')

MYSQL_MASTER=$(${MYSQL_COMMAND} 'SHOW SLAVE STATUS\G' | grep Master_Host | awk {'print $2'})

MYSQL_UPTIME=$(${MYSQL_COMMAND} 'SELECT TIME_FORMAT(SEC_TO_TIME(VARIABLE_VALUE ),"%Hh %im")  AS Uptime FROM information_schema.GLOBAL_STATUS WHERE VARIABLE_NAME="Uptime"')



# coloring

bold=$(tput bold)

red=$(tput setaf 1)

green=$(tput setaf 2)

normal=$(tput sgr0)



MYSQL_SHOW=1

if [ $MYSQL_READONLY == 'ON' ]; then

        CURRENT_MYSQL_ROLE='Slave'

        if ${MYSQL_COMMAND} 'SHOW SLAVE STATUS\G' | egrep 'Slave_.*_Running: Yes$'&>/dev/null ; then

                lag=$(${MYSQL_COMMAND} 'SHOW SLAVE STATUS\G' | egrep 'Seconds_Behind_Master:' | awk {'print $2'})

                if [ $lag -eq 0 ]; then

                        REPLICATION_STATUS="${green}Healthy  "

                else

                        if [ $lag == 'NULL' ]; then

                                REPLICATION_STATUS=${red}Unhealthy

                        else

                                REPLICATION_STATUS="${red}Lagging ${lag}s"

                        fi

                fi

        else

                REPLICATION_STATUS=${red}Unhealthy

        fi



elif [ $MYSQL_READONLY == 'OFF' ]; then

        CURRENT_MYSQL_ROLE='Master'

        SLAVE_HOSTS=$(${MYSQL_COMMAND} 'SHOW SLAVE HOSTS' | awk {'print $1'})

else

        MYSQL_SHOW=0

fi



if [ $TIER == 'Production' ]; then

        TIER=${green}Production

fi



if [ $PREFER_ROLE == $CURRENT_MYSQL_ROLE ]; then

        MYSQL_ROLE=${green}$CURRENT_MYSQL_ROLE

else

        MYSQL_ROLE=${red}$CURRENT_MYSQL_ROLE

fi



echo

echo "HOST INFO"

echo "========="

echo -e "  Hostname       : ${bold}$HOSTNAME${normal} \t Server Uptime  : ${bold}$UPTIME${normal}"

echo -e "  IP Address       : ${bold}$MAIN_IP${normal} \t Tier           : ${bold}$TIER${normal}"

echo

if [ $MYSQL_SHOW -eq 1 ]; then

echo "MYSQL STATE"

echo "==========="

echo -e "  Current role      : ${bold}$MYSQL_ROLE${normal} \t\t Read-only      : ${bold}$MYSQL_READONLY${normal}"

echo -e "  Preferred role    : ${bold}$PREFER_ROLE${normal} \t\t DB Uptime      : ${bold}$MYSQL_UPTIME${normal}"

if [ $CURRENT_MYSQL_ROLE == 'Slave' ]; then

echo -e "  Replication state : ${bold}$REPLICATION_STATUS${normal} \t Current Master : ${bold}$MYSQL_MASTER${normal}"

else

echo -e "  Slave Hosts(s) ID : "

for i in $SLAVE_HOSTS; do

echo -e "      - ${bold}$i${normal} \t"; done

fi

echo

fi

Choose one of the MySQL roles, either a master or a slave on line 8 or 9 and save the script. This script requires MySQL option file to store the database user credentials, so we have to create it first:

$ vim ~/.my.cnf

And add the following lines:

[client]

user=root

password='YourRootP4ssw0rd'

Replace the password part with the actual MySQL root password. Then, apply executable permission to the script:

$ chmod 755 ~/.motd.sh

Test the executable script whether it produces the correct output or not:

$ ~/.motd.sh

If the output looks good (no errors or warnings), add the script into ~/.bash_profile so it will be automatically loaded when a user logs in:

$ whoami

root

$ echo '~/.motd.sh'>> ~/.bash_profile

Re-login the terminal and you should see something like this on the master:

While on the slave, you should see something like this:

Note that this script is specifically written for a simple MySQL/MariaDB one-tier master-slave replication. You probably have to modify the script if you have a more complex setup, or you want to use other MySQL clustering technology like Galera Cluster, Group Replication or NDB Cluster. The idea is to retrieve the database node status and information right when we logged in so we are aware of the current state of the database server that we are working on.

Sensors and Temperature

This part is commonly being ignored by many SysAdmins. Monitoring the temperatures is crucial as we do not want to get a big surprise if the server behaves unexpectedly when overheating. A physical server commonly consists of hundreds of electronic parts glued together in a box and are sensitive to temperature changes. One failed cooling fan could spike a CPU temperature to hit its hard limit, which eventually causes the CPU clock to be throttled down and affects the data processing performance as a whole.

We can use the lm-sensors package for this purpose. To install it, simply do:

$ yum install lm-sensors # apt-get install lm-sensors for APT

Then run the sensors-detect program to automatically determine which kernel modules you need to load to use lm_sensors most effectively:

$ sensors-detect

Answers all questions (commonly accept all the suggested values). Some servers like virtual machines do not support this module. Sensors really need to be at the hosts (bare-metal) level. Check out this list for more information.

Then, run the sensors command:

$ sensors

i350bb-pci-0203

Adapter: PCI adapter

loc1:         +53.0°C (high = +120.0°C, crit = +110.0°C)



power_meter-acpi-0

Adapter: ACPI interface

power1:        4.29 MW (interval =   1.00 s)



coretemp-isa-0000

Adapter: ISA adapter

Package id 0:  +55.0°C (high = +85.0°C, crit = +95.0°C)

Core 0:        +45.0°C (high = +85.0°C, crit = +95.0°C)

Core 1:        +51.0°C (high = +85.0°C, crit = +95.0°C)

Core 2:        +47.0°C (high = +85.0°C, crit = +95.0°C)

Core 3:        +51.0°C (high = +85.0°C, crit = +95.0°C)

Core 4:        +49.0°C (high = +85.0°C, crit = +95.0°C)

Core 5:        +48.0°C (high = +85.0°C, crit = +95.0°C)

Core 8:        +47.0°C (high = +85.0°C, crit = +95.0°C)

Core 9:        +49.0°C (high = +85.0°C, crit = +95.0°C)

Core 10:       +48.0°C (high = +85.0°C, crit = +95.0°C)

Core 11:       +48.0°C (high = +85.0°C, crit = +95.0°C)

Core 12:       +46.0°C (high = +85.0°C, crit = +95.0°C)

Core 13:       +49.0°C (high = +85.0°C, crit = +95.0°C)



coretemp-isa-0001

Adapter: ISA adapter

Package id 1:  +53.0°C (high = +85.0°C, crit = +95.0°C)

Core 0:        +46.0°C (high = +85.0°C, crit = +95.0°C)

Core 1:        +48.0°C (high = +85.0°C, crit = +95.0°C)

Core 2:        +47.0°C (high = +85.0°C, crit = +95.0°C)

Core 3:        +45.0°C (high = +85.0°C, crit = +95.0°C)

Core 4:        +46.0°C (high = +85.0°C, crit = +95.0°C)

Core 5:        +47.0°C (high = +85.0°C, crit = +95.0°C)

Core 8:        +47.0°C (high = +85.0°C, crit = +95.0°C)

Core 9:        +45.0°C (high = +85.0°C, crit = +95.0°C)

Core 10:       +45.0°C (high = +85.0°C, crit = +95.0°C)

Core 11:       +46.0°C (high = +85.0°C, crit = +95.0°C)

Core 12:       +46.0°C (high = +85.0°C, crit = +95.0°C)

Core 13:       +46.0°C (high = +85.0°C, crit = +95.0°C)

The above result shows the overall CPU temperature, together with its every CPU core. Another tool that we can use to see the overall state of the server components is ipmitool. To install, simply do:

$ yum -y install ipmitool

By running the following command, we can tell the overall state of the physical components in the server:

$ ipmitool sdr list full

Inlet_Temp       | 20 degrees C   | ok

PCIe_Inlet_Temp  | 37 degrees C   | ok

Outlet_Temp      | 20 degrees C   | ok

CPU0_VR_Temp     | 39 degrees C   | ok

CPU1_VR_Temp     | 41 degrees C   | ok

CPU0_Temp        | 55 degrees C   | ok

CPU1_Temp        | 52 degrees C   | ok

PCH_Temp         | 58 degrees C   | ok

DIMMG0_Temp      | 35 degrees C   | ok

DIMMG1_Temp      | 32 degrees C   | ok

PSU0_Temp        | 0 degrees C   | ok

PSU1_Temp        | 0 degrees C   | ok

SYS_3.3V         | 3.30 Volts   | ok

SYS_5V           | 5 Volts   | ok

SYS_12V          | 12.10 Volts   | ok

CPU0_VCORE       | 1.79 Volts   | ok

CPU1_VCORE       | 1.79 Volts   | ok

CPU0_DDR_VDD     | 1.23 Volts   | ok

CPU1_DDR_VDD     | 1.23 Volts   | ok

SYS_FAN1_Speed   | 4018 RPM   | ok

SYS_FAN2_Speed   | 4116 RPM   | ok

SYS_FAN3_Speed   | 4116 RPM   | ok

SYS_FAN4_Speed   | 4116 RPM   | ok

SYS_FAN5_Speed   | 4018 RPM   | ok

SYS_FAN6_Speed   | 4116 RPM   | ok

SYS_FAN7_Speed   | 4018 RPM   | ok

SYS_FAN8_Speed   | 4116 RPM   | ok

SYS_FAN9_Speed   | 4018 RPM   | ok

SYS_FAN10_Speed  | 4116 RPM   | ok

SYS_FAN11_Speed  | 4116 RPM   | ok

SYS_FAN12_Speed  | 4116 RPM   | ok

SYS_FAN13_Speed  | 4116 RPM   | ok

SYS_FAN14_Speed  | 4214 RPM   | ok

Airflow_rate     | 16 CFM   | ok

PSU1_PIN         | 0 Watts   | ok

PSU2_PIN         | 0 Watts   | ok

PSU1_POUT        | 0 Watts   | ok

PSU2_POUT        | 0 Watts   | ok

PSU1_IIN         | 0 Amps   | ok

PSU2_IIN         | 0 Amps   | ok

PSU1_VIN         | 0 Volts   | ok

PSU2_VIN         | 0 Volts   | ok

CPU_Power        | 63 Watts   | ok

MEM_Power        | 8 Watts   | ok

Total_Power      | 0 Watts   | ok

BP_Power         | 8 Watts   | ok

FAN_Power        | 6 Watts   | ok

MB_Power         | 0 Watts   | ok

The list is long but is self-explanatory and you should be able to oversee the overall server components' state. There could be cases where some of the fans are not running at full speed which then increase the CPU temperature. Hardware replacement might be required to fix the problem.

Note that the Intelligent Platform Management Interface (IPMI) kernel module requires Baseboard Management Controller (BMC) to be enabled on the motherboard. Use dmesg to verify if it is available:

$ dmesg | grep -i bmc

[    8.063470] ipmi_si IPI0001:00: Found new BMC (man_id: 0x000000, prod_id: 0x02f3, dev_id: 0x20)

Otherse, check the server's BIOS setting if this controller is disabled.

That's it for now. Part two of this blog series will cover the remaining 5 topics like backup tool configuration, stress tests, and server lock down.

In our previous blog, we have discussed ClusterControl CMON HA for Distributed Database High Availability written by Krzysztof Ksiazek in two separate posts. In this blog, we'll cover the distribution of nodes via on-prem and on a public cloud (using Google Cloud Platform (GCP)).

The reason we wrote this blog is because we have received questions about how to implement a high availability instance of ClusterControl having CMON node(s) running on-prem and another CMON node(s) running on a different data center (such as a public cloud). In our previous blog ClusterControl CMON HA for Distributed Database High Availability, we were using Galera Cluster nodes, but this time we'll use MySQL Replication using Percona Server 5.7. An ideal setup for this is to always encapsulate the communication of nodes from your on-prem and your nodes residing in a public cloud via VPN or a secure channel. 

ClusterControl CMON HA is at the early stages for which we believe it's not yet mature enough. Yet, our CMON HA is able to provide you the sense of functionality for deploying a ClusterControl to make it highly available. Let's proceed on how you can deploy and setup distributing the nodes via on-prem  through the public cloud.

What is a CMON?

Before going to the main topic, let us introduce to you what is CMON. CMON stands for ClusterControl Controller, which is the “primary brain” of ClusterControl. A backend service performing automation, management, monitoring scheduling tasks, and also the HA availability. Data that are collected are stored into the CMON database, for which we're using MySQL compatible databases as the datastore.

The Architectural Setup

Some of you might not have known the capabilities of ClusterControl that it can perform and be set up for high-availability. If you have multiple ClusterControl (or CMON) nodes running, that is possible at no cost. You might be able to run tons of ClusterControl nodes whenever you needed. 

For this setup, we'll have ClusterControl nodes on-top of a ClusterControl in order to create or deploy the database nodes and manage an auto failover whenever a failure occurs, for example. Although you can use MHA, Orchestrator, or Maxscale to manage the auto-failover, but for efficiency and speed, I'll use ClusterControl to do the special things that other tools I have mentioned do not have.

So let's have a look at the diagram for this setup:

The setup based on that diagram shows that on top of three-node CMON, a running CMON (ClusterControl) is on-top of them which will monitor the automatic failover. Then, HAProxy will be able to load balance between the monitored three CMON nodes, wherein one node is located in a separate region hosted in GCP for this blog. You might notice that we didn't include Keepalived, that's because we cannot place a VIP under GCP since it's on a different network.

As you might have noticed, we place a total of three nodes. CMON HA requires that we need at least 3 nodes in order to proceed a voting process or so called quorum. So for this setup, we require that you have at least 3 nodes to have higher availability.

Deploying an On-Prem ClusterControl Nodes

In this section, we do expect that you have already setup or installed your ClusterControl UI which we will use to deploy a three node MySQL Replication cluster using Percona Server.

Let's first create the cluster by deploying a new MySQL Replication as shown below.

Take note that I am using Percona Server 5.7 here, for which the default setup by ClusterControl works efficiently.

Then define the hostname or IP of your nodes,

At this point, we expect that you have already set up a two node Master/Slave replication which is hosted or running on-prem. The screenshot below should show how your nodes will look like:

Setup & Install ClusterControl and Enable CMON HA On The First Node

From this previous blog  ClusterControl CMON HA for Distributed Database High Availability, we have briefly provided the steps on how to do this. Let's go down again and do the steps as stated but for this particular Master/Slave replication setup.

First thing to do, pick one node you want first ClusterControl to be installed (on this setup, I end up installing first on 192.168.70.80 node) and do the steps below.

Step One

Install ClusterControl

$ wget http://www.severalnines.com/downloads/CMON/install-cc

$ chmod +x install-cc

$ sudo ./install-cc   # omit sudo if you run as root

Take note that once you are prompted that a current mysql instance is detected, you need to let ClusterControl use the existing mysqld running since that's one of our goals here for CMON HA and for this setup to use the already setup MySQL.

Step Two

Bind CMON not only to allow via localhost, but also on the specific IP address (since we'll be enabling HA)

## edit /etc/default/CMON  and modify the line just like below or add the line if it doesn't exist

RPC_BIND_ADDRESSES="127.0.0.1,192.168.70.80"

Step Three

Then restart CMON,

service CMON restart

Step Four

Install s9s CLI tools

$ wget http://repo.severalnines.com/s9s-tools/install-s9s-tools.sh

$ chmod 755 install-s9s-tools.sh

$ ./install-s9s-tools.sh

During this installation, the s9s tool will setup an admin user for which you can use when dealing with s9s command, just like enabling CMON HA.

Step Five

Enable the CMON HA

$ s9s controller --enable-CMON-ha

Step Six

Lastly, modify the /etc/my.cnf and add,

slave-skip-errors = 1062

under the [mysqld] section. Once added, do not forget to restart mysql as,

service mysql restart

or

systemctl restart mysql

Currently, this is the limitation we're facing with CMON HA since it tries to insert log entries to the slave but this can be fine for now.

Setup, Install ClusterControl and Enable CMON HA On The Second Node

Simple as that for the first node. Now, on the 2nd node (192.168.70.70),  we need to do the same steps but instead we need to do some adjustments in the steps to make this HA possible.

Step One

Copy the configuration to the 2nd node (192.168.70.70) from first node (192.168.70.80)

$ scp -r /etc/CMON* 192.168.70.70:/etc/

Step Two

In the 2nd node, edit the /etc/CMON.cnf and ensure that the host is correctly configured. e.g.

vi /etc/CMON.cnf

Then assign hostname param as,

hostname=192.168.70.70

Step Three

Install ClusterControl,

$ wget http://www.severalnines.com/downloads/CMON/install-cc

$ chmod +x install-cc

$ sudo ./install-cc   # omit sudo if you run as root

However, skip the installation of CMON (or ClusterControl Controller) once you encounter this line,

=> An existing Controller installation detected!

=> A re-installation of the Controller will overwrite the /etc/CMON.cnf file

=> Install the Controller? (y/N):

The rest, just do as what you've done on the first node such as setting up the hostname, use the existing mysqld running instance, providing the MySQL password, and password for your CMON which must be both has the same password with the first node.

Step Four

Install s9s CLI tools

$ wget http://repo.severalnines.com/s9s-tools/install-s9s-tools.sh

$ chmod 755 install-s9s-tools.sh

$ ./install-s9s-tools.sh

Step Five

Copy the remaining configuration from 1st node to the 2nd node.

$ scp -r ~/.s9s/ 192.168.70.70:/root/

$ scp /etc/s9s.conf 192.168.70.70:/etc/

$ scp /var/www/html/clustercontrol/bootstrap.php 192.168.70.70:/var/www/html/clustercontrol/

Step Six

Install clustercontrol-controller package,

For Ubuntu/Debian,

$ apt install -y clustercontrol-controller

For RHEL/CentOS/Fedora,

$ yum install -y clustercontrol-controller

Step Seven

Copy the /etc/default/CMON file and modify the IP address for the RPC bind address

scp /etc/default/CMON 192.168.70.70:/etc/default

RPC_BIND_ADDRESSES="127.0.0.1,10.0.0.103"

Then restart CMON as follows,

service CMON restart

Step Eight

Modify the /etc/my.cnf and add,

slave-skip-errors = 1062

under the [mysqld] section. Once added, do not forget to restart mysql as,

service mysql restart

or

systemctl restart mysql

Currently, this is the limitation we're facing with CMON HA since it tries to insert log entries to the slave but this can be fine for now.

Step Nine

Finally, check how the CMON HA nodes look like,

[root@node7 ~]#  s9s controller --list --long

S VERSION    OWNER GROUP NAME            IP PORT COMMENT

l 1.7.5.3735 system admins 192.168.70.80   192.168.70.80 9501 Acting as leader.

f 1.7.5.3735 system admins 192.168.70.70   192.168.70.70 9501 Accepting heartbeats.

Total: 2 controller(s)

Deploying Your ClusterControl Node In the Cloud

As we have mentioned earlier, the ideal setup for communication is to encapsulate the packets over the VPN or other means of secure channel. If you have concerns about how to do this, check our previous blog Multi-DC PostgreSQL: Setting Up a Standby Node at a Different Geo-Location Over a VPN for which we have tackled how you can create a simple VPN setup using OpenVPN. 

So in this section, we expect that you have already set up the VPN connection. Now, what we're going to do is add a slave that we're supposed to distribute the availability of CMON into Google Cloud Platform. To do this, just go to Add Replication Slave which can be found by clicking the cluster icon near the right corner. See how it looks like below:

Now, this is how we'll end up with:

Now, since we have a new slave added which is hosted under GCP, you may need to follow again on what we did earlier on the 2nd node. I'll relay you to follow those steps and follow the instructions on how we did on the 2nd node.

Once you have it correctly, you'll end up with the following result:

[root@gnode1 ~]# s9s controller --list --long

S VERSION    OWNER GROUP NAME            IP PORT COMMENT

l 1.7.5.3735 system admins 192.168.70.80   192.168.70.80 9501 Acting as leader.

f 1.7.5.3735 system admins 192.168.70.70   192.168.70.70 9501 Accepting heartbeats.

f 1.7.5.3735 system admins 10.142.0.39     10.142.0.39 9501 Accepting heartbeats.

where in nodes 

• 192.168.70.80 -  (node8) and is residing in my on-prem
• 192.168.70.70 - (node7) and is residing in my on-prem
• 10.142.0.39  - (gnode1) is hosted in GCP and on different region

CMON HA In Action

My colleague Krzysztof Ksiazek already provided the setup for HA using HAProxy here on this blog ClusterControl CMON HA for Distributed Database High Availability - Part Two (GUI Access Setup). 

To follow the procedure stated in the blog, ensure you have xinetd and pathlib packages. You can install xinetd and pathlib as follows,

$ sudo yum install -y xinetd python-pathlib.noarch

Ensure also that you have the CMONhachk defined in /etc/services just as below:

[root@node7 ~]# grep 'CMONhachk' /etc/services 

CMONhachk       9201/tcp

and ensure changes and restart xinetd,

service xinetd restart

I'll skip the Keepalived and HAProxy procedure and expect you have set up accordingly. One take away you have to consider on this setup is that using Keepalived cannot be applicable if you are dispersing the VIP from on-prem to the public cloud network because they're totally a different network.

Now, let's see how CMON HA reacts if nodes are down. As shown earlier, node 192.168.70.80 (node8), was acting as a leader just like as shown below:

Wherein the master node database also shows node8 is the master from ClusterControl topology view. Let's try to kill node8 and see how CMON HA proceeds,

As you see, gnode1 (GCP node) is taking over as a leader as node8 goes down. Checking the HAProxy results to the following,

and our ClusterControl nodes shows that node8 is down, whereas GCP node is taking over as the master,

Lastly, accessing my HAProxy node which is running on host 192.168.10.100 at port 81 shows the following UI,

Conclusion

Our ClusterControl CMON HA has been since version 1.7.2 but it has also been a challenge for us since various questions and preferences of how to deploy this such as using MySQL Replication over Galera Cluster. 

Our CMON HA is not mature yet but it is now ready to cater your high availability needs. Different approaches can be applicable as long as your checks will determine the right node that is up and running.

We encourage you to setup and deploy using CMON HA and let us know how well suits your needs or if the problem persists, please let us know how to help you cater your high availability necessities.

Mysqldump is the most popular logical backup tool for MySQL. It is included in the MySQL distribution, so it’s ready for use on all of the MySQL instances. 

Logical backups are not, however, the fastest nor the most space-efficient way of backing up MySQL databases, but they have a huge advantage over physical backups. 

Physical backups are usually all or nothing type of backups. While it might be possible to create partial backup with Xtrabackup (we described this in one of our previous blog posts), restoring such backup is tricky and time-consuming. 

Basically, if we want to restore a single table, we have to stop the whole replication chain and perform the recovery on all of the nodes at once. This is a major issue - these days you rarely can afford to stop all of the databases. 

Another problem is the table level is the lowest granularity level you can achieve with Xtrabackup: you can restore a single table but you cannot restore part of it. Logical backup, though, can be restored in the way of running SQL statements, therefore it can easily be performed on a running cluster and you can (we wouldn’t call it easily, but still) pick which SQL statements to run so you can do a partial restore of a table. 

Let’s take a look at how this can be done in the real world.

Restoring a Single MySQL Table Using mysqldump

At the beginning, please keep in mind that partial backups do not provide a consistent view of the data. When you take backups of separate tables, you cannot restore such backup to a known position in time (for example, to provision the replication slave) even if you would restore all of the data from the backup. Having this behind us, let’s proceed.

We have a master and a slave:

Dataset contains of one schema and several tables:

mysql> SHOW SCHEMAS;

+--------------------+

| Database           |

+--------------------+

| information_schema |

| mysql              |

| performance_schema |

| sbtest             |

| sys                |

+--------------------+

5 rows in set (0.01 sec)



mysql> SHOW TABLES FROM sbtest;

+------------------+

| Tables_in_sbtest |

+------------------+

| sbtest1          |

| sbtest10         |

| sbtest11         |

| sbtest12         |

| sbtest13         |

| sbtest14         |

| sbtest15         |

| sbtest16         |

| sbtest17         |

| sbtest18         |

| sbtest19         |

| sbtest2          |

| sbtest20         |

| sbtest21         |

| sbtest22         |

| sbtest23         |

| sbtest24         |

| sbtest25         |

| sbtest26         |

| sbtest27         |

| sbtest28         |

| sbtest29         |

| sbtest3          |

| sbtest30         |

| sbtest31         |

| sbtest32         |

| sbtest4          |

| sbtest5          |

| sbtest6          |

| sbtest7          |

| sbtest8          |

| sbtest9          |

+------------------+

32 rows in set (0.00 sec)

Now, we have to take a backup. There are several ways in which we can approach this issue. We can just take a consistent backup of the whole dataset but this will generate a large, single file with all the data. To restore the single table we would have to extract data for the table from that file. It is of course possible, but it is quite time-consuming and it’s pretty much manual operation that can be scripted but if you do not have proper scripts in place, writing ad hoc code when your database is down and you are under heavy pressure is not necessarily the safest idea.

Instead of that we can prepare backup in a way that every table will be stored in a separate file:

root@vagrant:~/backup# d=$(date +%Y%m%d) ; db='sbtest'; for tab in $(mysql -uroot -ppass -h127.0.0.1 -e "SHOW TABLES FROM ${db}" | grep -v Tables_in_${db}) ; do mysqldump --set-gtid-purged=OFF --routines --events --triggers ${db} ${tab} > ${d}_${db}.${tab}.sql ; done

Please note that we set --set-gtid-purged=OFF. We need it if we’d be loading this data later to the database. Otherwise MySQL will attempt to set @@GLOBAL.GTID_PURGED, which will, most likely, fail. MySQL would as well set SET @@SESSION.SQL_LOG_BIN= 0; which is definitely not what we want. Those settings are required if we’d make a consistent backup of the whole data set and we’d like to use it to provision a new node. In our case we know it is not a consistent backup and there is no way we can rebuild anything from it. All we want is to generate a dump that we can load on the master and let it replicate to slaves.

That command generated a nice list of sql files that can be uploaded to the production cluster:

root@vagrant:~/backup# ls -alh

total 605M

drwxr-xr-x 2 root root 4.0K Mar 18 14:10 .

drwx------ 9 root root 4.0K Mar 18 14:08 ..

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest10.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest11.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest12.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest13.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest14.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest15.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest16.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest17.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest18.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest19.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest1.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest20.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest21.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest22.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest23.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest24.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest25.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest26.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest27.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest28.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest29.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest2.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest30.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest31.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest32.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest3.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest4.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest5.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest6.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest7.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest8.sql

-rw-r--r-- 1 root root  19M Mar 18 14:10 20200318_sbtest.sbtest9.sql

When you would like to restore the data, all you need to do is to load the SQL file into the master node:

root@vagrant:~/backup# mysql -uroot -ppass sbtest < 20200318_sbtest.sbtest11.sql

Data will be loaded into the database and replicated to all of the slaves.

How to Restore a Single MySQL Table Using ClusterControl?

Currently ClusterControl does not provide an easy way of restoring just a single table but it is still possible to do it with just a few manual actions. There are two options you can use. First, suitable for small number of tables, you can basically create schedule where you perform partial backups of a separate tables one by one:

Here, we are taking a backup of sbtest.sbtest1 table. We can easily schedule another backup for sbtest2 table:

Alternatively we can perform a backup and put data from a single schema into a separate file:

Now you can either find the missing data by hand in the file, restore this backup to a separate server or let ClusterControl do it:

You keep the server up and running and you can extract the data that you wanted to restore using either mysqldump or SELECT … INTO OUTFILE. Such extracted data will be ready to be applied on the production cluster.

Having a multi-datacenter setup is a common topology for a Disaster Recovery Plan(DRP), but there are some limitations around implementing this kind of environment. 

You should first solve the communication between the data centers by using SSH access or configuring a VPN. Then, you have the latency that (depending on the configuration) could affect your database cluster. Finally, you should think about how to perform the failover. Can the application access the remote node in case of master failure?

In this blog, we will show how to implement a multi-datacenter setup for PostgreSQL covering all these points mentioned earlier, some of them using ClusterControl. To not make it too boring, we will split it into two parts. In the first part, we will cover the connectivity between the data centers. The second one will be about the deployment and configuration itself, so let’s start!

Objective

Let’s say you want to have the following topology:

Where you have your application connected to a load balancer, a primary database node, and one standby node in one datacenter, and another standby node in a secondary datacenter for DR purposes. This could be a minimal setup for having a multi-datacenter environment. You can avoid using the load balancer, but in case of failover, you should reconfigure your application to connect to the new master, so to avoid that we recommend using it, or even use two of them (one on each DC) to avoid single point of failure. 

To make it more clear, let’s assign some public IP addresses to both datacenter 1 and 2 as an example.

In datacenter 1, the public network is 35.166.37.0/24, so let’s assign the following IP addresses in this way:

APP: 35.166.37.10

Load Balancer + ClusterControl: 35.166.37.11

Primary Node: 35.166.37.12

Standby 1 Node: 35.166.37.13

In datacenter 2, the public network is 18.197.23.0/24, so:

Standby 2 Node: 18.197.23.14

Data Center Connectivity

The first problem could be this one. You can configure a VPN between them, and that must be the most secure way, but as we covered a VPN configuration in a previous blog, and to make it as short as possible, we will connect them via SSH access using private/public keys.

Let’s create a user called ‘remote’ in all the nodes (to avoid using root):

$ useradd remote

$ passwd remote

Changing password for user remote.

New password:

Retype new password:

passwd: all authentication tokens updated successfully.

And you can add it to the sudoers file to assign privileges:

$ visudo

remote    ALL=(ALL)       ALL

Now, in the load balancer server (which will be also the ClusterControl server), generate the key pair for the new user:

$ su remote

$ ssh-keygen

Generating public/private rsa key pair.

Enter file in which to save the key (/home/remote/.ssh/id_rsa):

Created directory '/home/remote/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/remote/.ssh/id_rsa.

Your public key has been saved in /home/remote/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:hgVe/unld9+r/Ynk1HM+t089A41bwxFlPYt5/q+ZyL8 remote@lb1

The key's randomart image is:

+---[RSA 3072]----+

|      . .   .=|

|     . +     oo|

|      . o o.o|

|       o . . o+o.|

|      . S o .oo= |

|       . . o =.o|

|          . .+.=*|

|           .+ooB@|

|            o=EB/|

+----[SHA256]-----+

Now you will have a new directory in the home

Copy the public key to each node using the remote public IP Address:

$ ssh-copy-id 35.166.37.12

/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/remote/.ssh/id_rsa.pub"

/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

remote@35.166.37.12's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '35.166.37.12'"

and check to make sure that only the key(s) you wanted were added.

This command will copy your public key to the remote node in the authorized_keys file, so you will access it using the private one.

Then, try to access them:

$ ssh 35.166.37.12

Make sure you have the SSH traffic allowed in your firewall, and to make it more secure, you should allow it only from a known source (e.g. from 35.166.37.0/24).

For example, if you’re using AWS, you should allow the traffic from 35.166.37.0/24 to the SSH port in this way:

Or if you’re using IPTABLES, you should run something like this:

$ iptables -A INPUT -p tcp -s 35.166.37.0/24 --destination-port 22 -j ACCEPT

Or a similar command if you’re using a different firewall solution.

To make it a bit more secure, we recommend using a different SSH port than the default one, and also could be useful using some tool to ban multiple failed attempts to access it, like fail2ban.

Conclusion

At this point, if everything went fine, you will have SSH communication between your data centers, so the next step is to deploy your PostgreSQL cluster and manage the failover in case of failure, as we will see in the second part of this blog.

ProxySQL has gained a lot of interest right now in the MySQL and MariaDB database world, not to mention ClickHouse which helps make the case for ProxySQL. 

It’s safe to say that ProxySQL has become the default database proxy for the MySQL family of databases (such as Percona Server, Oracle MySQL, Galera Cluster or even with MariaDB). 

ProxySQL is, in fact, an efficient problem solver with extremely rich functionalities that manage database client-server communication; acting as the middleware in a very advanced and performant approach. 

It has made possible the ability to shape the database traffic by delaying, caching, or rewriting queries on the fly. It can also be used to create an environment in which failovers will not affect applications and will be transparent to them. The ProxySQL community is very responsive and constantly builds fixes, patches, and version releases on a timely basis. 

But how performant is your ProxySQL setup, and how can you determine that your setup has been tuned correctly? This blog focuses on determining how performant your ProxySQL nodes are and how to monitor it efficiently.

Common Problems You Can Encounter With ProxySQL

ProxySQL’s default installation comes with a light-weight, simple tuning tool that is able to handle average to heavy load. Although this can depend on the type of queries sent to the middleware, it can impact and start to experience bottlenecks and latency.

Latency Issues

For example, what can lead to latency issues can be hard to determine if you lack a monitoring system. Likewise, you can manually monitor or check the stats schema just like below:

mysql> select * from stats_mysql_connection_pool\G

*************************** 1. row ***************************

        hostgroup: 20

         srv_host: 192.168.10.225

         srv_port: 3306

           status: ONLINE

         ConnUsed: 0

         ConnFree: 0

           ConnOK: 0

          ConnERR: 0

      MaxConnUsed: 0

          Queries: 0

Queries_GTID_sync: 0

  Bytes_data_sent: 0

  Bytes_data_recv: 0

       Latency_us: 1151

*************************** 2. row ***************************

        hostgroup: 20

         srv_host: 192.168.10.226

         srv_port: 3306

           status: ONLINE

         ConnUsed: 0

         ConnFree: 0

           ConnOK: 0

          ConnERR: 0

      MaxConnUsed: 0

          Queries: 0

Queries_GTID_sync: 0

  Bytes_data_sent: 0

  Bytes_data_recv: 0

       Latency_us: 470

*************************** 3. row ***************************

        hostgroup: 10

         srv_host: 192.168.10.227

         srv_port: 3306

           status: ONLINE

         ConnUsed: 0

         ConnFree: 0

           ConnOK: 0

          ConnERR: 0

      MaxConnUsed: 0

          Queries: 0

Queries_GTID_sync: 0

  Bytes_data_sent: 0

  Bytes_data_recv: 0

       Latency_us: 10855

*************************** 4. row ***************************

        hostgroup: 40

         srv_host: 192.168.10.225

         srv_port: 3306

           status: ONLINE

         ConnUsed: 0

         ConnFree: 0

           ConnOK: 0

          ConnERR: 0

      MaxConnUsed: 0

          Queries: 0

Queries_GTID_sync: 0

  Bytes_data_sent: 0

  Bytes_data_recv: 0

       Latency_us: 1151

*************************** 5. row ***************************

        hostgroup: 40

         srv_host: 192.168.10.226

         srv_port: 3306

           status: ONLINE

         ConnUsed: 0

         ConnFree: 0

           ConnOK: 0

          ConnERR: 0

      MaxConnUsed: 0

          Queries: 0

Queries_GTID_sync: 0

  Bytes_data_sent: 0

  Bytes_data_recv: 0

       Latency_us: 470

5 rows in set (0.01 sec)

This allows you to monitor latency based on the hostgroup. But it adds up the hassle unless you have to innovate and develop a script(s) that will manage to notify you.

Client Connection Errors

Maximum connection timeout due to maximum connections in the backend (database node itself) can lead you to perplexity if you are not able to determine what's the main source of the problem. You can check the stats database though to check for such aborted connections in the client or even the server and it's denied connections as follows,

mysql> select * from stats.stats_mysql_global where variable_name like '%connect%';

+-------------------------------------+----------------+

| Variable_Name                       | Variable_Value |

+-------------------------------------+----------------+

| Client_Connections_aborted          | 0 |

| Client_Connections_connected        | 205 |

| Client_Connections_created          | 10067 |

| Server_Connections_aborted          | 44 |

| Server_Connections_connected        | 30 |

| Server_Connections_created          | 14892 |

| Server_Connections_delayed          | 0 |

| Client_Connections_non_idle         | 205 |

| Access_Denied_Max_Connections       | 0 |

| Access_Denied_Max_User_Connections  | 0 |

| MySQL_Monitor_connect_check_OK      | 41350 |

| MySQL_Monitor_connect_check_ERR     | 92 |

| max_connect_timeouts                | 0 |

| Client_Connections_hostgroup_locked | 0              |

| mysql_killed_backend_connections    | 0 |

+-------------------------------------+----------------+

15 rows in set (0.01 sec)

It's also ideal if you can verify and check the backend user's max number of connections to see what are the number of connection limits it can open or use. For example, I have the following in my test,

mysql> select username, active, transaction_persistent, max_connections from mysql_users;

+---------------+--------+------------------------+-----------------+

| username      | active | transaction_persistent | max_connections |

+---------------+--------+------------------------+-----------------+

| proxydemo     | 1 | 1                   | 10000 |

| proxysql-paul | 1      | 1 | 10000           |

+---------------+--------+------------------------+-----------------+

2 rows in set (0.00 sec)

Slow Queries

Identifying the slow queries cannot be that difficult in ProxySQL, but it can be inefficient if done manually. You can check this if doing manual with the variable,

mysql> select * from stats_mysql_global where  variable_name like '%slow%';

+---------------+----------------+

| Variable_Name | Variable_Value |

+---------------+----------------+

| Slow_queries  | 2 |

+---------------+----------------+

1 row in set (0.00 sec)

While that can provide you some numbers, you might check on the table stats_mysql_query_digest under the stats schema if you want to dig deeper. For example below,

mysql> select count_star,sum_time,(sum_time/count_star)/1000 as average_time_ms,digest_text

    -> from stats_mysql_query_digest

    -> where count_star > 100 order by average_time_ms desc limit 10;

+------------+----------+-----------------+--------------------------------------+

| count_star | sum_time | average_time_ms | digest_text                          |

+------------+----------+-----------------+--------------------------------------+

| 884        | 15083961 | 17              | UPDATE sbtest1 SET k=k+? WHERE id=?  |

| 930        | 16000111 | 17              | UPDATE sbtest9 SET k=k+? WHERE id=?  |

| 914        | 15695810 | 17              | UPDATE sbtest4 SET k=k+? WHERE id=?  |

| 874        | 14467420 | 16              | UPDATE sbtest8 SET k=k+? WHERE id=?  |

| 904        | 15294520 | 16              | UPDATE sbtest3 SET k=k+? WHERE id=?  |

| 917        | 15228077 | 16              | UPDATE sbtest6 SET k=k+? WHERE id=?  |

| 907        | 14613238 | 16              | UPDATE sbtest2 SET k=k+? WHERE id=?  |

| 900        | 15113004 | 16              | UPDATE sbtest5 SET k=k+? WHERE id=?  |

| 917        | 15299381 | 16              | UPDATE sbtest7 SET k=k+? WHERE id=?  |

| 883        | 15010119 | 16              | UPDATE sbtest10 SET k=k+? WHERE id=? |

+------------+----------+-----------------+--------------------------------------+

10 rows in set (0.01 sec)

which catches top 10 slow queries based on a sampling by 100. 

Memory Utilization

Hardware items such as CPU, Disk, and Memory have to be monitored to ensure that your ProxySQL is performant. However, the most crucial thing is the memory, as ProxySQL will utilize heavily in the memory due to the query cache mechanism. By default, the query cache, which is dependent on the variable mysql-query_cache_size_MB defaults to 256 Mib. With that regard, it can come to a situation where it uses memory and you need to determine and diagnose if you find issues within your ProxySQL node or even being noticed within the application layer.

When identifying this, you might end up checking the tables in the stats_history and stats schemas. You can see the list of tables which can help you during diagnosis,

mysql> show tables from stats;

| stats_memory_metrics                 |

19 rows in set (0.00 sec)

or,

mysql> show tables from stats_history;

+------------------------+

| tables                 |

+------------------------+

| mysql_connections      |

| mysql_connections_day  |

| mysql_connections_hour |

| mysql_query_cache      |

| mysql_query_cache_day  |

| mysql_query_cache_hour |

| system_cpu             |

| system_cpu_day         |

| system_cpu_hour        |

| system_memory          |

| system_memory_day      |

| system_memory_hour     |

+------------------------+

15 rows in set (0.00 sec)

Efficiently Determining The Performance of your ProxySQL

There are multiple ways to determine the performance of your ProxySQL node. Using ClusterControl offers you the ability to determine this with simple yet straightforward graphs. For example, when ProxySQL is integrated into your cluster, you'll be able to set your query rules, change user's max_connections, determine the top queries, change a user host group, and provide you the performance of your ProxySQL node. See the screenshots below...

All you see is the proof of how proficiently ClusterControl can give you insights of the performance of your ProxySQL node. But this does not limit you to that. ClusterControl also has rich and powerful dashboards we call SCUMM, which includes ProxySQL Overview dashboard. 

If you intend to determine slow queries, you can simply take a glance to the dashboard. Checking your latency distribution over the different hostgroups where your backend nodes are assigned helps you to have a quick insight of the performance based on distribution. You can monitor the client and server connections, providing you query cache insights. Most importantly and not the least, it gives you the memory utilization that ProxySQL node is using. See the graphs below...

These graphs are part of the dashboard which simply helps you to easily determine the performance of your ProxySQL node.

ClusterControl doesn't limit you when dealing with ProxySQL. Also, there's a rich feature here where you can also take a backup or import the configuration which is very important when you are dealing with high-availability for your ProxySQL nodes.

Conclusion

It's never been easier to monitor and determine if you have any issues with your ProxySQL. Like in the example of this blog, we're showcasing ClusterControl as a tool that can provide you efficiency and give you insights to determine the outstanding issues that you are dealing with your performance related problems.

This blog is the second part of Implementing a Multi-Datacenter Setup for PostgreSQL. In this blow, we will show how to deploy PostgreSQL in this type of environment and how to failover in the case of master failure using the ClusterControl auto-recovery feature.

At this point, we will assume you have connectivity between the data centers (as we saw in the first part of this blog) and you have the necessary servers for this task (as we also mentioned in the previous part).

Deploy a PostgreSQL Cluster

We’ll use ClusterControl for this task, so we will assume you have it installed (it could be installed on the same Load Balancer server, but if you can use a different one even better).

Go to your ClusterControl server, and select the option ‘Deploy’. If you already have a PostgreSQL instance running, then you need to select the ‘Import Existing Server/Database’ instead.

When selecting PostgreSQL, you must specify User, Key or Password and port to connect by SSH to our PostgreSQL hosts. You also need the name for your new cluster and if you want ClusterControl to install the corresponding software and configurations for you.

Please check the ClusterControl user requirements for this task here, but if you followed the previous blog, you should use the ‘remote’ user here and the correct SSH port (as we mentioned, is recommended to use a different one if you are using the public IP address to access it instead of a VPN).

After setting up the SSH access information, you must define the database user, version and datadir (optional). You can also specify which repository to use. In the next step, you need to add your servers to the cluster you are going to create.

When adding your servers, you can enter IP or hostname. In this part, you will use the public IP addresses of your servers, and as you can see in the red box, I’m using a different network for the second standby node. ClusterControl doesn’t have any limitations about the network to be used. The only requirement about this is to have SSH access to the node.

So following our previous example, these IP address should be:

Primary Node: 35.166.37.12

Standby 1 Node: 35.166.37.13

Standby 2 Node: 18.197.23.14 (red box)

In the last step, you can choose if your replication will be Synchronous or Asynchronous.

In this case, it is important to use Asynchronous replication for your remote node, if not, your cluster could be affected by the latency or network issues.

You can monitor the status of the creation of your new cluster from the ClusterControl activity monitor.

Once the task is finished, you can see your new PostgreSQL cluster in the main ClusterControl screen.

Adding a PostgreSQL Load Balancer (HAProxy)

Once you have your cluster created, you can perform several tasks on it, like adding a load balancer (HAProxy) or a new replica.

To follow our previous example, let’s add a load balancer that, as we mentioned, it will help you to manage your HA environment. For this, go toClusterControl -> Select PostgreSQL Cluster -> Cluster Actions -> Add Load Balancer.

Here you must add the information that ClusterControl will use to install and configure your HAProxy load balancer. This Load Balancer can be installed in the same ClusterControl server, but if you can use a different one, even better.

The information that you need to introduce is:

Action: Deploy or Import.

Server Address: IP Address for your HAProxy server (It can be the same ClusterControl IP Address).

Listen Port (Read/Write): Port for read/write mode.

Listen Port (Read Only): Port for read only mode.

Policy: It can be:

• leastconn: The server with the lowest number of connections receives the connection.
• roundrobin: Each server is used in turns, according to their weights.
• source: The source IP address is hashed and divided by the total weight of the running servers to designate which server will receive the request.

Install for read/write splitting: For master-slave replication.

Build from Source: You can choose Install from a package manager or build from source.

And you need to select which servers you want to add to the HAProxy configuration.

Also, you can configure Advanced Settings like Admin User, Backend Name, Timeouts, and more.

When you finish the configuration and confirm the deploy, you can follow the progress in the Activity section on ClusterControl UI.

And when this finishes, you can go to ClusterControl -> Nodes -> HAProxy node, and check the current status.

By default, ClusterControl configures HAProxy with two different ports, one for Read-Write, that will be used for the application or user to write (and read) data, and another one for Read-Only, that will be used for balancing the read traffic between all the nodes. In the Read-Write port, only the master node is enabled, and in case of master failure, ClusterControl will promote the most advanced slave to master and will reconfigure this port to disable the old master and enable the new one. In this way, your application can still work in case of a master database failure, as the traffic is redirected by the Load Balancer to the correct node.

You can also monitor your HAProxy servers checking the Dashboard section.

Now, you can improve your HA design adding a new HAProxy node in the remote datacenter and configuring Keepalived service between them. Keepalived will allow you to use a virtual IP address that is assigned to the active Load Balancer node. If this node fails, this virtual IP will be migrated to the secondary HAProxy node, so having this IP configured in your application will allow you to keep everything working in case of a Load Balancer issue.

All this configuration can be performed using ClusterControl.

Conclusion

By following this two-part blog you can implement a multi-datacenter setup for PostgreSQL with High Availability and SSH connectivity between the datacenter, to avoid the complexity of a VPN configuration. 

Using asynchronous replication for the remote node you will avoid any issue related to the latency and network performance, and using ClusterControl you will have automatic (or manual) failover in case of failure (among other several features). This could be the simplest way to reach this topology and we hope this would be useful for you.

Are you an SysAdmin who is now responsible for your companies database operations? Then this is the webinar for you. Learn from a Senior DBA the basics you need to know to keep things up-and-running and how automation can help.

There is no need to register for this webinar, just click the link when it's time to start and enter Access Code: 364-606-317. If you can't make it, don't worry... We will record the session and publish the replay at a later time.

If you need a dial-in number for this webinar, they are listed below...

Australia: +61 2 8355 1038
Austria: +43 7 2081 5337
Belgium: +32 28 93 7002
Brazil: +55 11 4118-4898
Bulgaria: +359 2 906 0606
Canada: +1 (647) 497-9373
Chile: +56 2 3214 9681
Colombia: +57 1 600 9954
Czech Republic: +420 2 96 21 62 28
Denmark: +45 32 72 03 69
Finland: +358 942 72 0972
France: +33 170 950 590
Germany: +49 692 5736 7300
Greece: +30 21 0 300 2693
Hungary: +36 1 933 3700
Ireland: +353 15 360 756
Israel: +972 3 376 3071
Italy: +39 0 230 57 81 80
Luxembourg: +352 34 2080 9220
Malaysia: +60 3 7724 4060
Mexico: +52 55 4624 4518
Netherlands: +31 207 941 375
New Zealand: +64 9 282 9510
Norway: +47 21 93 37 37
Panama: +507 308 4337
Peru: +51 1 642 9425
Romania: +40 31 780 1159
South Africa: +27 11 259 4925
Spain: +34 932 75 1230
Sweden: +46 853 527 818
Switzerland: +41 225 4599 60
Turkey: +90 212 900 4812
United Kingdom: +44 330 221 0097
 

Are you an SysAdmin who is now responsible for your companies database operations? Then this is the webinar for you. Learn from a Senior DBA the basics you need to know to keep things up-and-running and how automation can help.

Join us April 22nd at 9AM CST (4PM CEST)

There is no need to register for this webinar, just click the link when it's time to start and enter Access Code: 364-606-317. If you can't make it, don't worry... We will record the session and publish the replay at a later time.

If you need a dial-in number for this webinar, they are listed below...

Australia: +61 2 8355 1038
Austria: +43 7 2081 5337
Belgium: +32 28 93 7002
Brazil: +55 11 4118-4898
Bulgaria: +359 2 906 0606
Canada: +1 (647) 497-9373
Chile: +56 2 3214 9681
Colombia: +57 1 600 9954
Czech Republic: +420 2 96 21 62 28
Denmark: +45 32 72 03 69
Finland: +358 942 72 0972
France: +33 170 950 590
Germany: +49 692 5736 7300
Greece: +30 21 0 300 2693
Hungary: +36 1 933 3700
Ireland: +353 15 360 756
Israel: +972 3 376 3071
Italy: +39 0 230 57 81 80
Luxembourg: +352 34 2080 9220
Malaysia: +60 3 7724 4060
Mexico: +52 55 4624 4518
Netherlands: +31 207 941 375
New Zealand: +64 9 282 9510
Norway: +47 21 93 37 37
Panama: +507 308 4337
Peru: +51 1 642 9425
Romania: +40 31 780 1159
South Africa: +27 11 259 4925
Spain: +34 932 75 1230
Sweden: +46 853 527 818
Switzerland: +41 225 4599 60
Turkey: +90 212 900 4812
United Kingdom: +44 330 221 0097
 

Agenda

• DB Monitoring Basics
  • Keeping tabs on CPU, Memory, Network, & Disk Space
  • DB Specific Metrics Basics
• DB Backup Basics
  • Checking Backup Status
  • Backup Verification
• DB Failover Basics
  • Getting DB Alerts
  • Understanding the Failover Process
  • Recovering Failed Nodes
• DB User Management Basics
  • Creating New Users
  • Managing Privileges
• DB Security Basics
  • Security Patches
• DB Reporting
  • Doing Regular Health Checks
• DB Automation Overview (Demo)

MongoDB is a NoSQL database that supports a wide variety of input dataset sources. It is able to store data in flexible JSON-like documents, meaning fields or metadata can vary from document to document and data structure can be changed over time. The document model makes the data easy to work with by mapping to the objects in the application code. MongoDB is also known as a distributed database at its core, so high availability, horizontal scaling, and geographic distribution are built-in and easy to use. It comes with the ability to seamlessly modify parameters for model training. Data Scientists can easily merge the structuring of data with this model generation.

What is Machine Learning?

Machine Learning is the science of getting the computers to learn and act like humans do and improve their learning over time in autonomous fashion. The process of learning begins with observations or data, such as examples, direct experience, or instruction, in order to look for patterns in data and make better decisions in the future based on the examples that we provide. The primary aim is to allow the computers to learn automatically without human intervention or assistance and adjust actions accordingly.

A Rich Programming and Query Model

MongoDB offers both native drivers and certified connectors for developers and data scientists building machine learning models with data from MongoDB. PyMongo is a great library to embed MongoDB syntax into Python code. We can import all the functions and methods of MongoDB to use them in our machine learning code. It is a great technique to get multi-language functionality in a single code. The additional advantage is that you can use the essential features of those programming languages to create an efficient application.

The MongoDB query language with rich secondary indexes enables developers to build applications that can query and analyze the data in multiple dimensions. Data can be accessed by single keys, ranges, text search, graph, and geospatial queries through complex aggregations and MapReduce jobs, returning responses in milliseconds.

To parallelize data processing across a distributed database cluster, MongoDB provides the aggregation pipeline and MapReduce. The MongoDB aggregation pipeline is modelled along the concept of data processing pipelines. Documents enter a multi-stage pipeline that transforms the documents into an aggregated result using native operations executed within MongoDB. The most basic pipeline stages provide filters that operate like queries, and document transformations that modify the form of the output document. Other pipeline operations provide tools for grouping and sorting documents by specific fields as well as tools for aggregating the contents of arrays, including arrays of documents. In addition, pipseline stages can use operators for tasks such as calculating the average or standard deviations across collections of documents, and manipulating strings. MongoDB also provides native MapReduce operations within the database, using custom JavaScript functions to perform the map and reduce stages.

In addition to its native query framework, MongoDB also offers a high performance connector for Apache Spark. The connector exposes all of Spark's libraries, including Python, R, Scala, and Java. MongoDB data is materialized as DataFrames and Datasets for analysis with machine learning, graph, streaming, and SQL APIs.

The MongoDB Connector for Apache Spark can take advantage of MongoDB's aggregation pipeline and secondary indexes to extract, filter, and process only the range of data it needs - for example, analysing all customers located in a specific geography. This is very different from simple NoSQL datastores that do not support either secondary indexes or in-database aggregations. In these cases, Spark would need to extract all data based on a simple primary key, even if only a subset of that data is required for the Spark process. This means more processing overhead, more hardware, and longer time-to-insight for data scientists and engineers. To maximize performance across large, distributed data sets, the MongoDB Connector for Apache Spark can co-locate Resilient Distributed Datasets (RDDs) with the source MongoDB node, thereby minimizing data movement across the cluster and reducing latency.

Performance, Scalability & Redundancy

Model training time can be reduced by building the machine learning platform on top of a performant and scalable database layer. MongoDB offers a number of innovations to maximize throughput and minimize latency of machine learning workloads:

• WiredTiger is known as the default storage engine for MongoDB, developed by the architects of Berkeley DB, the most widely deployed embedded data management software in the world. WiredTiger scales on modern, multi-core architectures. Using a variety of programming techniques such as hazard pointers, lock-free algorithms, fast latching and message passing, WiredTiger maximizes computational work per CPU core and clock cycle. To minimize on-disk overhead and I/O, WiredTiger uses compact file formats and storage compression.
• For the most latency-sensitive machine learning applications, MongoDB can be configured with the In-Memory storage engine. Based on WiredTiger, this storage engine gives users the benefits of in-memory computing, without trading away the rich query flexibility, real-time analytics, and scalable capacity offered by conventional disk-based databases.
• To parallelize model training and scale input datasets beyond a single node, MongoDB uses a technique called sharding, which distributes processing and data across clusters of commodity hardware. MongoDB sharding is fully elastic, automatically rebalancing data across the cluster as the input dataset grows, or as nodes are added and removed.
• Within a MongoDB cluster, data from each shard is automatically distributed to multiple replicas hosted on separate nodes. MongoDB replica sets provide redundancy to recover training data in the event of a failure, reducing the overhead of checkpointing.

MongoDB’s Tunable Consistency

MongoDB is strongly consistent by default, enabling machine learning applications to immediately read what has been written to the database, thus avoiding the developer complexity imposed by eventually consistent systems. Strong consistency will provide the most accurate results for machine learning algorithms; however, in some scenarios it is acceptable to trade consistency against specific performance goals by distributing queries across a cluster of MongoDB secondary replica set members.

Flexible Data Model in MongoDB

MongoDB's document data model makes it easy for developers and data scientists to store and aggregate data of any form of structure inside the database, without giving up sophisticated validation rules to govern data quality. The schema can be dynamically modified without an application or database downtime that results from costly schema modifications or redesign incurred by relational database systems.

Saving models in a database and loading them, using python, is also an easy and much-required method. Choosing MongoDB is also an advantage as it is an open-source document database and also a leading NoSQL database. MongoDB also serves as a connector for apache spark distributed framework.

MongoDB’s Dynamic Nature

MongoDB’s dynamic nature enables its usage in database manipulation tasks in developing Machine Learning applications. It is a very efficient and easy way to carry out an analysis of datasets and databases. The output of the analysis can be used in training machine learning models. It has been recommended that data analysts and Machine Learning programmers gain mastery in MongoDB and apply it in many different applications. MongoDB’s Aggregation framework is used for data science workflow for performing data analysis for numerous applications.

Conclusion

MongoDB offers several different capabilities such as: flexible data model, rich programming, data model, query model and its tunable consistency that make training and using machine learning algorithms much easier than with traditional, relational databases. Running MongoDB as the backend database will enable storing and enriching machine learning data allows for persistence and increased efficiency.

In the previous blog, we have covered some tips and tricks to prepare a MySQL server for production usage from a system administrator perspective. This blog post is the continuation... 

Use a Database Backup Tool

Every backup tool has its own advantages and disadvantages. For example, Percona Xtrabackup (or MariaDB Backup for MariaDB) can perform a physical hot-backup without locking the databases but it can only be restored to the same version on another instance. While for mysqldump, it is cross compatible with other MySQL major versions and way simpler for partial backup, albeit it is relatively slower during restoration if compared to Percona Xtrabackup on big databases. MySQL 5.7 also introduces mysqlpump, similar to mysqldump with parallel processing capabilities to speed up the dump process.

Do not miss to configure all of these backup tools in your MySQL server as they are freely available and very critical for data recovery. Since mysqldump and mysqlpump are already included in MySQL 5.7 and later, we just need to install Percona Xtrabackup (or MariaDB Backup for MariaDB) but it requires some preparations, as shown in the following steps:

Step One

Make sure the backup tool and its dependencies are installed:

$ yum install -y epel-release

$ yum install -y socat pv percona-xtrabackup

For MariaDB servers, use MariaDB Backup instead:

$ yum install -y socat pv MariaDB-Backup

Step Two

Create user 'xtrabackup' on master if it doesn't exist:

mysql> CREATE USER 'xtrabackup'@'localhost' IDENTIFIED BY 'Km4z9^sT2X';

mysql> GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO 'xtrabackup'@'localhost';

Step Three

Create another user called 'mysqldump' on master if it doesn't exist. This user will be used for 'mysqldump' and 'mysqlpump':

mysql> CREATE USER 'mysqldump'@'localhost' IDENTIFIED BY 'Km4z9^sT2X';

mysql> GRANT SELECT, SHOW VIEW, EVENT, TRIGGER, LOCK TABLES, RELOAD, REPLICATION CLIENT ON *.* TO 'mysqldump'@'localhost';

Step Four

Add the backup users' credentials inside MySQL configuration file under [xtrabackup], [mysqldump] and [mysqlpump] directive:

$ cat /etc/my.cnf

...

[xtrabackup]

user=xtrabackup

password='Km4z9^sT2X'



[mysqldump]

user=mysqldump

password='Km4z9^sT2X'



[mysqlpump]

user=mysqldump

password='Km4z9^sT2X'

By specifying the above lines, we don't need to specify username and password in the backup command since the backup tool will automatically load those configuration options from the main configuration file.

Make sure the backup tools are properly tested beforehand. For Xtrabackup which supports backup streaming via network, this has to be tested first to make sure the communication link can be established correctly between the source and destination server. On the destination server, run the following command for socat to listen to port 9999 and ready to accept incoming streaming:

$ socat -u tcp-listen:9999,reuseaddr stdout 2>/tmp/netcat.log | xbstream -x -C /var/lib/mysql

Then, create a backup on the source server and stream it to port 9999 on the destination server:

$ innobackupex --socket=/var/lib/mysql/mysql.sock --stream=xbstream /var/lib/mysql/ | socat - TCP4:192.168.0.202:9999

You should get a continuous stream of output after executing the backup command. Wait until you see the 'Completed OK' line indicating a successful backup.

With pv, we can throttle the bandwidth usage or see the progress as a process being piped through it. Commonly, the streaming process will saturate the network if no throttleting is enabled and this could cause problems with other servers to interact with another in the same segment. Using pv, we can throttle the streaming process before we pass it to the streaming tool like socat or netcat. The following example shows the backup streaming will be throttled around 80 MB/s for both incoming and outgoing connections:

$ innobackupex --slave-info --socket=/var/lib/mysql/mysql.sock --stream=xbstream /var/lib/mysql/ | pv -q -L 80m | socat - TCP4:192.168.0.202:9999

Streaming a backup is commonly used to stage a slave or store the backup remotely on another server.

For mysqldump and mysqlpump, we can test with the following commands:

$ mysqldump --set-gtid-purged=OFF --all-databases

$ mysqlpump --set-gtid-purged=OFF --all-databases

Make sure you see non-error lines appear in the output.

Stress Test the Server

Stress testing the database server is important to understand the maximum capacity that we can anticipate for the particular server. This will become useful when you are approaching thresholds or bottlenecks at a later stage. You can use many benchmarking tools available in the market like mysqlslap, DBT2 and sysbench. 

In this example, we use sysbench to measure the server's peak performance, saturation level and also the components' temperature while running in a high database workload environment. This will give you a ground understanding on how good the server is, and anticipate the workload that the server can process for our application in production.

To install and configure sysbench, you can compile it from the source or install the package from Percona repository:

$ yum install -y https://repo.percona.com/yum/percona-release-latest.noarch.rpm

$ yum install -y sysbench

Create the database schema and user on the MySQL server:

mysql> CREATE DATABASE sbtest;

mysql> CREATE USER 'sbtest'@'localhost' IDENTIFIED BY 'sysbenchP4ss';

mysql> GRANT ALL PRIVILEGES ON sbtest.* TO sbtest@'localhost';

Generate the test data:

$ sysbench \

/usr/share/sysbench/oltp_common.lua \

--db-driver=mysql \

--mysql-host=localhost \

--mysql-user=sbtest \

--mysql-password=sysbenchP4ss \

--tables=50 \

--table-size=100000 \

prepare

Then run the benchmark for 1 hour (3600 seconds):

$ sysbench \

/usr/share/sysbench/oltp_read_write.lua \

--report-interval=2 \

--threads=64 \

--max-requests=0 \

--db-driver=mysql \

--time=3600 \

--db-ps-mode=disable \

--mysql-host=localhost \

--mysql-user=sbtest \

--mysql-password=sysbenchP4ss \

--tables=50 \

--table-size=100000 \

run

While the test is running, use iostat (available in sysstat package) in another terminal to monitor the disk utilization, bandwidth, IOPS and I/O wait:

$ yum install -y sysstat

$ iostat -x 60

avg-cpu:  %user %nice %system %iowait  %steal %idle

          40.55    0.00 55.27    4.18 0.00 0.00



Device:         rrqm/s wrqm/s     r/s w/s rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await svctm  %util

sda               0.19 6.18 1236.23  816.92 61283.83 14112.44    73.44 4.00 1.96 2.83    0.65 0.34 69.29

The above result will be printed every 60 seconds. Wait until the test finishes and take the average of r/s (reads/second), w/s (writes/second), %iowait, %util, rkB/s and wkB/s (bandwidth). If you are seeing a relatively low utilization of disk, CPU, RAM or network, you probably need to increase the "--threads" value to an even higher number so it will make use of all the resources to the limit.

Consider the following aspects to be measured of:

• Queries per Second = Sysbench summary once the test completes under SQL statistics -> Queries -> Per sec.
• Query latency = Sysbench summary once the test completes under Latency (ms) -> 95th percentile.
• Disk IOPS = Average of r/s + w/s
• Disk utilization = Average of %util
• Disk bandwidth R/W = Average of rkB/s / Average of wkB/s
• Disk IO wait = Average of %iowait
• Average server load = Average load average as reported by top command.
• MySQL CPU usage = Average CPU utilization as reported by top command.

With ClusterControl, you can easily observe and get the above information via Nodes Overview panel, as shown in the following screenshot:

Furthermore, the information gathered during the stress test can be used to tune MySQL and InnoDB variables accordingly like innodb_buffer_pool_size, innodb_io_capacity, innodb_io_capacity_max, innodb_write_io_threads, innodb_read_io_threads and also max_connections.

To learn more about MySQL performance benchmark using sysbench, check out this blog post, How to Benchmark Performance of MySQL & MariaDB Using SysBench.

Use an Online Schema Change Tool

Schema change is something that is inevitable in relational databases. As the application grows and becomes more demanding over time, it does certainly require some structure change to the database. There are some DDL operations that will rebuild the table thus blocking other DML statements to run and this could impact your database availability if you are performing structural changes on a huge table. To see the list of blocking DDL operations, check out this MySQL documentation page and look for operations that have "Permits Concurrent DML" = No.

If you can't afford downtime on the production servers when performing schema change, it's probably a good idea to configure the online schema change tool at the early stage. In this example, we install and configure gh-ost, an online schema change built by Github. Gh-ost uses the binary log stream to capture table changes and asynchronously applies them onto the ghost table.

To install gh-ost on a CentOS box, simply follow the following steps: 

Step One

 Download latest gh-ost from here: 

$ wget https://github.com/github/gh-ost/releases/download/v1.0.48/gh-ost-1.0.48-1.x86_64.rpm

Step Two

Install the package:

$ yum localinstall gh-ost-1.0.48-1.x86_64.rpm 

Step Three

Create a database user for gh-ost if it does not exist, and grant it with proper privileges:

mysql> CREATE USER 'gh-ost'@'{host}' IDENTIFIED BY 'ghostP455';

mysql> GRANT ALTER, CREATE, DELETE, DROP, INDEX, INSERT, LOCK TABLES, SELECT, TRIGGER, UPDATE ON {db_name}.* TO 'gh-ost'@'{host}';

mysql> GRANT SUPER, REPLICATION SLAVE ON *.* TO 'gh-ost'@'{host}';

** Replace the {host} and {db_name} with their appropriate values. Ideally, the {host} is one of the slave hosts that will perform the online schema change. Refer to gh-ost documentation for details.

Step Four

Create gh-ost configuration file to store the username and password under /root/.gh-ost.cnf:

[client]

user=gh-ost

password=ghostP455

Similarly, you can have Percona Toolkit Online Schema Change (pt-osc) configured on the database server. The idea is to make sure you are prepared with this tool first on the database server that is likely to be running this operation in the future.

Utilize the Percona Toolkit

Percona Toolkit is a collection of advanced open source command-line tools, developed by Percona, that are engineered to perform a variety of MySQL, MongoDB and PostgreSQL server and system tasks that are too difficult or complex to perform manually. These tools have become the ultimate saviour, used by DBAs around the world to address or solve technical issues found in MySQL and MariaDB servers.

To install Percona Toolkit, simply run the following command:

$ yum install https://repo.percona.com/yum/percona-release-latest.noarch.rpm

$ yum install percona-toolkit

There are over 30 tools available within this package. Some of them are specifically designed for MongoDB and PostgreSQL. Some of the most popular tools for MySQL troubleshooting and performance tuning are pt-stalk, pt-mysql-summary, pt-query-digest, pt-table-checksum, pt-table-sync and pt-archiver. This toolkit can help DBAs to verify MySQL replication integrity by checking master and replica data consistency, efficiently archive rows, find duplicate indexes, analyze MySQL queries from logs and tcpdump and much more.

The following example shows one of the tools (pt-table-checksum) output where it can perform online replication consistency check by executing checksum queries on the master, which produces different results on replicas that are inconsistent with the master:

$ pt-table-checksum --no-check-binlog-format --replicate-check-only

Checking if all tables can be checksummed ...

Starting checksum ...

Differences on mysql2.local

TABLE CHUNK CNT_DIFF CRC_DIFF CHUNK_INDEX LOWER_BOUNDARY UPPER_BOUNDARY

mysql.proc 1 0 1

mysql.tables_priv 1 0 1

mysql.user 1 1 1

The above output shows that there are 3 tables on the slave (mysql2.local) which are inconsistent with the master. We can then use the pt-table-sync tool to patch up the missing data from the master, or simply resyncing the slave once more.

Lock Down the Server

Finally, after the configuration and preparation stage is complete, we can isolate the database node from the public network and restrict the server access to known hosts and networks. You can use firewall (iptables, firewalld, ufw), security groups, hosts.allow and/or hosts.deny or simply disable the network interface that faces the internet if you have multiple network interfaces.

For iptables, it's important to specify a comment for every rule using the '-m comment --comment' flag:

$ iptables -A INPUT -p tcp -s 192.168.0.0/24 --dport 22 -m comment --comment 'Allow local net to SSH port' -j ACCEPT

$ iptables -A INPUT -p tcp -s 192.168.0.0/24 --dport 3306 -m comment --comment 'Allow local net to MySQL port' -j ACCEPT

$ iptables -A INPUT -p tcp -s 192.168.0.0/24 --dport 9999 -m comment --comment 'Allow local net to backup streaming port' -j ACCEPT

$ iptables -A INPUT -p tcp -s 0.0.0.0/0 -m comment --comment 'Drop everything apart from the above' -j DROP

Similarly for Ubuntu's Firewall (ufw), we need to define the default rule first and then can create a similar rules for MySQL/MariaDB similar to this:

$ sudo ufw default deny incoming comment 'Drop everything apart from the above'

$ sudo ufw default allow outgoing comment 'Allow outgoing everything'

$ sudo ufw allow from 192.168.0.0/24 to any port 22 comment 'Allow local net to SSH port'

$ sudo ufw allow from 192.168.0.0/24 to any port 3306 comment 'Allow local net to MySQL port'

$ sudo ufw allow from 192.168.0.0/24 to any port 9999 comment 'Allow local net to backup streaming port'

Enable the firewall:

$ ufw enable

Then, verify the rules are loaded correctly:

$ ufw status verbose

Status: active

Logging: on (low)

Default: deny (incoming), allow (outgoing), disabled (routed)

New profiles: skip



To                         Action From

--                         ------ ----

22                         ALLOW IN 192.168.0.0/24             # Allow local net to SSH port

3306                       ALLOW IN 192.168.0.0/24             # Allow local net to MySQL port

9999                       ALLOW IN 192.168.0.0/24             # Allow local net to backup streaming port

Again, it's very important to specify comments on every rule to help us understand the rule better.

For remote database access restriction, we can also use VPN server as shown in this blog post, Using OpenVPN to Secure Access to Your Database Cluster in the Cloud. 

Conclusion

Preparing a production server is obviously not an easy task, which we have shown in this blog series. If you are worried that you would screw up, why don't you use ClusterControl to deploy your database cluster? ClusterControl has a very good track record in database deployment and has enabled more than 70,000 MySQL and MariaDB deployments for all environments to date. 

The day happened when your database administrator did not show up on the daily online standup. Shortly after you have learned that he’ll be unavailable for an unknown period of time and it is you, who will have to step in and replace him. In a series of posts we would like to explain what are the most important areas you may have to take care of while your colleague will be off. Let’s take a look at security, one of the most important pieces of puzzle in a database environment.

What’s good, most likely you are pretty much covered. Your colleague should have secured the environment and, without human intervention, the environment should stay secure. Here’s a list of things you would like to check.

Checking Passwords

It’s quite important to ensure that all of the users have a proper password defined. You do that in a couple of ways. First, you can just check in the database. For MySQL you are looking for a query like this:

mysql> SELECT host, user FROM mysql.user where authentication_string='';

+------+-----------+

| host | user      |

+------+-----------+

| %    | emptypass |

+------+-----------+

1 row in set (0.00 sec)

For PostgreSQL you may want to review your pg_hba.conf file and verify that all of the entries have an ‘md5’ authentication method.

It is quite important to ensure that the access to your database is secured.

Checking Access

All existing servers should have already been secured but it might be that you will be asked to provision new instances. If you can reuse existing scripts, playbooks or other automation tools, you would probably be good. If you’d have to provision a new server by hand, though, please keep in mind that the access to such a server should be secured. Depending on the policies in your company this may mean different things but here’s a list of steps you can take to protect your database:

1. Do not bind to a public IP. Your database should not be exposed to the open world. Always use a private IP to bind to.
2. Make sure you have a proper network security in place. Firewalls, ACL’s etc - block everything, allow access only to the ports you have to access.
3. Do not use default, “official” ports. Instead of running SSH on port 22, run it on 2345. Instead of running your MySQL database on port 3306 use 3333 or any other port that you have available. It is security through obscurity, sure, but you’ll be amazed how far it can get you.

Security Patches

In a production environment you should do your best to stay on top of the security updates. Please keep in mind that it is not only a matter of database security fixes but also other packages that are installed in the system. Any service that runs on an exposed port has to be kept up to date because, if exploited, it could become an entry point to your infrastructure. If you do not have an automated patching process in place, spend some time and try to stay up to date with any security updates your distribution of choice may be publishing.

How ClusterControl Can Help With the Security of Your Database?

ClusterControl comes with a couple of features that may help you ensure your database is secured. First of all, it comes with user management. You can create new or edit existing users.

This can be used to change insecure passwords or create new passwords if the user does not have a password defined. You can also quickly check which users do have extensive privileges and, if needed, reduce them to the minimum required to perform their tasks.

Additional feature is the list of inactive users.

ClusterControl can prepare a list of accounts that have not been used to access MySQL database since its restart. This may help you to understand which accounts are inactive and, as a result, remove them to reduce the potential access paths to your database.

Another very useful feature that will help you to stay on top of the security updates are the Operational Reports. It comes with, among others, a Package Upgrade Report.

Such reports can be scheduled and executed on a regular basis. It may be delivered to one or more recipients via email. 

It creates a list of packages installed on the system, checks if the package is in the latest version and if not, you will have a notification in the report. Executing such reports on a regular basis helps to stay on top of the upgrades and keep your environment secure.

In case you would notice that the database packages have available updates, you can use another feature of ClusterControl to perform a minor version upgrade:

With a couple of clicks you can easily perform an upgrade of your MySQL or MariaDB database to the latest version, ensuring that you are protected from known security vulnerabilities.

As you can see, keeping the system secure requires several actions to be taken and it does require regular checks to stay on top of the updates. With database management platforms like ClusterControl, it becomes much easier to accomplish.

The alarming nature of COVID-19 is being felt globally. Many companies have closed or filed bankruptcy because of the global pandemic. 

Most organizations did not see this coming. Large companies have a better chance of being well prepared for this type of scenario for which a Disaster Recovery Plan (DRP) has been laid-out. At Severalnines, we advocate the best practices consisting of conventional approaches when dealing with crisis, especially in securing your database technology. 

When things go south and nothing has been prepared (or worse, discussed but never implemented) the end result can be a hard blow to the company. These types of scenarios can’t be avoided, only planned for.

Today’s blog will walk you through some things to consider in your organization in regards to process planning and database backups for these types of scenarios.

Always Document The Work

Whenever your engineers start working on projects, it's best to always document the work such as the layout, project specification, requirements, and it's underlying procedures required to run the playbooks/runbooks. The documentation must also cover a proper escalation when things go wrong and what actions are required to be taken.

It can also be ideal to host these documents accessible via a centralized environment or intranet (accessed via connecting to VPN if done remotely), or over the internet as long as it is stored securely and can be accessed with multiple layers such as using 2FA (Two-Factor Authentication) or MFA (Multi-Factor Authentication). It's purpose is to have these documents accessible wherever the engineer is located. So when specific actions have to be done remotely, it can be done and not to physically access this via the office. Otherwise, it can lead to difficulties and inconvenience to the engineer's perspective. 

No Documentation, No Wiki - What Shall I Do?

This scenario is a pure pain and is most likely being experienced at companies around the world right now. When things go awry, it's best to start investigating the tools being used and lay out the procedures that have to be done. Worse case scenario, you might end up re-creating the whole thing or some part of the specific actions that are led to unresolved investigation will be reset. 

If this happens it may make sense to reach out for some temporary database support. This way you will have experts in the technology available to talk to.

Best case for this scenario is to leverage on specific technologies that offer a full-stack solution which can offer you backup solutions, cluster management, monitoring tools, and support. For example, here at Severalnines, we recommend that you try our software ClusterControl.

Try to Automate the Work

Writing scripts for automation is ideal. You can leverage automation tools to manage things for you, such as Chef, Puppet, Ansible, Salt, or Terraform. In our previous blogs, we shared some how-to's you can use for these automation tools. Take a look at How to Automate Daily DevOps Database Tasks with Chef or Database Automation with Puppet: Deploying MySQL & MariaDB Galera Cluster for more info. 

Automating the workload can relieve situations during a DBA. As stated earlier, it's best that procedures are written in a knowledge-based platform which is accessible anywhere.

Although creating this automation can be difficult for some organizations as a limited number of engineers can handle this type of work. Your best bet is to leverage tools or solutions that offer automation without high levels of technical complexity. 

For example, if your database is hosted in AWS or are hosted in a managed services platform, there are big advantages. Your team can deploy a backup schedule, set the backup policy, and set its retention; then just hit submit. Whenever your database crashes (for example in AWS Aurora) it will be handled in the background with autorecovery dispatched without the end users knowing that something happened. From a  business perspective, the end result is business-as-usual and no downtime occurring.

Although that's a great option to take, it might not be an appropriate choice for your technology stack. There are certain solutions that can offer you automation to handle backups for you. For example, you can take advantage of Backup Ninja, or even use ClusterControl to manage backup and restore. You can also set schedules for your backup with support for various open-source databases.

Keep Documentation Up-To-Date

In my experience, this is a problem that you will most likely encounter when things go south. When changes have been made, but go undocumented, then things can go wrong when performing vital tasks. This can get you into a dilemma, because the result comes out different than what was documented. 

This is scary in situations like when you are in a deadlock mode (where you're in the middle of performing a backup, causing your production to be locked up). Afterwards, you are then informed (after a day) that the process has been changed, but the lack of documentation by the DBA has left damage to your database nodes and caused problems.

When it comes to changes and documentation process, there's a huge advantage when it comes to availing third party solutions. These third-party companies know how to handle the business and always kept the process in-line so with changes to the software. 

For example, in ClusterControl (MySQL database systems) you can choose a backup method using mysqldump or using Percona Xtrabackup/Mariabackup (for MariaDB). Customers that are unsure of what to choose have the liberty to contact support to discuss. 

ClusterControl is also a good example here as it extends as your virtual DBA that mostly does things that your company’s DBA can do. There are certain actions that only DBA's can understand or oversee the result, but having an extended DBA that you are sure of 24/7 is available is a perfect route to avoid business from being impacted.

Using Third Party Applications

There are multiple options to choose from nowadays. These types of scenarios are not new, especially to companies who have invested in security and assurance for their business to continue to flourish and avoid huge downtimes due to lack of skills or engineers that can do the specific task. 

Relying on third party applications that offer managed databases (including backup solutions) or availing SaaS to handle backup automation and restoration are very advantageous in these types of scenarios. This comes up with a price, though, so you should really avail yourself on what offers the functions that your company requires.

Using a fully-managed database service such as Amazon RDS helps you easily do this with ease. Although the backups are stored as snapshots, yet if you need more flexibility such as restoring a particular table is not supported. It is doable, but requires knowledge and skills on what tools or particular commands need to be invoked.

In the case of ClusterControl, you have easy-to-use options when managing your backups. System administrators won't have to worry about the technology and commands to use as it's already managed by the software. For example, take a look at the screenshot below,

you can create backup, or schedule backup by through user clicks.

You can also view the scheduled backups with the list of backups that had proceed successfully,

These are all done with a couple of clicks with no scripting or extra engineering work that has to be done. Backups can also be verified automatically, restored with PITR support, uploaded to the cloud, compressed, and encrypted to comply with security and regulatory requirements. 

Not only do these features allow you to manage backups with ease, it also enables you to be notified real-time with various integrations supported, either through e-mail or by using third-party notifications such as Slack, PagerDuty, Servicenow, and a lot more.

Conclusion

These rough times we are experiencing shows that many businesses need to change how to approach the possibility of your technical resources not being available to do their jobs. This elevates how seriously to consider Disaster Recovery Planning and how well prepared your organization is when a crisis hits.

Database monitoring is a key function for Database Administrators (DBA). With a proper monitoring platform, you can know immediately if there is something wrong with the system. 

Should your DBA be unavailable and you may need to take over the platform to monitor and handle non-critical issues. In this blog are tips that you need to know if you are monitoring your database with tips for troubleshooting any issues.

Database Monitoring Documentation

It is always good to have proper documentation about Database Monitoring. This information should contain information like what tool is being used, what metrics database are being monitored, and what operating system metrics are being monitored. DBAs must prepare the documentation with restricted access to the document. The document must describe information how-to access the database and provide credentials to access the monitoring tools.

How ClusterControl Monitoring Tools Can Help

There are various monitoring tools for databases, providing metrics related to the operating system; database memory, threads, locking, session, slow query. ClusterControl provides a great monitoring tool for your database that is free to use. You can enable agent-based Monitoring in your ClusterControl as shown below 

You just need to click agent-based monitoring in the dashboard tab, it will install prometheus, and the exporter for database and the operating system. After installation, it will appear the monitoring tool in your dashboard as shown below

The System Overview will show you basic monitoring usage for operating systems such as CPU Usage, Disk Space Usage and Memory Usage. On the CPU Usage, you will see utilization for User, Idle, System, Iowait, while in the Memory Usage it is monitoring related to Total Memory, Available, and Used. Disk Space usage monitored the disk space utilization on each device. Beside the System Overview, there are also MySQL Server Overview which have information about database status and utilization as shown below.

There is a lot of information shown in MySQL Server Overview such as MySQL Uptime, current running Queries, Connection Used, and Bufferpool Hit Ratio. There is a Handler Stats metric to show your current handler for read first, read last, read next, delete, update and insert. DB Connection monitors your current connection, from aborted connections, aborted clients, max connections, thread connected, to threads connected. MySQL InnoDB Metrics show you current InnoDB status information as shown below :

You can see the InnoDB BufferPool Activity from Created, Read, and Written. InnoDB Bufferpool Hit Ratio, and InnoDB Bufferpool Usage from Dirty page, Read from Disk, Bufferpool Size. 

Worse Case Scenario 

The worst case scenario if DBA does not implemented the Database Monitoring Tool yet, as a SysAdmin, you can go through command line with some utility tools in the Operating System,  as shown below :

• CPU Utilization
  • You can use top or htop command to see the running process in the operating system that eats your CPU.
• Memory Utilization
  • vmstat can help you check your memory utilization. Also, you can check your current memory usage using free command,  it will show you the swap utilization (if it was used), total memory, used memory, and free memory.
• Disk Utilization
  • iostat command can show you the read and write rate to the disk per second. While iotop shows you the percentage of IO Utilization within the services.

If you have access to the database with superuser privileges, you can do the basic monitoring for database with below command :

• show [full] processlist, it will show you the thread with running operation in database, user connected, ip address, database, and Time.
• show engine innodb status \G, is a command to monitor your current InnoDB information from row operations, buffer pool and memory usage, IO thread usage, transaction and locks.

Conclusion

Having proper documentation for databases and monitoring tools is a must for a DBA. It will be useful and helpful for others, like the SysAdmins when they need to run an investigation while the DBA is unavailable. 

SysAdmins also needs to understand a basic troubleshooting process that enables them to troubleshoot some issues and elaborate the root cause of issue when escalating it to the related party, ie. DBA.

In systems engineering, communication is a key element to achieving success on any project. This is because it’s critical to the entire cycle of development; starting from collecting requirements to delivering a minimum viable product.

Using a distributed version control system (like Git - an industry standard), developers can submit small pieces of code and work together with services like Bitbucket built around it. Hosting Bitbucket is possible when the data generated by its users have a database to stay, just like PostgreSQL, but integrating both requires additional configurations for executing in different machines.

Networking Overview

A local area network can pass the information between the programs without the need of exposing it to the outside network, depending on where the users would be.

1.1. Local area network (LAN).

With the separation of concerns, Bitbucket and PostgreSQL can talk to each other for achieving its common objective, providing a distributed version control system platform.

1.2. Bitbucket accessing PostgreSQL.

PostgreSQL relies upon a socket provided by the operating system, so the data of the cluster is kept behind a door protected by firewall rules.

1.3. PostgreSQL data source.

Getting Started

There is not much to say as this is a pretty simple process. PostgreSQL will need to be set up with a new user and database ready for Bitbucket.

PostgreSQL

# Part 1: Preparing the database.

$ pg_lsclusters

$ sudo systemctl -a | grep postgres

$ sudo -u postgres psql -c "\du" -c "\l"

2.1. Verifying if there are clusters already running.

# Part 2: Creating a new user (role) and database.

$ sudo -u postgres psql -c "create role thiago with createdb login password 'Th14g0_P4ssw0rd'"

$ psql -d postgres -c "create database bitbucket_db"

$ psql -d bitbucket_db -c "\du" -c "\l"

2.2. The owner of the database isn't the superuser postgres.

# Part 3: Changing the cluster configuration (postgresql.conf).

$ sudo -u postgres psql -c "show config_file"

$ sudo cat /etc/postgresql/11/main/postgresql.conf | grep listen_addresses

$ sudo sed -i "s|#listen_addresses = 'localhost'|listen_addresses = '\*'\t|" /etc/postgresql/11/main/postgresql.conf

$ sudo cat /etc/postgresql/11/main/postgresql.conf | grep listen_addresses

2.3. Allowing remote connections (postgresql.conf).

# Part 4: Changing the cluster configuration (pg_hba.conf).

$ sudo wc -l /etc/postgresql/11/main/pg_hba.conf

$ sudo tail -3 /etc/postgresql/11/main/pg_hba.conf

$ sudo sed -i "$ a # Allow remote connections (listen_addresses = '*') with authentication" /etc/postgresql/11/main/pg_hba.conf

$ sudo sed -i "$ a host\tall\t\tall\t\t192.168.0.0/16\t\tmd5" /etc/postgresql/11/main/pg_hba.conf

$ sudo sed -i "$ a host\tall\t\tall\t\t::/0\t\t\tmd5" /etc/postgresql/11/main/pg_hba.conf

$ sudo wc -l /etc/postgresql/11/main/pg_hba.conf

$ sudo tail -3 /etc/postgresql/11/main/pg_hba.conf

2.4. Changing access permissions (pg_hba.conf).

# Part 5: Restarting the cluster.

$ sudo -u postgres psql -c "show listen_addresses"

$ ss -nlp | grep 5432

$ sudo systemctl restart postgresql@11-main

$ sudo -u postgres psql -c "show listen_addresses"

$ ss -nlp | grep 5432

2.5. Applying the changes.

# Part 6: Opening the door.

$ sudo ufw status

$ sudo ufw allow 5432/tcp

$ sudo ufw status

$ ip addr show

2.6. Configuring the firewall and displaying the IP address.

# Part 7: Set a password for the superuser role.

$ sudo -u postgres psql -c "\password"

2.7. Changing the admin password.

Bitbucket

Here is what you will need to then do on the Bitbucket side.

# Part 1: Verifying if the earlier set up was correct.

$ telnet 192.168.0.106 5432

# (Optional) Using psql.

$ sudo -u postgres psql -h 192.168.0.106 -p 5432 -d bitbucket_db -U thiago -c "\conninfo"

3.1. Reaching the database remotely.

# Part 2: Extracting the Bitbucket Server.

$ ls

$ tar xzf atlassian-bitbucket-6.10.0

$ ls

$ du -sh atlassian-bitbucket-6.10.0

$ tree -L 1 atlassian-bitbucket-6.10.0

$ tree -L 1 atlassian-bitbucket-6.10.0/bin

3.2. Scripts for Linux and Windows.

# Part 3: Modifying the script (set-bitbucket-home.sh).

$ mkdir bitbucket-home

$ echo $(pwd)/bitbucket-home

$ cat atlassian-bitbucket-6.10.0/bin/set-bitbucket-home.sh | grep BITBUCKET_HOME=$

$ sed -i 's|BITBUCKET_HOME=$|BITBUCKET_HOME=/home/thiago/Documents/severalnines.com/database-blog/bitbucket-home|' atlassian-bitbucket-6.10.0/bin/set-bitbucket-home.sh

$ cat atlassian-bitbucket-6.10.0/bin/set-bitbucket-home.sh | grep BITBUCKET_HOME=/

3.3. Configuring BITBUCKET_HOME.

# Part 4: Modifying the script (set-jre-home.sh).

$ readlink -f $(which java)

$ cat atlassian-bitbucket-6.10.0/bin/set-jre-home.sh | grep JRE_HOME=$

$ sed -i 's|JRE_HOME=$|JRE_HOME=/usr/lib/jvm/java-11-openjdk-amd64|' atlassian-bitbucket-6.10.0/bin/set-jre-home.sh

$ cat atlassian-bitbucket-6.10.0/bin/set-jre-home.sh | grep JRE_HOME=/

3.4. Configuring JRE_HOME (Java).

# Part 5: Checking the hardware.

$ cat /proc/cpuinfo | grep processor | wc -l

$ free -h

3.5. CPU cores and RAM at startup.

# Part 6: Running the Bitbucket Server with Elasticsearch.

$ ./atlassian-bitbucket/bin/start-bitbucket.sh

$ free -h

3.6. Executing with Elasticsearch (Default).

# Part 7: Running the Bitbucket Server without Elasticsearch.

$ ./atlassian-bitbucket/bin/start-bitbucket.sh --no-search

$ free -h

3.7. Executing without Elasticsearch (Saves 1 GB of RAM).

# Part 8: Quick look at BITBUCKET_HOME.

$ du -sh bitbucket-home

$ tree -L 1 bitbucket-home

3.8. Inside of BITBUCKET_HOME.

Integrating PostgreSQL & Bitbucket

After configuring PostgreSQL and Bitbucket, their integration must be done through the browser (http://localhost:7990/).

4.1. Bitbucket starting up.

# Displaying tables

$ psql -h 192.168.0.106 -d bitbucket_db -c "\dt"

4.2. Listing current database tables.

You can now setup Bitbucket to use Java Persistence API, with Hibernate as the implementation, for creating the domain model in the database, using the PostgreSQL JDBC driver.

4.3. Data Source configuration.

# Displaying tables (again)

$ psql -h 192.168.0.106 -d bitbucket_db -c "\dt"

4.4. Listing again, showing 10 of 164 rows.

# Verifying the connection pool.

$ psql -h 192.168.0.106 -d bitbucket_db -c "select pid,usename,application_name,state from pg_stat_activity where datname = 'bitbucket_db'"

4.5. Displaying the connection pool.

Conclusion

Keep in mind that if your network is using DHCP, it's a good idea to configure the IP address to static in your router, or Bitbucket may fail trying to find PostgreSQL later on.

Most of this blog has made use of regular expressions for changing configuration files without opening text editors, but they can be used as well on web browsers for debugging purposes, try searching for the error message “could not change directory to ?: Permission denied” with double quotes, or any other issue you may find, replacing the computer specific path with wildcard '?'.

User management is one of the day-to-day tasks that are performed by database administrators. If your database administrator is not available and you have been appointed to perform his tasks, you may have been asked to fill and perform user management tasks as well. Let’s take a look at what is the most important when managing access to your database.

The details will differ depending on the scale of your environment. Larger the database infrastructure, more likely user management is automated to some extent. Ansible playbooks that run and ensure all of the required users are available and have correct privileges, scripts that create or modify users across the whole infrastructure. On the other hand, in smaller setups, it is more likely that you will be making changes by hand, logging into databases and executing direct statements that will create or alter existing users. No matter how you will achieve that, there are a couple of best practices you probably would like to stick to.

Make Sure All Users Have a Proper Password

Users without a password are a security threat, that’s no-brainer. What you want is a password that is decent - simple passwords like ‘1234qwerty’ or ‘pa$$word’ will not do. Some of the database systems can assist you in enforcing proper passwords. For example, MySQL can be configured to enforce password strength. You can configure exact requirements in terms of the password length, existence of lowercase or uppercase characters, special characters and so on. You can even require a dictionary search to ensure password is not a common word. In MySQL world, if you are not the only person who has access to creating new users, you may want to run regular checks using pt-show-grants, a tool that dumps the users. It allows you to easily track users without passwords.

Make Sure All Users Have Only Those Privileges That They Require

Privileges are designed to limit the access to the database on per-user (per-database, per-table or even per-column basis, all depends on the database system in question) basis. It is quite important to use it to your advantage. If you have to create a user that has to have access to a subset of data, do not grant him access to all of it. If a user has to execute SELECT queries, there’s no reason to grant him write access to the data. This is very important to keep in mind that no matter how secure your database is, if someone gains access to the web server, he’ll be able to access database credentials used by the application. If your regular pattern is to grant too many privileges, this will seriously affect the security of your database. Again, just like in the previous section, pt-show-grants will be useful to track the users created in your MySQL database and make sure that they do not have extensive privileges.

Don’t Ever Use Superuser for Applications to Connect To

This is sort of similar to what we discussed in the previous section. Superuser does have extensive privileges and it will become a security threat when used in the application. There is more to this than just a security, though. Please keep in mind that the database is configured to accept a limited number of connections from the application. This is quite important to keep in mind that if you ran out of available connections, you won’t be able to connect to the database. Quite straightforward, isn’t it? Yet there’s a feature in MySQL that allows for one connection from the superuser even if all of the connections, as per limits, are used. This allows the administrator to make a connection and change the configuration even if the application cannot connect to the database. Using superuser by the application makes this feature pointless - application will use all available connections and that one additional slot for the superuser, leaving you without easy ways to connect to the database and reconfigure it.

Make Sure Users Are Limited to a Required Host

When creating a user you should think where that user connects from. Ideally, you would be able to identify a host or a subnet used by that user for database connections. This will allow you to reduce a risk that a given application user would be used to access the database from an unauthorized location. If you create a user account dedicated to the application access to the database, it probably should not attempt to access your database from some other location like a bastion host or a backup server.

How to Use ClusterControl for Database User Management?

 ClusterControl comes with a built-in user management interface.

As you can see above, you can easily see the users that have been created in your system. You can as well create more of them:

If you need, you can easily edit the existing user:

It is also possible to check the inactive users.

If a user stays inactive, you may want to revoke its privileges to remove unnecessary access paths to the database.

User management plays an important role in database security, we hope this short blog will help you to understand better its most important aspects.