Feed Aggregator

When it comes to MariaDB/MySQL Database consulting engagements we often see, that our customers are very confused by many different technologies advertised to them.

Below you will find some simple concepts and ideas we use during our consulting engagements to solve some issues and answer some questions.

General problem solving approach

• 1
  . Written description of the problem to solve. This helps to learn about your problem and better understand your problem.
• 2
  . Searching for a suitable method, technology or architecture solving the described problem.
• 3
  . Evaluating a product or solution which is specialised solving the described problem. Check if the product/solution matches your problem.
  Caution: In this topic we are strongly MariaDB/MySQL biased!
• 4
  . Do a Proof of Concept (PoC) to verify the chosen solution will fix your described problem (or not).

Splitting Database Tables (BLOB/TEXT)

Sometimes it might be useful to split 1 entity (table) into 2 separate tables. This makes sense when you have either:

• hot …

FromDual has the pleasure to announce the release of the new version 1.1.0 of its popular Database Performance Monitor for MariaDB, MySQL and Galera Cluster fpmmm.

The FromDual Performance Monitor for MariaDB and MySQL (fpmmm) enables DBAs and System Administrators to monitor what is going on inside their MariaDB and MySQL databases and on their machines where the databases reside.

More detailed information your can find in the fpmmm Installation Guide.

Download

The new FromDual Performance Monitor for MariaDB and MySQL (fpmmm) can be downloaded from here. How to install and use fpmmm is documented in the fpmmm Installation Guide.

In case you find a bug in the FromDual Performance Monitor for MariaDB and MySQL please report it to the FromDual Bugtracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for fpmmm).

Any feedback, statements and testimonials are welcome as well! Please send them [to us](mailto:feedback@fromdual.com?Subject=Feedback for fpmmm “Feedback for …

You are using an old version of the FromDual Performance Monitor for MariaDB and MySQL (fpmmm). Please consider using a more recent version.

Your InnoDB Redo Log Buffer is too small for large transactions. Make the InnoDB Redo Log Buffer (innodb_log_buffer_size) bigger if you have enough RAM available to avoid additional I/O.
1 Mibyte is good for databases with small transactions. 8 Mibyte is good for medium size transactions. 64 MiByte is good for large transactions.

See also for MariaDB MySQL.

Table of Contents

• Before you start
• Infinite Loop Example

Before you start

SQL> SET SESSION sql_mode='ORACLE';

Infinite Loop Example

DELIMITER /

BEGIN

LOOP
  SELECT 'Hello world from MariaDB anonymous PL/SQL block!';
END LOOP; 

END;
/

DELIMITER ;

Up to version 5.5 MariaDB and MySQL can be considered as “the same” databases. The official wording at those times was “drop-in-replacement”. But now we are a few years later and times and features changed. Also the official wording has slightly changed to just “compatible”.
FromDual recommends that you consider MariaDB 10.3 and MySQL 8.0 as completely different database products (with some common roots) nowadays. Thus you should work and act accordingly.

Because more and more FromDual customers consider a migration from MySQL to MariaDB we were testing some migration paths to find the pitfalls. One upgrade of some test schemas led to the following warnings:

# mysql_upgrade --user=root
MariaDB upgrade detected
Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv                                 OK
...
mysql.user                                         OK
Phase 2/7: Installing used storage engines
Checking for tables with unknown …

Table of Contents

• ifconfig
• ip
• ethtool --statistics
• nstat
• netstat --statistics
• ss
• Reasons for dropped packets
• Features of Network card
• Features of Network driver
• Network driver ring buffer
• TCP socket received and send buffer
• Flow Control
• Check over time: sar
• Dropwatch
• perf
• tcpdump
• /proc/net/netstat
• System Tap (stap)
• Literature

When we do MariaDB and MySQL Database server analysis we see from time to time systems with a lot of dropped packets on a network interface:

ifconfig

# export DEV=enp0s25
# ifconfig ${DEV}
eth0 Link encap:Ethernet  HWaddr d0:50:99:70:f8:8e
     inet addr:192.168.1.135  Bcast:192.168.1.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:4005096 errors:0 dropped:365956 overruns:0 frame:0
     TX packets:3293230 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:1000
     RX bytes:2897493579 (2.8 GB)  TX bytes:461542567 (461.5 MB)

As we all learned ifconfig is old and thus bad so we should use the new command ip instead. See …

Modern Linux Systems have a mechanism called logrotate to rotate different log files.

The general configuration file is located under /etc/logrotate.conf and specific changes are under /etc/logrotate.d/

By default the logrotate job is started once a day by a O/S cron.daily job: /etc/cron.daily/logrotate

Because the default log rotation configuration does not exactly what I want I have adapted it a bit:

First I need a database user for log rotation:

CREATE USER 'logrotate'@'localhost' IDENTIFIED BY 'secret';
GRANT RELOAD ON *.* to 'logrotate'@'localhost';

This user needs a password file:

#
# /root/.logrotate.cnf (chmod 600)
#

[mysql]
user     = logrotate
password = secret

And this is my log rotation configuration file:

#
# /etc/logrotate.d/mysql
#

/var/log/mysql/error.log {
  compress
  notifempty
  size 100M
  missingok
  create 640 mysql mysql
  rotate 5
  maxage 365
  postrotate
    /usr/bin/mysql --defaults-extra-file=/root/.logrotate.cnf --execute='FLUSH ERROR LOGS'
  endscript
} …

Since version 2.1.0 the FromDual Recovery Manager (rman) for MariaDB and MySQL has also a progress indicator for the restore of logical backups made with mysqldump. This feature was implemented because of the numerous requests of FromDual rman users who were not happy with the default mysql behavior.

You can check your current rman version as follows:

# ./bin/rman --version
2.2.1

As with all FromDual tools you get a command overview with examples with the --help option:

# ./bin/rman --help | less
...
  progress      Print progress information to STDOUT.
...

A backup for example is done as follows:

# ./bin/bman --target=brman:secret@127.0.0.1:3308 --type=full --mode=logical --policy=daily --instance-name=qamariadb102

The Recovery Manager progress indicator logs to STDOUT:

# ./bin/rman --target=brman:secret@127.0.0.1:3308 --type=full --mode=logical --policy=daily --instance-name=qamariadb102 --progress --backup-name=bck_qamariadb102_full_2019-08-20_21:15:23.sql

Reading configuration from /etc/mysql/my.cnf …

Table of Contents

• Introduction
• Character Sets
• Steps to convert Character Set to utf8mb4
  • Analyzing the Server
  • Analyzing the Application and the Clients
  • Preparation of the Server Settings and the Application
  • Convert Tables to utf8mb4
  • Testing of new Character Set
• MySQL Pump
• MySQL Master/Slave Replication for Character Set conversion
• MySQL Shell, mysqlsh Upgrade Checker Utility

Introduction

Recently we had a consulting engagement where we had to help the customer to migrate from latin1 Character Set to utf8mb4 Character Set. In the same MySQL consulting engagement we considered to upgrade from MySQL 5.6 to MySQL 5.7 as well
[ Lit.
]. We decided to split the change in 2 parts: Upgrading to 5.7 in the first step and converting to uft8mb4 in the second step. There were various reasons for this decision:

• 2 smaller changes are easier to control then one big shot.
• We assume that in 5.7 we experience less problems with utf8mb4 because the trend given by MySQL was more towards utf8mb4 in 5.7 than in MySQL 5.6. So …

FromDual has the pleasure to announce the release of the new version 0.9.2 of its popular FromDual Ops Center for MariaDB and MySQL focmm.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBA’s and System Administrators to manage MariaDB and MySQL database farms. Ops Center makes DBA and Admins life easier!

The main task of Ops Center is to support you in your daily MySQL and MariaDB operation tasks. More information about FromDual Ops Center you can find here.

Download

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from here. How to install and use focmm is documented in the Ops Center User Guide.

In the inconceivable case that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for focmm).

Any feedback, statements and testimonials are welcome as well! Please send them to …

FromDual has the pleasure to announce the release of the new version 2.2.1 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is describe in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.2.1

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.2.1.tar.gz
shell> rm -f brman
shell> ln -s brman-2.2.1 brman

Changes in FromDual Backup and Recovery Manager 2.2.1

This release is a new minor release. It contains only bug fixes. We have tried to …

Table of Contents

• Unlink multiple records from Target List
• Find Target Lists of Target/Prospect
• List all email addresses of all accounts
• Objects without Target List
• Find PLP without Objects
• Find Prospect List Entries per Prospect Type
• Quick set-up on Ubuntu 18.04
• Add targets from old target lists to new target list
• Undelete accounts

Unlink multiple records from Target List

1. Find id of record to unlink.

SQL> SET @uid = 'f232e8d0-9b82-20f7-7c35-5c1a5ce94e0a';

2. Find entry in Target List table:

SQL> SELECT plp.id, plp.related_type, plp.date_modified, plp.deleted
     , pl.name
  FROM prospect_lists_prospects AS plp
  JOIN prospect_lists AS pl ON plp.prospect_list_id = pl.id
 WHERE plp.related_id = @uid
;

3. Update entry in Target List table:

SQL> START TRANSACTION;
SQL> UPDATE prospect_lists_prospects
   SET deleted = 1
 WHERE related_id = @uid
   AND id = '5b3047b2-3b45-1a6e-e6fb-5d3fe1efa93b'
;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
SQL> …

FromDual has the pleasure to announce the release of the new version 2.2.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is describe in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.2.0

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.2.0.tar.gz
shell> rm -f brman
shell> ln -s brman-2.2.0 brman

Changes in FromDual Backup and Recovery Manager 2.2.0

This release is a new major release series. It contains some new features. We have tried …

See also our former articles:

• Do not underestimate performance impacts of swapping on NUMA database systems
• MariaDB and MySQL swap analysis

When we do analysis of MariaDB Database servers we also check the memory (RAM and Swap) available:

# free --kilo --wide
              total        used        free      shared     buffers       cache   available
Mem:       16106252     4329952      703356      199008      307872    10765072    11042748
Swap:      31250428      528684    30721744

The values for buffers and especially for cache can be sometimes quite big. In this case they use about 10 GiB. So let us have a look what these things called buffers and cache are, using our valuable RAM… When we check the man pages of free we will find:

# man free
...
buffers    Memory used by kernel buffers (Buffers in /proc/meminfo)
cache      Memory used by the page cache and slabs (Cached and Slab in /proc/meminfo)
buff/cache Sum of buffers and cache

So let us check a more fine grained information in /proc/meminfo …

The General Query Log in MariaDB is a Log (file or table) that logs every statement sent from the Application to the MariaDB Database Server from the Connect Command to the Quit Command. A simple example you can find here:

bin/mysqld, Version: 10.4.6-MariaDB-log (MariaDB Server). started with:
Tcp port: 3306  Unix socket: /var/run/mysqld/mysql.sock
Time                Id Command  Argument
190709 21:27:30   3872 Connect  root@localhost as anonymous on test
                  3872 Query    INSERT INTO test VALUES (NULL, CONCAT('Test data insert from laptop5 on ', @@hostname), CURRENT_TIMESTAMP())
                  3872 Quit

The General Query Log can be enabled dynamically with the following command:

SQL> SET GLOBAL general_log = 1;

and is written either to a file (defined with general_log_file) or a table (mysql.general_log, defined with log_output=TABLE).

But unfortunately the General Query Log logs all Commands from all Connections. So it can grow quickly and you loose overview over what happens… …

The story goes on here.

Ups, that was a quick reaction… https://jira.mariadb.org/browse/MDEV-9077

The MySQL sys Schema is a cool thing and was mostly developed by Mark Leith and Jesper Wisborg Krogh from Oracle/MySQL. The sys Schema is based on the somehow cryptic MySQL PERFORMANCE_SCHEMA which was developed by Oracle/MySQL as well.

The sys Schema is: “A collection of views, functions and procedures to help MySQL administrators get insight in to MySQL Database usage.”

Unfortunately the sys Schema was not ported from MySQL to MariaDB by the company MariaDB Corp. so far. So a good opportunity to act as a community…

In the beginning the sys Schema was hosted on GitHub: mysql/mysql-sys. So we could take the sys Schema from there for MariaDB. But since 3 years ago no more changes appeared on GitHub. It seems like the project was abandoned.

According to a well informed source the project on GitHub is not dead but the developers have just been working on other priorities. And the source announced another release soon (they are working on it at the moment). This statement was made in March 2018 …

I could not resist! I have forked the MySQL sys Schema on GitHub.