EditLine vs ReadLine on FromDual GmbH

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 23 Mar 2026 17:42:00 +0100

FromDual has the pleasure to announce the release of the new version 3.0.0 of its popular MariaDB, MySQL and PostgreSQL multi-instance environment MyEnv.

The new MyEnv can be downloaded here. How to install MyEnv is described in the MyEnv Installation Guide.

In the inconceivable case that you find a bug in the MyEnv please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Upgrade from 2.x to 3.0

Please check the MyEnv Installation Guide.

Changes in MyEnv 3.0.0

MyEnv

Template warning improved.
Distro version in --version added.
Check MyEnv configuration permissions.
#fd increased for MyEnv.
myenv.conf should have more secure permissions now.
Situation caught when my.cnf is missing in myenv.conf.
Directories home and run moved to dba and myenv.
Unit file mariadb.service and mysql.service replaced by dba.service.
User mysql replaced by dba in template.
start_stop fixed warning in case argv[1] is missing.
sys_uid filter fixed for Rocky Linux.
dba unit file added to package.
Nagios plugins detection removed from showMyEnvVersion.
Old SysV init files removed and replaced by Systemd unit files.
dba user was introduced and check for system user added.
User dba changed an cosmetic fixes.

MyEnv Installer

2 concurrent installMyEnv versions cannot run any more.
Directroy binlog, cgroups and angel removed from postgresql type installation.
Wrapper script installMyEnv.sh removed.
Cosmetics fixed in installer.
Installation made more mysql friendly.
libaio1t64 considered during installation recommendations on DEB systems.
Next free port suggestion during installMyEnv improved. It will suggest the first free port now.
apt-get and yum replaced by apt and dnf.

MyEnv Utilities

insert_test.sh made PostgreSQL ready.

PostgreSQL

PostgreSQL instance is stopped with fast instead of immediate now.
show_create_table.sh for PostgreSQL made nicer.
PostgreSQL status.sql added.
Minor fixes for PostgreSQL.

General

CHANGELOG updated.
rc made unique.
Minor bugs …

FromDual Performance Monitor 2.2.1 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 19 Feb 2026 18:18:00 +0100

FromDual has the pleasure to announce the release of the new version 2.2.1 of its popular Database Performance Monitor for MariaDB, Galera Cluster, MySQL and PostgreSQL fpmmm.

The FromDual Performance Monitor enables Database and System Administrators to monitor and understand what is going on inside their databases and on the machines where the databases reside.

More information you can find here: FromDual Performance Monitor.

Download

The new FromDual Performance Monitor can be downloaded from our Sofware Download page or you can use our repositories. How to install and use the FromDual Performance Monitor is documented in the Documentation.

In the inconceivable case that you find a bug in the FromDual Performance Monitor please report it to us by sending an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Monitoring as a Service (MaaS)

You do not want to set-up your database monitoring yourself? No problem: Choose our Monitoring as a Service (MaaS) to safe time and costs!

Installation of Performance Monitor 2.2.1

How to install the FromDual Performance Monitor you can find in the Installation Guide.

Upgrade of fpmmm tar ball from 1.x to 2.2.1

There are some changes in the configuration file (fpmmm.conf):

The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
The key Methode was spelled wrong in the configuration file. It was renamed to Method.
The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tar balls follow this instruction:

$ cd /opt
$ tar xf /download/fpmmm-2.2.1.tar.gz
$ rm -f fpmmm
$ ln -s fpmmm-2.2.1 fpmmm

Changes in FromDual Performance Monitor 2.2.1

These release notes include both the changes that came with version 2.2.0 and version 2.2.1.

This release contains new features and various bug fixes.

You can verify your current FromDual Performance Monitor version with the …

What is the quickest way to load data into the database?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 11 Feb 2026 10:04:00 +0100

We had some really exciting problems to solve for the last customer! Especially because the database wasn’t exactly small.

Here are some key data: CPU: 2 sockets x 24 cores x 2 threads = 96 vCores, 756 G RAM, 2 x 10 Tbyte PCIe SSD in RAID-10 and 7 Tbyte data, several thousand clients, rapidly growing.

The current throughput: 1 M SELECT/min, 56 k INSERT/min, 44 k UPDATE/min, 7 k DELETE/min averaged over 30 days. With a strong upward trend. Application and queries not consistently optimised. Database configuration: ‘state of the art’ not verified with benchmarks. CPU utilisation approx. 50% on average, more at peak times. I/O system still has available resources.

The customer collects position and other device data and stores it in the database. In other words, a classic IoT problem (with time series, index clustered table, etc.).

The question he has asked is: What is the fastest way to copy data from one table (pending data, a kind of queue) to another table (final data, per client)?

The data flow looks something like this:

+------------+
| IoT Device |--+
+------------+ \
 \ +-----+
+------------+ \ | AS | +--------------+ Processing +------------+
| IoT Device |------+-->| |-->| Pending data |------------->| Final data |
+------------+ / | 400 | +--------------+ of data +------------+
 / +-----+
+------------+ /
| IoT Device |--+
+------------+

3 different variants to copy the data were available for selection.

Variant 1: `INSERT` and `DELETE` (simplest form)

The simplest variant is a simple INSERT and DELETE. This variant is particularly problematic because MariaDB/MySQL and PostgreSQL have AUTOCOMMIT enabled by default (here, here, here and here).

To help you visualise this a little better, here is some pseudocode:

// 20k rows
for (i = 1; i <= 2000; i++) {

 SELECT * FROM pending LIMIT 10;
 foreach ( row ) {
 INSERT INTO final;
 -- implicit COMMIT
 DELETE FROM pending WHERE id = row[id];
 -- implicit COMMIT …

MariaDB has broken the concept of dynamically configurable buffer pools!

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 09 Feb 2026 19:14:00 +0100

Problem description

MySQL introduced the dynamically configurable InnoDB buffer pool with 5.7.5 in September 2014 (here and here):

The innodb_buffer_pool_size configuration option can be set dynamically using a SET statement, allowing you to resize the buffer pool without restarting the server. For example:

mysql> SET GLOBAL innodb_buffer_pool_size=402653184;

MariaDB 10.2.2 adopted this feature in September 2016 (source):

InnoDB was merged from MySQL-5.7.14 (XtraDB is disabled in MariaDB-10.2.2 pending a similar merge)

The problematic thing is, on the one hand, that this feature now no longer works as it did before and no longer works as expected. On the other hand, they changed the behaviour in spring 2025 within a major release series (LTS), which in my opinion is an absolute no-go (source):

From MariaDB 10.11.12 / 11.4.6 / 11.8.2, there are significant changes to the InnoDB buffer pool behavior.

And what is more, the description of this is quite poor (source):

decreasing innodb_buffer_pool_size at runtime does not release memory (MDEV-32339)

reorganise innodb buffer pool (and remove buffer pool chunks) (MDEV-29445)

The Linux memory pressure interface, which could previously not be disabled and could cause performance anomalies, was rewritten and is disabled by default. (MDEV-34863)

Server crashes when resizing default innodb buffer pool after setting innodb-buffer-pool-chunk-size to 1M (MDEV-34677)

In the corresponding worklog (MDEV-36197) MarkoM also describes a different behaviour:

innodb_buffer_pool_size_auto_max (my proposal for this task) would set the maximum for the automation (default: 0 to disable the logic).

which would have made much more sense in my opinion.

How did it work before?

How did it work in the past with MariaDB and still today with MySQL:

SQL> SHOW GLOBAL VARIABLES LIKE 'innodb_buffer_pool%size';
+-------------------------------+-----------+
| Variable_name | Value |
+-------------------------------+-----------+
| …

How much space does NULL need?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sun, 08 Feb 2026 16:15:00 +0100

The last time I consulted a customer, he came up to me beaming with joy and said that he had taken my advice and changed all the primary key columns from BIGINT (8 bytes) to INT (4 bytes) and that had made a big difference! His MySQL 8.4 database is now 750 Gbyte smaller (from 5.5 Tbyte). Nice!

And yes, I know that contradicts the recommendations of some of my PostgreSQL colleagues (here and here). In the MySQL world, more emphasis is placed on such things (source):

Use the most efficient (smallest) data types possible. MySQL has many specialized types that save disk space and memory. For example, use the smaller integer types if possible to get smaller tables

Also, InnoDB works a wee bit differently (index clustered table and primary key in all secondary keys) than PostgreSQL (heap table, indices with row pointer (ctid)).

But that’s not really the issue. Immediately afterwards, he asked me whether the deletion of columns of type DOUBLE (8 bytes, in PostgreSQL-speak DOUBLE PRECISION) would also save space or whether he should rather drop the columns straight away. My first reflex response to DOUBLE was: NULL is good, followed by OPTIMIZE TABLE (VACUUM FULL in PostgreSQL parlance). But the second thought was, DOUBLE is a data type of fixed length, does NULL also apply there or only for data types with variable length? Caution is the mother of the porcelain box! Love to consult the manual first…

And there it says (source):

Declare columns to be NOT NULL if possible. It makes SQL operations faster, by enabling better use of indexes and eliminating overhead for testing whether each value is NULL. You also save some storage space, one bit per column. If you really need NULL values in your tables, use them. Just avoid the default setting that allows NULL values in every column.

and (source):

The variable-length part of the record header contains a bit vector for indicating NULL columns. … Columns that are NULL do not occupy space other than the bit in this vector. The variable-length part of …

Someone is deleting my shared memory segments!

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sun, 08 Feb 2026 06:27:00 +0100

When we work with PostgreSQL under our myEnv, we regularly get shared memory segment errors. Example:

psql: error: connection to server on socket "/tmp/.s.PGSQL.5433" failed:
FATAL: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

or we see similar messages in the PostgreSQL error log:

ERROR: could not open shared memory segment "/PostgreSQL.4220847662":
No such file or directory

Because I am a MariaDB/MySQL admin, I am not very familiar with shared memory problems (MariaDB/MySQL does not work with shared memory). Fortunately, a search on the Internet led us on the right track (source). It is noted there:

The documentation of systemd states that this only happens for non-system users. Can you check whether your “postgres” user (or whatever you are using) is a system user?

Linux System User

First I had to find out what a system user under Linux actually is. I found an answer here: What’s the difference between a normal user and a system user?.

That is not a technical difference but an organizational decision. E.g. it makes sense to show normal users in a login dialog (so that you can click them instead of having to type the user name) but it wouldn’t to show system accounts (the UIDs under which daemons and other automatic processes run) there.

The LSB standard says: User ID Ranges:

The system User IDs from 0 to 99 should be statically allocated by the system, and shall not be created by applications.
The system User IDs from 100 to 499 should be reserved for dynamic allocation by system administrators and post install scripts using useradd.

On my Ubuntu system it looks like this:

$ grep SYS_ /etc/login.defs
#SYS_UID_MIN 100
#SYS_UID_MAX 999
#SYS_GID_MIN 100
#SYS_GID_MAX 999

This would be correct for the PostgreSQL user:

$ id postgres
uid=130(postgres) gid=142(postgres) groups=142(postgres),116(ssl-cert)

But since the PostgreSQL instance in question runs under our …

FromDual Backup and Recovery Manager

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 07 Feb 2026 20:17:00 +0100

Coming soon…

Have also a look at the Backup and Recovery Manager Documentation.

Load CSV files into the database

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 06 Feb 2026 18:04:00 +0100

Recently, I wanted to display the places of residence of the members of my club on a map for a personal gimmick (IGOC members). I knew the addresses of the club members. But not the coordinates of their places of residence.

So I went in search of the coordinates and found what I was looking for at the Federal Office of Topography (swisstopo).

The data is available there as a CSV file. Details here: Swiss town coordinates.

How do I load this data into a database?

Loading the data with MariaDB/MySQL

MariaDB and MySQL have the LOAD DATA INFILE command:

SQL> DROP TABLE IF EXISTS wgs84;

SQL> -- SET GLOBAL local_infile = ON; -- Only needed with MySQL

SQL> CREATE TABLE wgs84 (
 ortschaftsname VARCHAR(32)
, plz4 SMALLINT
, zusatzziffer SMALLINT
, zip_id SMALLINT UNSIGNED
, gemeindename VARCHAR(32)
, bfs_nr SMALLINT
, kantonskuerzel CHAR(2)
, adressenanteil varchar(8)
, e DOUBLE
, n DOUBLE
, sprache VARCHAR(8)
, validity VARCHAR(12)
);

SQL> -- TRUNCATE TABLE wgs84;

SQL> LOAD DATA LOCAL INFILE '/tmp/AMTOVZ_CSV_WGS84/AMTOVZ_CSV_WGS84.csv'
INTO TABLE wgs84
FIELDS TERMINATED BY ';'
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
;
Query OK, 5713 rows affected
Records: 5713 Deleted: 0 Skipped: 0 Warnings: 0

You can then query the data in the database:

SQL> SELECT * FROM wgs84 ORDER BY ortschaftsname LIMIT 5;
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| ortschaftsname | plz4 | zusatzziffer | zip_id | gemeindename | bfs_nr | kantonskuerzel | adressenanteil | e | n | sprache | validity |
+----------------+------+--------------+--------+--------------+--------+----------------+----------------+-------------------+--------------------+---------+------------+
| Aadorf | 8355 | 0 | 4672 | Aadorf | 4551 | TG | 96.802 % | 8.903193007810433 | 47.491079014637265 | de | 2008-07-01 …

Attribute promotion and demotion in the MariaDB Galera Cluster

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 28 Nov 2025 16:26:48 +0000

In MariaDB master/slave replication there is a feature called attribute promotion/demotion.

Simply put, it is about how the slave behaves or should behave if the master and slave have different column definitions or even a different number of columns or a different sequence of columns.

Use case of the customer

This week we discussed with a customer the case of how he could perform a rolling schema upgrade (RSU) in a Galera cluster.

With previous schema changes he has always had problems, which has led to a total failure of the cluster for several hours.

The customer says that columns are never deleted and new columns are only ever added at the end of a table.

And that it is NOT possible to ensure that there are no more write connections during the rolling schema upgrade.

The PHP ORM framework Doctrine is used.

What does the MariaDB documentation say about this?

The study of the MariaDB documentation did not lead to a conclusive result whether a rolling schema upgrade in the running Galera Cluster operation WITH changes (DML statements) on the schema to be changed (AND the tables to be changed) is supported or not and thus should work or not.

Source: Rolling Schema Upgrade (RSU)

When replicating with different table structures, there is only general information on replication but nothing specific to Galera (whether it works or not):

“Tables on the replica and the primary do not need to have the same definition in order for replication to take place. There can be differing numbers of columns, or differing data definitions and, in certain cases, replication can still proceed.”

Source: Replication When the Primary and Replica Have Different Table Definitions

For the attribute promotion/demotion feature there is a special MariaDB Server configuration parameter that controls the behaviour: slave_type_conversions.

If you read between the lines here, this could also work for Galera Cluster:

“Determines the type conversion mode on the replica when using row-based replication, including replications in …

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.2 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 22 Oct 2025 20:41:15 +0200

FromDual has the pleasure to announce the release of the new version 2.3.2 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.2

$ cd /opt
$ tar xf /download/brman-2.3.2.tar.gz
$ rm -f brman
$ ln -s brman-2.3.2 brman

Changes in FromDual Backup and Recovery Manager 2.3.2

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1, 2.2 and 2.3 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

$ bman --version
$ rman --version

General

Tests improved.
New features documented and documentation updated.
Libraries from MyEnv project updated.

FromDual Backup Manager (bman)

SSL/TLS implemented for bman only.
Binary log position should be gathered correctly now also with MySQL 8.4.

FromDual Recovery Manager (rman)

No changes.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from [from us](mailto:contact@fromdual.com?Subject=Commercial use of FromDual brman).

MariaDB Honeypot

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 06 Mar 2025 09:50:18 +0000

In our MariaDB advanced training courses, which we hold approximately every two months, we use machines that are directly exposed to the Internet with a public IP address. Warning: You should NEVER expose a database directly to the Internet without protection! Typically, it takes less than 72 hours (3 days) before we are exposed to the first external access attempts.

This looks something like this in the MariaDB error log:

[Warning] Aborted connection 22939 to db: 'unconnected' user: 'unauthenticated' host: '118.193.58.125' (This connection closed normally without authentication)
[Warning] Aborted connection 22940 to db: 'unconnected' user: 'unauthenticated' host: '118.193.58.125' (This connection closed normally without authentication)
[Warning] Access denied for user ''@'118.193.58.125' (using password: NO)
[Warning] Access denied for user 'root'@'118.193.58.125' (using password: YES)
[Warning] Access denied for user 'root'@'118.193.58.125' (using password: YES)

First of all, it was checked whether a database is listening and how it responds. Then various attempts are made to penetrate the database. It looks like there are various sampling and attack patterns. The anonymous user (''@'%') and 'root'@'%' are checked with and without a password. Whether other users will be tested remains to be seen over a longer observation period.

And this is what it looks like from the MariaDB General Query Log:

287793 Connect root@196.251.91.77 on using TCP/IP
287793 Connect Access denied for user 'root'@'196.251.91.77' (using password: NO)
287794 Connect root@196.251.91.77 on using TCP/IP
287794 Connect Access denied for user 'root'@'196.251.91.77' (using password: YES)
287796 Connect root@196.251.91.77 on using TCP/IP
287796 Connect Access denied for user 'root'@'196.251.91.77' (using password: YES)

Preparing the honeypot

At the end of the training, we no longer need the machines, they will be removed. That’s why I was tempted to try out what happens when an access attempt is successful. To test this, the user …

How does Galera Cluster behave with many nodes?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 24 Jan 2025 18:12:55 +0000

Recently I had the opportunity to have a lot of Linux systems (VMs with Rocky Linux 9) from one of our regular Galera Cluster trainings all to myself for a week. And MariaDB 11.4.4 with Galera Cluster was already installed on the machines.

Since I had long wanted to try out how a Galera Cluster behaves with an increasing number of nodes, now was the opportunity to try it out.

The following questions were to be answered:

How does the throughput of a Galera cluster behave depending on the number of Galera nodes?
Which configuration gives us the highest throughput?

A total of 5 different test parameters were experimented with:

Number of Galera nodes.
Number of client machines (= instances).
Number of threads per client (--threads=).
Number of Galera threads (wsrep_slave_threads).
Runtime of the tests. This parameter was varied because some tests were cancelled during the run. It may be possible to eliminate this parameter with a lower rate (--rate) in the load test. As it turned out, it did have an influence on the result or the measured throughput (e.g. test 4b and 5 or 18 and 19).

A total of 35 different tests were run. See raw data.

Throughput as a function of the number of Galera nodes

Throughput related to # nodes
Test	# gal nodes	# threads/client	runtime [s]	tps	runtime [s]
7	1	8	180	596.3	180
8	2	8	180	567.8	180
9	3	8	180	531.9	180
11	4	8	180	495.2	180
12	5	8	180	492.2	180
13	6	8	180	502.9	180
14	7	8	180	459.5	180
15	8	8	180	458.6	180
16	9	8	180	429.2	180

Throughput related to
# nodes

The throughput in the Galera cluster decreased slightly from 600 tps to 430 tps (28%) when the number of nodes was increased from 1 to 9.

Throughput as a function of the number of connections

The main variation here was with the number of clients and threads per client. The optimum seems to be 30 - 40 connections in this setup. Varying the number of Galera threads (wsrep_slave_threads) does not seem to have had much effect in our case. The system does not seem to be able to deliver much more than …

Model v2 slightly improves results

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Aug 2024 15:55:43 +0200

We have improved our model (see v2) so the results are a bit better/more accurate now.

For usage see ./fromdual_llm_v2.php --help

Playing with MariaDB Vector for initial AI tests

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Aug 2024 10:51:23 +0000

Artificial intelligence (AI) and vector databases are on everyone’s lips these days. Since MariaDB will soon be coming onto the market with vector database functionality, as a database consultant I thought it was about time I got to grips with the subject so that I have at least a hint of what it’s all about…

Since I’m not so much of a theoretician but rather like to do something practical, I’ve built a small “AI” prototype that anyone can build very quickly and easily on their laptop (without a GPU)…

I also took the liberty of stealing the graphs from the MariaDB Foundation presentation (see sources at the end).

Downloading the MariaDB database with vector functionality

There are no MariaDB packages with vector functionality yet, but the source code is already available. So you can quickly build the binaries yourself. This took just under an hour on my old box. Once the binaries are built, you can make a tarball out of them:

# tar xf mariadb-11.6.0_vector.tar.gz
# cd mariadb-11.6.0_vector/
# cmake .
# make
# make package

The MariaDB database then only needs to be started.

The model

To show the concept of tokenisation I decided to build an AI for URLs and to show the concept of different models and their potential for improvement I built a very stupid model in PHP, which simply decomposes a URL.

The question that this model should be able to answer is: “Give me similar URLs to the following URL.”

The corresponding table looks like this:

DROP TABLE IF EXISTS `urls`;
-- TRUNCATE TABLE is NOT sufficient!!!
CREATE TABLE `urls` (
 `id` INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY
, `url` varchar(1024) DEFAULT NULL
, `title` varchar(2000) DEFAULT NULL
, `embedding` blob NOT NULL
, VECTOR KEY `embedding` (`embedding`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
;

The model fromdual_llm_v1 can be downloaded here.

This diagram from the MariaDB Foundation shows roughly how the whole thing works:

Training the AI

The database is then trained: The URL is taken …

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.1 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 12 Aug 2024 16:25:13 +0200

FromDual has the pleasure to announce the release of the new version 2.3.1 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.1

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.3.1.tar.gz
shell> rm -f brman
shell> ln -s brman-2.3.1 brman

Changes in FromDual Backup and Recovery Manager 2.3.1

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version
shell> rman --version

General

Testing: LXC library updated.
Testing: Test preparation improved.
General: CHANGELOG updated.
General: rc made unique.
Documentation: Documentation improved.
Documentation: Create user documented.
Documentation: New release notes added to documentation.
Packaging: Distro Debian 10 removed.

FromDual Backup Manager (bman)

Utility mariabackup replaced by mariadb-backup.
Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.

FromDual Recovery Manager (rman)

Branch guessing improved.
Code refactored.
No realistic password is used any more.
Error message for missing client utilities made more distro specific.
Typo …

Partial physical database restore for MariaDB and MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 02 Jul 2024 18:13:47 +0000

What is it about?

When describing backup and restore scenarios, a full backup and a full restore of the database instance (mariadbd/mysqld) are usually assumed. This means that the entire database instance, including all databases (schemas), is backed up and restored.

In practice, however, the situation is often different: An entire database instance is not to be restored, but only individual databases or even individual tables, because only these have broken.

In many cases, this can be done quite easily with the tools mariadb-dump/mariadb or mysqldump/mysql (logical backup). However, if the database or table is very large, the restore will not be completed in a reasonable time (some minutes to a few hours).

This is exactly where the so-called partial physical restore comes into play. Partial stands for one or more tables (or an entire database), physical for: Individual SQL statements are not executed, but the data files are physically restored. In this scenario, very large amounts of data can be restored very quickly, provided the appropriate infrastructure is in place. Rule of thumb: On fat hardware: 1 Tbyte per hour. In this way, database restores can be carried out very quickly.

MariaDB and MySQL already offer this functionality out of the box. The mechanism is reasonably practicable for individual tables (see Restore partial backup). For entire databases with possibly dozens or hundreds of tables, however, the on-board mechanism is very cumbersome and error-prone.

Use case

This is exactly where the new functionality of the FromDual Backup and Recovery Manager (brman) v2.3.0 comes into play: it considerably simplifies the partial physical database restore.

A second scenario in which this new functionality can also be used is when moving a large database from one database instance to another (e.g. from Dev to Prod).

Preparations for the partial physical database restore

In order to be able to restore a database, a clean backup must of course first be available. This can either be created with the FromDual …

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 25 Jun 2024 15:12:02 +0200

FromDual has the pleasure to announce the release of the new version 2.3.0 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.3.0

shell> cd ${HOME}/product
shell> tar xf /download/brman-2.3.0.tar.gz
shell> rm -f brman
shell> ln -s brman-2.3.0 brman

Changes in FromDual Backup and Recovery Manager 2.3.0

This release is a new major release. The most important new feature is the partial physical restore. We have tried to maintain backward-compatibility with the 1.2, 2.0, 2.1 and 2.2 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version
shell> bman --version
shell> rman --version

General

Packaging: Release el9 added and debian format increase from 9 to 10 because of debian12.
Documentation prepared for Asciidoctor.
Distribtion dependent things fixed.
Build script improved.
Typos fixed.
All print replaced by standard PHP echo commands.
Redhat replace by Red Hat and CentOS by Rocky Linux.
Copyright year updated to 2024.
Library myEnv updated (getDistribution, lsb_release removed).

FromDual Backup Manager (bman)

Bug in sftp transfer for archiving files fixed.
Also create subdirectories only once if necessary in physical backup.
Archive location directory is only created once in physical backup. This avoids errors and makes the backup slightly faster.
In …

Shrinking the InnoDB system tablespace

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 14:45:32 +0000

One feature that really excited me in the new MariaDB 11.4 LTS release is the shrinking of the system tablespace (ibdata1). I have been eagerly waiting for this feature since around 2006 and now it has finally arrived with MariaDB 11.4. Actually, this feature has been available since MariaDB 11.2 IR (June 2023).

Unfortunately, the announcement of this feature came a little too short. In the MariaDB release notes it says succinctly:

The InnoDB system tablespace is now shrunk by reclaiming unused space at startup (MDEV-14795)

From the MariaDB 11.2.0 Release Notes.

The reasons why this file can grow immeasurably have actually been known for a long time and the measures against it are also clear (see literature). But again and again we see MariaDB users out in the field who didn’t realise the problem or realised it too late and are now left with an ibdata1 file that is far too large…

How can the problem be provoked?

The problem can be provoked by creating a table in the system tablespace:

SQL> SET global innodb_file_per_table = off;

SQL> CREATE TABLE `test` (
 `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `data` varchar(128) DEFAULT NULL,
 `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
 PRIMARY KEY (`id`)
) ENGINE=InnoDB;

and then filling it with data:

SQL> INSERT INTO test
SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW()
;

SQL> INSERT INTO test
SELECT NULL, 'Some data to provoke huge data growth in system tablespace', NOW()
 FROM test LIMIT 1000000
;
...

While the table is being filled, you can observe how the file ibdata1 swells on the file system:

$ while [ 1 ] ; do ll -h ibdata1 ; sleep 5 ; done
-rw-rw---- 1 mysql mysql 12M Jun 2 13:57 ibdata1
-rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1
-rw-rw---- 1 mysql mysql 76M Jun 12 13:57 ibdata1
-rw-rw---- 1 mysql mysql 140M Jun 12 13:58 ibdata1
-rw-rw---- 1 mysql mysql 204M Jun 12 13:58 ibdata1
-rw-rw---- 1 mysql mysql 268M Jun 12 13:58 ibdata1
-rw-rw---- 1 …

MDEV-34412

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 12:05:44 +0200

https://jira.mariadb.org/browse/MDEV-34412

Migration Problem mysql 5.7 to mariadb/galera

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 11:53:57 +0200

Hi dcz01

No it is not solved… It is closed and “not a bug”… I try to intervene

Regards, Oli

Possibly a bug

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 17 Jun 2024 11:29:33 +0200

Hello dcz01

I would say, this is probably a bug. I can reproduce it on my system (MySQL 5.7 Master and MariaDB 10.6 Galera Cluster).

Please contact your MariaDB support contact… If you need assistance we are happy to sell you our remote-DBA services…

Regards, Oli

How to downgrade MariaDB or MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 21 May 2024 09:54:14 +0000

On this page we have summarised information about downgrading a MariaDB/MySQL database. We do not cover migrating from MySQL to MariaDB or vice versa. For migrations (sidegrade) please look here: MySQL - MariaDB migration and here: Migration between MySQL/Percona Server and MariaDB.

The downgrade process tends to be neglected because nobody really wants to do it. So we are in swampy territory here. Unforeseen situations can occur very easily and good preparation is essential.

I can think of the following possible options for the downgrade:

Restore the backup that was made BEFORE the upgrade began.
Binary inplace downgrade (which is NOT supported in many cases).
Dump (mariadb-dump/mysqldump) of the upgraded database and restore (mariadb/mysql) to the old version (logical backup/restore).
Master/slave replication (with Statement Based Replication (SBR) or Row Based Replication (RBR)) and fallback to the slave with the old version.
Partial Backup/Restore with mariadb-backup/xtrabackup (Transportable Tablespace (TTS) method).

All these options have their advantages and disadvantages and must be carefully tested in advance.

Downgrade MariaDB

Downgrading MariaDB is not officially supported between major versions.

First sentence in Downgrading between Major Versions of MariaDB by MariaDB Foundation.

FromDual says: But it usually works. Downgrading from minor versions practically always works.

Physical (in-place) downgrade NOT possible:

From version	To version
11.0 or later	to 10.4 or earlier: NOT possible
11.0 or later	to 10.5 or later: Possible with restrictions
10.8 or later	unclear description
10.5	to 10.4: With restrictions
10.5	to 10.3: With restrictions or problems
10.3	to 10.2: With restrictions …

FromDual Ops Center 1.2.2 for MariaDB and MySQL databases has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 29 Apr 2024 17:41:04 +0200

FromDual is pleased to announce the release of the new version 1.2.2 of the popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBAs and system administrators to better manage their MariaDB and MySQL databases and Galera Cluster farms. Ops Center makes life easier for DBAs and Admins!

The main purpose of the Ops Center is to help you with your daily MariaDB and MySQL operations. More information about FromDual Ops Center can be found in the General Information section.

Download now

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download page. How to install and use focmm is documented in the Ops Center User Guide.

In the unlikely event that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for focmm).

Any feedback, comments and testimonials are also welcome! Please send them to [feedback@fromdual.com](mailto:feedback@fromdual.com?Subject=Feedback for focmm “Feedback for focmm”).

Installing Ops Center 1.2.2

Information on how to install FromDual Ops Center can be found in the Ops Center User Guide.

Upgrading from 0.9.x to 1.2.2

The upgrade from 0.9.x to 1.2.2 should be automatic. Please make a backup of your Ops Center database instance before upgrading! See also Upgrading.

Important: In some cases the directory focmm/tmp/start_jobs.lock is missing. In this case jobs will not start. Please check the log file under focmm/log/start_jobs.log and create the folder accordingly. There should also be a file called pid in this folder. Create this file if it does not exist.

Downgrading from 1.2.2

Downgrading from 1.2.2 to 1.2.1 should work.

Changes in Ops Center 1.2.2

Machine

Machine information is automatically gathered after first successful check now.
Machine function calls optimized.
Ssh key check added for machine. …

Ops Center Release Notes

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 29 Apr 2024 16:42:39 +0200

Ops Center 1.2.2 Release Notes, Release Date: 25 April 2024
Ops Center 1.2.1 Release Notes, Release Date: 7 Februray 2023
Ops Center 1.2.0 Release Notes, Release Date: 17 January 2023
Ops Center 1.1.0 Release Notes, Release Date: 7 May 2021
Ops Center 1.0.0 Release Notes, Release Date: 7 May 2020
Ops Center 0.9.3 Release Notes, Release Date: 18 February 2020
Ops Center 0.9.2 Release Notes, Release Date: 13 August 2019
Ops Center 0.9.1 Release Notes, Release Date: 7 May 2019
Ops Center 0.9.0 Release Notes, Release Date: 16 April 2019
Ops Center 0.3.0 Release Notes, Release Date: 13 January 2016
Ops Center 0.2.0 Release Notes, Release Date: 29 May 2014
Ops Center 0.1.0 Release Notes, Release Date: 02 January 2014
Ops Center 0.0.1 Release Notes, Release Date: 01 January 2014

dbstat for MariaDB after one month of productive use

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 26 Apr 2024 14:13:55 +0000

Review

After we introduced dbstat for MariaDB (and MySQL) a good 5 weeks ago, we naturally also rolled it out on our systems to test the behaviour in daily use (eat your own dog food).

This went quite well until we came up with the idea of activating dbstat on the passive dbstat node on our MariaDB active/passive master/master replication cluster (a similar situation would also occur with a Galera cluster). We realised that the design of dbstat still had potential. After this problem was fixed (v0.0.2 and v0.0.3) and the problem of how to activate events on master AND slave was solved (MDEV-33782: Event is always disabled on slave), everything seemed fine at first glance. Unfortunately, we did not realise that the data also had to be adjusted. As a result, our replication came to a complete stop over the Easter holidays, which then led to another problem when catching up (MDEV-33923: MariaDB parallel replication causes Foreign Key errors).

After this minor incident was also resolved, dbstat has been running flawlessly on our MariaDB master/master replication cluster ever since… The product dbstat is open source (GPLv2) and can be downloaded from GitHub.

One month later

Databases should NOT grow over time but only over the number of {customers, products, etc.} once the desired equilibrium (steady state) is reached. In our dbstat installation, we have set this equilibrium state to 30 days. So it is now time that the size of dbstat stabilises and the database stops growing…

It would also be interesting to understand what practical use dbstat has. That is why we have now set to work and are trying to analyse the results of dbstat.

Here is an overview of the 11 current running database events:

SQL> SELECT db, name, definer, CONCAT(interval_value, ' ', interval_field) AS 'interval'
 , last_executed, ends, status
 FROM mysql.event
 ORDER BY db, name ASC …

MariaDB's parallel replication to catch up

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 09 Apr 2024 10:53:32 +0000

Due to an application error, our replication stopped for 5 days (over Easter). After the problem was solved, the replication was supposed to catch up, which turned out to be very slow. All the usual tricks (innodb_flush_log_at_trx_commit, sync_binlog, etc.) had already been exhausted. So we tried our hand at parallel replication of the MariaDB server.

Parallel replication is deactivated by default:

SQL> SHOW GLOBAL VARIABLES LIKE '%parallel%';
+-------------------------------+------------+
| Variable_name | Value |
+-------------------------------+------------+
| slave_domain_parallel_threads | 0 |
| slave_parallel_max_queued | 131072 |
| slave_parallel_mode | optimistic |
| slave_parallel_threads | 0 |
| slave_parallel_workers | 0 |
+-------------------------------+------------+

Parallel replication is activated by setting the server variables slave_parallel_threads:

SQL> SET GLOBAL slave_parallel_threads = 8;
ERROR 1198 (HY000): This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first

However, this must be done when replication is stopped:

SQL> STOP SLAVE;
SQL> SET GLOBAL slave_parallel_threads = 8;
SQL> START SLAVE;

Replication then caught up a little faster. However, as we were impatient, we tried to make it even faster. With the command:

SQL> SHOW SLAVE STATUS<br>G
...
Slave_SQL_Running_State: Waiting for room in worker thread event queue
...

we found the following message. You would also see it using the SHOW PROCESSLIST command:

SQL> SHOW PROCESSLIST;
+--------+-------------+- ... -+-----------+------+-----------------------------------------------+- ... -+
| Id | User | ... | Command | Time | State | ... |
+--------+-------------+- ... -+-----------+------+-----------------------------------------------+- ... -+
... ... …

Building MariaDB Server from the sources

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 05 Apr 2024 08:47:13 +0000

Recently I had to test a new MariaDB feature that was developed at our request (MDEV-33782). To test this feature I had to build the MariaDB server myself from source, which I have not done for a long time. So a new challenge, especially with CMake…

I followed the MariaDB documentation Get, Build and Test Latest MariaDB the Lazy Way to build the server.

On Ubuntu 22.04 it did not work for me, for reasons unknown to me. So I cloned an Ubuntu 23.04 (Lunar Lobster) LXC container and built the MariaDB server in it.

To make the whole thing work, however, the package sources had to be added to the file /etc/apt/sources.list in the container first:

deb-src http://de.archive.ubuntu.com/ubuntu lunar main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-updates main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-security main restricted universe multiverse
deb-src http://de.archive.ubuntu.com/ubuntu lunar-backports main restricted universe multiverse

Then we proceeded according to the instructions:

shell> apt install build-essential bison
shell> apt build-dep mariadb-server

The corresponding branch was cloned:

shell> # git clone https://github.com/andremralves/server.git mariadb-MDEV-33782
shell> # git branch --all
shell> git clone --branch MDEV-33782 --single-branch https://github.com/andremralves/server.git mariadb-MDEV-33782
shell> cd mariadb-MDEV-33782
shell> # git checkout 11.5

and then the server was build. This took about 20 minutes on my old machine. CMake still ran into an error, which was solved by installing the corresponding package (MDEV-33815):

shell> apt install libgnutls28-dev
shell> cmake . -DBUILD_CONFIG=mysql_release && make -j8

The tests were executed:

shell> cd mysql-test
shell> ./mtr rpl.rpl_create_drop_event
Logging: ./mtr rpl.rpl_create_drop_event
VS config:
vardir: /root/mariadb-MDEV-33782/mysql-test/var
Checking leftover processes...
Removing old var directory...
Creating var …

MaxScale configuration synchronisation

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 04 Apr 2024 09:53:46 +0000

Overview

A feature that I recently discovered while browsing is the MaxScale configuration synchronisation functionality.

This is not primarily about a MariaDB replication cluster or a MariaDB Galera cluster, but about a cluster consisting of two or more MaxScale nodes. Or more precisely, the exchange of the configuration between these MaxScale nodes.

Pon Suresh Pandian has already written a blog article about this feature in 2022, which is even more detailed than this post here.

Preparations

An Incus container environment was prepared, consisting of 3 database containers (deb12-n1 (10.139.158.33), deb12-n2 (10.139.158.178), deb12-n3 (10.139.158.39)) and 2 MaxScale containers (deb12-mxs1 (10.139.158.66), deb12-mxs2 (10.139.158.174)). The database version is a MariaDB 10.11.6 from the Debian repository and MaxScale was downloaded in version 22.08.5 from the MariaDB plc website.

The database configuration looks similar for all 3 nodes:

#
# /etc/mysql/mariadb.conf.d/99-fromdual.cnf
#

[server]

server_id = 1
log_bin = deb12-n1-binlog
binlog_format = row
bind_address = *
proxy_protocol_networks = ::1, 10.139.158.0/24, localhost
gtid_strict_mode = on
log_slave_updates = on
skip_name_resolve = on

The MaxScale nodes were built as described in the article Sharding with MariaDB MaxScale.

The maxscale_admin user has exactly the same rights as described there, the maxscale_monitor user has the following rights:

RELOAD, SUPER, REPLICATION SLAVE, READ_ONLY ADMIN

Sharding with MariaDB MaxScale

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 19 Mar 2024 17:02:58 +0000

Overview

This feature should more or less work with MariaDB MaxScale 6.x.y, 22.08.x, 23.02.x, 23.08.x and 24.02.x. We have tested it with the latest MaxScale version 23.08.05, as we encountered problems with an older version (MXS-5026).

shell> maxscale --version
MaxScale 23.08.5

We used MariaDB 10.11 as the database backend (shards).

Less than approx. 2% of all MariaDB installations known to us are what we technically understand by multi-tenant systems (each customer in its own database (also called a schema)).

This MariaDB MaxScale feature is therefore used relatively rarely and there is an increased risk of encountering bugs that no-one has come across before!

This feature is called SchemaRouter at MariadDB MaxScale and is still declared as beta quality (MXS-5025):

maxctrl> show module schemarouter
┌─────────────┬────────────────────────────────────────────────┐
│ Module │ schemarouter │
├─────────────┼────────────────────────────────────────────────┤
│ Type │ Router │
├─────────────┼────────────────────────────────────────────────┤
│ Version │ V1.0.0 │ …

dbstat for MariaDB (and MySQL)

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 14 Mar 2024 15:36:57 +0000

An idea that I have been thinking about for a long time and have now, thanks to a customer, finally tackled is dbstat for MariaDB/MySQL. The idea is based on sar/sysstat by Sebastien Godard:

sar - Collect, report, or save system activity information.

and Oracle Statspack:

Statspack is a performance tuning tool … to quickly gather detailed analysis of the performance of that database instance.

Functionality of `dbstat`

Although we have had the performance schema for some time, it does not cover some points that we see as a problem in practice and that are requested by customers:

The table_size module collects data on the growth of tables. This allows statements to be made about the growth of individual tables, databases, future MariaDB Catalogs or the entire instance. This is interesting for users who are using multi-tenant systems or are otherwise struggling with uncontrolled growth.
The processlist module takes a snapshot of the process list at regular intervals and saves it. This information is useful for post-mortem analyses if the user was too slow to save his process list or to understand how a problem has built up.
The problem is often caused by long-running transactions, row locks or metadata locks. These are recorded and saved by the trx_and_lck and metadata_lock modules. This means that we can see problems that we did not even notice before or we can see what led to the problem after the accident (analogous to a tachograph in a vehicle).
Another question that we sometimes encounter in practice is: When was which database variable changed and what did it look like before? This is covered by the global_variables module. Unfortunately, it is not possible to find out who changed the variable or why. Operational processes are required for this.
The last module, global_status, actually covers what …

MyEnv Release Notes

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 28 Feb 2024 16:51:26 +0100

MyEnv 3.0.0 Release Notes, Release Date: 23 March 2026
MyEnv 2.1.1 Release Notes, Release Date: 12 December 2025
MyEnv 2.1.0 Release Notes, Release Date: 28 February 2024
MyEnv 2.0.3 Release Notes, Release Date: 2 July 2021
MyEnv 2.0.2 Release Notes, Release Date: 17 January 2019
MyEnv 2.0.1 Release Notes, Release Date: 3 October 2018
MyEnv 2.0.0 Release Notes, Release Date: 15 March 2018
MyEnv 1.3.1 Release Notes, Release Date: 3 August 2016
MyEnv 1.3.0 Release Notes, Release Date: 31 January 2016
MyEnv 1.2.2 Release Notes, Release Date: 12 October 2015
MyEnv 1.2.1 Release Notes, Release Date: 27 July 2015
MyEnv 1.2.0 Release Notes, Release Date: 18 April 2015
MyEnv 1.1.4 Release Notes, Release Date: 11 February 2015
MyEnv 1.1.3 Release Notes: Skipped, see v1.1.4, Release Date: 30 January 2015
MyEnv 1.1.2 Release Notes, Release Date: 23 October 2014
MyEnv 1.1.1 Release Notes, Release Date: 8 September 2014
MyEnv 1.1.0 Release Notes, Release Date: 23 August 2014
MyEnv 1.0.5 Release Notes, Release Date: 13 June 2014
MyEnv 1.0.4 Release Notes, Release Date: 29 April 2014
MyEnv 1.0.3 Release Notes, Release Date: 25 April 2014
MyEnv 1.0.2 Release Notes, Release Date: 11 December 2013
MyEnv 1.0.1 Release Notes, Release Date: 5 November 2013
MyEnv 1.0 Release Notes, Release Date: 4 October 2013
MyEnv 0.4 Release notes: none, Release Date: 30 March 2011
MyEnv 0.3.6 Release notes: none, Release Date: 23 January 2011
MyEnv 0.3.5 Release notes: none, Release Date: 22 January 2011
MyEnv 0.3.4 Release notes: none, Release Date: 21 November 2010
MyEnv 0.3.3 Release notes: none, Release Date: 21 November 2010
MyEnv 0.3.2 Release notes: none, Release Date: 14 November 2010
MyEnv 0.3.1 Release notes: none, Release Date: 14 November 2010
MyEnv 0.3 Release notes: none, Release Date: 13 November 2010
MyEnv 0.2.7 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.6 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.5 Release notes: none, Release Date: 12 August 2010
MyEnv 0.2.4 Release notes: none, Release …

We build a data warehouse from the General Query Log

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 31 Jan 2024 16:41:13 +0100

The design of a data warehouse differs from relational design. Data warehouses are often designed according to the concept of the star schema.

When building a data warehouse, you usually put the cart before the horse:

What questions should my data warehouse be able to answer?
How do I have to design my model so that my questions can be answered easily?
Where do I get the data to populate the model?
How do I fill my model with the data?

For training purposes, we have investigated an issue that arises from time to time with our support team: The system suddenly and unexpectedly starts to behave unusually, nobody has done anything and nobody knows why. Example with a customer last week: The system starts to become unstable at 3 pm, is then restarted hard and then stabilises again from 4 pm…

The easiest thing to do in such a case would be to quickly look at the database with the SHOW PROCESSLIST command and then it often becomes immediately clear where the problem lies. But customers often forget this or they are not fast enough. The General Query Log was already switched on for this customer, so this would be a great case for our General Query Log Data Warehouse!

What questions should my data warehouse be able to answer?

The generic question for this problem should be something like: “Who or what caused my system to behave abnormally?”

In technical terms, the question would be something like:

Who: Which user or account was on the database with how many connections at the time in question? What was unusual about it?
What: Which queries were running in which schema on the system at the time in question? Which of these queries were unusual?

What should my model look like?

We can already derive some facts and dimensions from the question:

User or account (user + host)
Time
Connections
Schema
Queries

And this also results in 4 dimensions and the fact table:

Data source

Where the data comes from is relatively easy to answer in this case: The customer provides his General Query Logs or you can …

FromDual Performance Monitor for MariaDB 2.1.0 has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 09 Dec 2023 18:27:02 +0100

FromDual has the pleasure to announce the release of the new version 2.1.0 of its popular Database Performance Monitor for MariaDB and Galera Cluster fpmmm.

The FromDual Performance Monitor for MariaDB (fpmmm) enables DBAs and System Administrators to monitor and understand what is going on inside their MariaDB database instances and on the machines where the databases reside.

More detailed information you can find in the fpmmm Installation Guide.

Download

The new FromDual Performance Monitor for MariaDB (fpmmm) can be downloaded from here or you can use our FromDual repositories. How to install and use fpmmm is documented in the fpmmm Installation Guide.

In the inconceivable case that you find a bug in the FromDual Performance Monitor for MariaDB please report it to the FromDual Bug-tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for fpmmm).

Any feedback, statements and testimonials are welcome as well! Please send them [to us](mailto:feedback@fromdual.com?Subject=Feedback for fpmmm “Feedback for fpmmm”).

Monitoring as a Service (MaaS)

You do not want to set-up your Database monitoring yourself? No problem: Choose our MariaDB Monitoring as a Service (Maas) program to safe time and costs!

Installation of Performance Monitor 2.1.0

A complete guide on how to install FromDual Performance Monitor you can find in the fpmmm Installation Guide.

Upgrade of fpmmm tarball from 1.x to 2.1.0

There are some changes in the configuration file (fpmmm.conf):

The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
The key Methode was spelled wrong in the configuration file. It was renamed to Method.
The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tarballs follow this:

shell> cd /opt
shell> tar xf /download/fpmmm-2.1.0.tar.gz
shell> rm -f fpmmm
shell> ln -s fpmmm-2.1.0 fpmmm

Changes in …

InnoDB Deadlock on SELECT? Not possible! Or Is It?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Sat, 25 Nov 2023 16:58:41 +0000

Translated by deepl.com

Introduction

Two points in advance:

A deadlock is a state in which two different transactions are no longer able to continue working because each transaction holds a lock that the other transaction would need. Because both transactions are now waiting for the other transaction to release their locks, neither transaction will release their respective locks. And that would last forever. To avoid this, the MariaDB instance intervenes and kills the transaction that has done less work. The application then receives a deadlock error message of this type:
```
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction
```
A general mantra in the MariaDB ecosystem is that a SELECT does not cause locks (exception: FOR UPDATE or LOCK IN SHARE MODE) and therefore cannot be part of a deadlock.

The problem

A long-standing customer comes to the FromDual remote DBA team with a request to explain a deadlock situation:

Hello FromDual Team, I need your expertise on the subject of deadlocks. When would it suit you?

The situation is as follows: Transaction 1 consists of a simple INSERT. Transaction 2 consists of a SELECT. This should NOT actually cause a deadlock!

We first check the following points

Are all tables affected by these queries properly indexed? Yes, they are. The queries are all running perfectly!
Is the SELECT query possibly part of a larger transaction (NOT an auto-commit transaction) and therefore not the actual cause of the deadlock? No, it is not. They are auto-commit transactions.

What now? What else needs to be said for clarification: The SELECT is sent with a very high cadence, i.e. approx. every 5 ms!

It is clear that the INSERT generates locks. It is also displayed. But why does the SELECT command generate locks? These are also displayed!

So we try to break the problem down into individual steps.

The approach

The query looks like this:

SQL> SET @id = (SELECT id FROM test WHERE id = 3);

If we pack this query into an explicit transaction, we can even see …

Ops Center File Transfer Tool

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 16 Mar 2023 15:47:34 +0000

Upgrading MariaDB to the pre-latest minor release

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 08 Feb 2023 13:45:30 +0000

Scenario: MariaDB Community Server is release every 3 months. The exact date is not known but it is typically about 2 to 3 weeks after the MySQL/Oracle CPU.

We upgrade regularly customer systems, also their MariaDB Galera Cluster systems. Because we made in the past some very bad experience upgrading production MariaDB Galera Clusters short after release date we became a bit more careful. Our new policy is: We wait a few weeks before installing a new release.

This week we had the situation that on Monday a new MariaDB release came out and on Wednesday we had to do the Cluster upgrade. So only 2 days in between. Too short time for my taste. But we did not want to omit the upgrade because it was already more than 6 months since the last upgrade…

So we decided to install the pre-latest MariaDB release. Technically it means: Customer was on 10.6.9, 10.6.12 is out (and already available in the repositories) but we want to upgrade to 10.6.11. Additionally we also wanted to upgrade the O/S (Debian 11). So how to do this quite complex task? In short:

Set all MariaDB related packages to hold.
Upgrade all other packages including a reboot of the machine.
Unhold all MariaDB related packages.
Install specific MariaDB packages which are not the newest one.

dpkg -l | grep mariadb
ii libdbd-mariadb-perl 1.21-3 amd64 Perl5 database interface to the MariaDB/MySQL databases
ii libmariadb3:amd64 1:10.6.9+maria~deb11 amd64 MariaDB database client library
ii libmariadb3-compat 1:10.6.9+maria~deb11 amd64 MariaDB database client library MySQL compat package
ii libmariadbclient18 1:10.6.9+maria~deb11 amd64 Virtual package to satisfy external libmariadbclient18 depends
ii mariadb-client-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database client binaries
ii mariadb-client-core-10.6 1:10.6.9+maria~deb11 amd64 MariaDB database core client …

FromDual Ops Center 1.2.1 for MariaDB, MySQL and compatible databases has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 07 Feb 2023 17:08:38 +0100

FromDual has the pleasure to announce the release of the new version 1.2.1 of its popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The FromDual Ops Center for MariaDB and MySQL (focmm) helps DBA’s and System Administrators to better manage their MariaDB and MySQL databases and Galera Cluster farms. Ops Center makes DBA and Admins life easier!

The main task of Ops Center is to support you in your daily MariaDB and MySQL operation tasks. More information about FromDual Ops Center you can find under General Information.

Download

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download site. How to install and use focmm is documented in the Ops Center User Guide.

In the inconceivable case that you find a bug in the FromDual Ops Center for MariaDB and MySQL please report it to the FromDual bug tracker or just send us an [email](mailto:contact@fromdual.com?Subject=Bug report for focmm).

Any feedback, statements and testimonials are welcome as well! Please send them to [feedback@fromdual.com](mailto:feedback@fromdual.com?Subject=Feedback for focmm “Feedback for focmm”).

Installation of Ops Center 1.2.1

Information on how to install FromDual Ops Center you can find in the Ops Center User Guide.

Upgrade from 0.9.x to 1.2.1

Upgrade from 0.9.x to 1.2.1 should happen automatically. Please do a backup of your Ops Center database instance before you upgrade! Please also check Upgrading.

Important: In some cases the folder focmm/tmp/start_jobs.lock is missing. In this case jobs are not started. Please check the log file under focmm/log/start_jobs.log and create the folder accordingly. Further a file named pid should be located in this folder. Create also this file if it is not there.

Changes in Ops Center 1.2.1

Machine

ssh “Suspect machine” message fix advice command is now done with the right user and file.
Check error fixed and output made nicer.
Instance link added in machine overview and code …

FromDual Ops Center 1.2.0 for MariaDB, MySQL and compatible databases has been released

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 19 Jan 2023 12:01:02 +0100

FromDual has the pleasure to announce the release of the new version 1.2.0 of its popular FromDual Ops Center focmm, a Graphical User Interface (GUI) for MariaDB, MySQL and compatible databases.

The main task of Ops Center is to support you in your daily MariaDB and MySQL operation tasks. More information about FromDual Ops Center you can find under General Information.

Download

The new FromDual Ops Center for MariaDB and MySQL (focmm) can be downloaded from our download site. How to install and use focmm is documented in the Ops Center User Guide.

Any feedback, statements and testimonials are welcome as well! Please send them to [feedback@fromdual.com](mailto:feedback@fromdual.com?Subject=Feedback for focmm “Feedback for focmm”).

Installation of Ops Center 1.2.0

Information on how to install FromDual Ops Center you can find in the Ops Center User Guide.

Upgrade from 0.9.x to 1.2.0

Upgrade from 0.9.x to 1.2.0 should happen automatically. Please do a backup of your Ops Center database instance before you upgrade! Please also check Upgrading.

Changes in Ops Center 1.2.0

Machine

Delete machine error message improved.
Add machine made more user friendly.
Delete machine warning message made more verbose.
Error handling improved and error messages made more verbose in gathering machine information.

Instance

Create instance: InnoDB buffer pool size estimation fixed.
Remove -core- packages from installation selection.
Access keys added to show instance.
Instance and machine names are sorted in selection now.
Error handling improved in performance views.
Restart …

Systemd related stuff for MariaDB and MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 06 Jan 2023 19:05:06 +0100

Systemd Notizen

Systemd paged per default nach less. Das ist ungewohnt und doof, weil es gegen allgemeines Linux-Verhalten verstösst. Abhilfe schafft:

$ systemctl --no-pager status
$ alias systemctl='systemctl --no-pager'

Das selbe können wir auch verwenden, wenn wir bei MariaDB start/stop Problemen im Systemd Journal nachschauen müssen:

$ journalctl --no-pager -xe --unit=mariadb

Eine weitere Herausforderung könnte die Anpassung des MariaDB Unit Files sein:

$ update-alternatives --config editor
$ export SYSTEMD_EDITOR=vi
$ export EDITOR=vi
$ export VISUAL=vi
$ export SELECTED_EDITOR="/usr/bin/vim.basic"
# C-x C-e
# /etc/systemd/system/mariadb.service.d/override.conf
$ systemctl show mariadb | grep NOFILE
$ systemctl edit mariadb

[Service]

LimitNOFILE=12288

$ systemctl edit --full mariadb
$ systemctl daemon-reload
$ systemctl restart mariadb
$ systemctl cat mariadb
$ systemctl revert mariadb

Leider funktioniert auf meinem Kubuntu 16.04 das setzten der $EDITOR Variable nicht wie erwartet… Bei älteren Systemd Versionen ist die Funktion edit noch nicht implementiert.

InnoDB NUMA interleave with MariaDB

The MariaDB server system variable innodb_numa_interleave is currently not supported (any more). See: MariaDB MDEV-21399.

As an alternative the mysqld_safe option --numa-interleave (MariaDB Systemd Options) or the Systemd property NUMAPolicy can be used instead.

NUMAPolicy=interleave works from Systemd v243 onwards. You can find your current Systemd version as follows:

$ systemctl --version
systemd 245 (245.4-4ubuntu3.6)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

The NUMA Policy of the the MariaDB …

Comparing Optimizer Results

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 18 Nov 2022 10:59:18 +0000

A while ago I read a paper about Optimizer Benchmarks: How Good Are Query Optimizers, Really? by Viktor Leis from Fakultät für Informatik of the Technische Universität München.

Because we have from time to time problems with slow queries of customers especially after upgrading to new database releases it is interesting for me, how different optimizers cope with a query. Sometimes it is not clear to me why the query is slow or how I can make it faster. So the mentioned paper inspired me to compare the same query with the same dataset among different optimizers or optimizer versions. The most recent query from our customer we were testing against MariaDB 10.6, 10.9, 10.10, MySQL 8.0 and PostgreSQL 15.1.

The test data

The table we were using is our generic test table which many people already know from our MariaDB and MySQL trainings:

CREATE TABLE `test` (
 `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `data` varchar(128) DEFAULT NULL,
 `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(),
 PRIMARY KEY (`id`)
);

Loading the data from MySQL

How to dump the data from MySQL to load them into PostgreSQL I have already described earlier. This is about how to create the table and load the data into PostgreSQL:

# sudo su - postgres
# psql
# postgres=# SELECT VERSION();
# postgres=# <br>l
# postgres=# CREATE DATABASE test;
CREATE DATABASE
postgres=# <br>c test
postgres=# CREATE TABLE test (
 id SERIAL NOT NULL PRIMARY KEY,
 data VARCHAR(128) DEFAULT NULL,
 ts TIMESTAMP NOT NULL
);
test=# <br>dt+
 List of relations
 Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+----------+-------+----------+-------------+---------------+---------+-------------
 public | test | table | postgres | permanent | heap | 0 bytes |

# psql test < /tmp/test_dump.sql

The Query

The query we had problems with looked something like this:

WITH t AS (
SELECT SUM(id)+0.0 AS c, data AS pn
 FROM …

Migration of your data from one database to another

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 17 Nov 2022 17:01:44 +0000

Before you consider migrating your data from MySQL to another database you have to know which objects have to be migrated.

With this query you will find the objects to consider:

SELECT TABLE_SCHEMA AS `SCHEMA`, IF(TABLE_TYPE = 'BASE TABLE', 'TABLE', TABLE_TYPE) AS OBJECT_TYPE, TABLE_NAME AS `OBJECT_NAME`
 , IFNULL(ENGINE, '') AS ENGINE, IFNULL(TABLE_ROWS, '') AS `ROWS`
 , IFNULL(DATA_LENGTH, '') AS DATA_SIZE, IFNULL(INDEX_LENGTH, '') AS INDEX_SIZE
 FROM information_schema.tables
 WHERE TABLE_SCHEMA NOT IN ('information_schema', 'mysql', 'performance_schema', 'sys')
UNION
SELECT '', IF(is_role = 'Y', 'ROLE', 'USER'), CONCAT("'", user, "'", '@', "'", host, "'") AS OBJECT_TYPE, '', '', '', ''
 FROM mysql.user
UNION
SELECT db, type, name, '', '', '', ''
 FROM mysql.proc
 WHERE db NOT IN ('sys', 'information_schema', 'performance_schema', 'mysql')
UNION
SELECT db, 'EVENT', name, '', '', '', ''
 FROM mysql.event
UNION
SELECT trigger_schema, 'TRIGGER', trigger_name, '', '', '', ''
 FROM information_schema.triggers
UNION
SELECT db, 'EVENT', name, '', '', '', ''
 FROM mysql.event
ORDER BY `SCHEMA` ASC, OBJECT_TYPE ASC, OBJECT_NAME ASC
;

+--------+-------------+---------------------------+--------+---------+-----------+------------+
| SCHEMA | OBJECT_TYPE | OBJECT_NAME | ENGINE | ROWS | DATA_SIZE | INDEX_SIZE |
+--------+-------------+---------------------------+--------+---------+-----------+------------+
| | ROLE | 'test_r'@'' | | | | |
| | USER | 'app'@'%' | | | | |
| | USER | 'app'@'127.0.0.1' | | | | |
| | USER | 'focmm'@'127.0.0.1' | | | | |
| | USER | 'test'@'localhost' | | | | |
| sbtest | TABLE | sbtest1 | InnoDB | 9680 | …

Graphical User Interfaces (GUI) for MariaDB and MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 04 Nov 2022 10:10:04 +0000

If you do not feel at home on the command line, you may prefer to use a graphical user interface (GUI). These also have advantages, for example when displaying data or structures, or they can simplify and therefore speed up work processes.

There are two main types of graphical user interfaces: On the one hand, the Integrated Development Environments (IDE), and on the other hand, tools that support the administration of the database.

This list contains all known graphical user interfaces for MariaDB and MySQL:

Name	O/S	Distro	License	Audience	other DBMS	ER- Diag.	Client Type	MySQL Support	MariaDB Support	Download
dbForge Studio	Linux, macOS, Windows	Deb, Ubu, Fed, RHEL	Proprietary	Dev, Admin, Anal.	x		Fat	x	x	Download
MySQL Workbench	Linux, macOS, Windows	Deb, Ubu, RHEL. Fed, OL	GPLv2, Prop.	Dev, Admin, Arch, Anal.		x	Fat	x	(x)	Download
phpMyAdmin	Linux	Deb, Ubu, Mint, Fed, RHEL, CentOS, Rocky, Alma, Arch openSuSE	OpenSource	User			Web	x	x	Download
Navicat	Linux, macOS, Windows	Deb, Ubu, openSuSE, CentOS, Fedora	Proprietary	Dev, Admin	x		Fat	x	x	Download
Valentina Studio	Linux, macOS, Windows	Deb, Ubu	Proprietary	Dev	x		Fat	x	x	Download
DBeaver	Linux, macOS, Windows	Deb, Ubu, Mint, Arch, RHEL	Apache, Prop.	Dev	x	x	Fat	x	x	Download
BeeKeeper Studio	Linux, macOS, Windows	Deb, Ubu, Fed, Arch	GPLv3, Proprietary	Dev	x		Fat	x	x	Download
DataGrip	Linux, macOS, Windows	Deb, Ubu, RHEL	Proprietary	Dev, Admin, Anal.	x		Fat	x	x	Download
SQuirreL SQL	Linux	Deb, Ubu, Fed, Mint, CentOS	GPLv2+, LGPLv2.1+	Dev, User	x		Fat	x	x	Download
Adminer	Linux	Deb, Ubu, Arch	OpenSource?	Dev, Admin			Web	x	x	Download
Toad Edge	Windows		Proprietary	Dev, Admin	x		Fat	x	x
Oracle SQL Developer	Windows		Proprietary	Dev	x		Fat	x	(x)
FromDual Ops Center	Linux	Deb, Ubu, RHEL	Proprietary	Admin			Web	(x)	x	Download
HeidSQL	Windows, Linux+Wine		GPLv2	Dev	x		Fat	x	x	Download
SQLPro	macOS		Proprietary	Dev			Fat	x	(x)	Download
Sequel Pro	macOS		OpenSource?	Dev, Beginners			Fat	x	(x)	Download
Querious	macOS …

EditLine vs ReadLine on FromDual GmbH

MariaDB/MySQL Environment MyEnv 3.0.0 has been released

Upgrade from 2.x to 3.0

Changes in MyEnv 3.0.0

MyEnv

MyEnv Installer

MyEnv Utilities

PostgreSQL

General

FromDual Performance Monitor 2.2.1 has been released

Download

Monitoring as a Service (MaaS)

Installation of Performance Monitor 2.2.1

Upgrade of fpmmm tar ball from 1.x to 2.2.1

Changes in FromDual Performance Monitor 2.2.1

What is the quickest way to load data into the database?

Variant 1: INSERT and DELETE (simplest form)

MariaDB has broken the concept of dynamically configurable buffer pools!

Problem description

How did it work before?

How much space does NULL need?

Someone is deleting my shared memory segments!

Linux System User

FromDual Backup and Recovery Manager

Load CSV files into the database

Loading the data with MariaDB/MySQL

Attribute promotion and demotion in the MariaDB Galera Cluster

Use case of the customer

What does the MariaDB documentation say about this?

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.2 has been released

Upgrade from 2.x to 2.3.2

Changes in FromDual Backup and Recovery Manager 2.3.2

General

FromDual Backup Manager (bman)

FromDual Recovery Manager (rman)

MariaDB Honeypot

Preparing the honeypot

How does Galera Cluster behave with many nodes?

Throughput as a function of the number of Galera nodes

Throughput as a function of the number of connections

Model v2 slightly improves results

Playing with MariaDB Vector for initial AI tests

Downloading the MariaDB database with vector functionality

The model

Training the AI

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.1 has been released

Upgrade from 2.x to 2.3.1

Changes in FromDual Backup and Recovery Manager 2.3.1

General

FromDual Backup Manager (bman)

FromDual Recovery Manager (rman)

Partial physical database restore for MariaDB and MySQL

What is it about?

Use case

Preparations for the partial physical database restore

FromDual Backup and Recovery Manager for MariaDB and MySQL 2.3.0 has been released

Upgrade from 2.x to 2.3.0

Changes in FromDual Backup and Recovery Manager 2.3.0

General

FromDual Backup Manager (bman)

Shrinking the InnoDB system tablespace

How can the problem be provoked?

MDEV-34412

Migration Problem mysql 5.7 to mariadb/galera

Possibly a bug

How to downgrade MariaDB or MySQL

Downgrade MariaDB

Physical (in-place) downgrade NOT possible:

FromDual Ops Center 1.2.2 for MariaDB and MySQL databases has been released

Download now

Installing Ops Center 1.2.2

Upgrading from 0.9.x to 1.2.2

Downgrading from 1.2.2

Changes in Ops Center 1.2.2

Machine

Ops Center Release Notes

dbstat for MariaDB after one month of productive use

Table of contents

Review

One month later

Variant 1: `INSERT` and `DELETE` (simplest form)

Functionality of `dbstat`