Sammlung von Newsfeeds

Auch im Jahr 2023 haben Sie wieder die Möglichkeit, sich bei unseren 3 Schulungspartnern in Essen, Köln und Berlin MariaDB und MySQL seitig fit zu machen.

Folgende Termine können wir Ihnen für das Jahr 2023 schon jetzt anbieten:

• 30. Januar bis 3. Februar 2023: MariaDB/MySQL für Fortgeschrittene, GFU Schulungszentrum, Köln
• 27. Februar bis 3. März 2023: MariaDB/MySQL für Fortgeschrittene, Heinlein Akademie, Berlin
• 20. bis 24. März 2023: MariaDB/MySQL für Fortgeschrittene, Linuxhotel, Essen
• 27. bis 29. März 2023: Galera Cluster für MariaDB/MySQL, GFU Schulungszentrum, Köln
• 24. bis 28. April 2023: MariaDB/MySQL für Fortgeschrittene, GFU Schulungszentrum, Köln
• 3. bis 5. Mai 2023: Galera Cluster für MariaDB/MySQL, Linuxhotel, Essen
• 15. bis 17. Mai 2023: Galera Cluster für MariaDB/MySQL, Heinlein Akademie, Berlin
• 5. bis 7. Juni 2023: Galera Cluster für MariaDB/MySQL, GFU Schulungszentrum, Köln
• 24. bis 28. Juli 2023: MariaDB/MySQL für Fortgeschrittene, GFU Schulungszentrum, Köln
• 11. bis 15. September 2023: MariaDB/MySQL für Fortgeschrittene, Heinlein Akademie, Berlin
• 25. bis 27. September 2023: Galera Cluster für MariaDB/MySQL, GFU Schulungszentrum, Köln
• 23. bis 27. Oktober 2023: MariaDB/MySQL für Fortgeschrittene, GFU Schulungszentrum, Köln
• 6. bis 8. November 2023: Galera Cluster für MariaDB/MySQL, Linuxhotel, Essen
• 13. bis 15. November 2023: Galera Cluster für MariaDB/MySQL, Heinlein Akademie, Berlin
• 27. November bis 1. Dezember 2023: MariaDB/MySQL für Fortgeschrittene, Linuxhotel, Essen
• 4. bis 6. Dezember 2023: Galera Cluster für MariaDB/MySQL, GFU Schulungszentrum, Köln

Diese Schulungen können online gebucht werden.

Weitere Termine können während des Jahres noch hinzukommen.

Bei Fragen, oder wenn Sie Spezialwünsche haben, stehen wir Ihnen gerne mit Rat und Tat zur Seite! Zögern Sie nicht, uns zu kontaktieren, wir helfen Ihnen gerne per eMail weiter.

Bei allen weiteren Galera, MariaDB und MySQL Problemen oder Fragen unterstützen wir Sie natürlich ebenfalls gerne!

Mit freundlichen Grüssen,
Ihr FromDual Team

Quellen: Bild von StartupStockPhotos auf Pixabay

Taxonomy upgrade extras: schulungtrainingseminar2023mysqlmariadbgaleramysql schulungmariadb schulunggalera schulung

A while ago I read a paper about Optimizer Benchmarks: How Good Are Query Optimizers, Really? by Viktor Leis from Fakultät für Informatik of the Technische Universität München.

Because we have from time to time problems with slow queries of customers especially after upgrading to new database releases it is interesting for me, how different optimizers cope with a query. Sometimes it is not clear to me why the query is slow or how I can make it faster. So the mentioned paper inspired me to compare the same query with the same dataset among different optimizers or optimizer versions. The most recent query from our customer we were testing against MariaDB 10.6, 10.9, 10.10, MySQL 8.0 and PostgreSQL 15.1. The test data

The table we were using is our generic test table which many people already know from our MariaDB and MySQL trainings:

CREATE TABLE `test` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `data` varchar(128) DEFAULT NULL, `ts` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), PRIMARY KEY (`id`) );
Loading the data from MySQL

How to dump the data from MySQL to load them into PostgreSQL I have already described earlier. This is about how to create the table and load the data into PostgreSQL:

# sudo su - postgres # psql # postgres=# SELECT VERSION(); # postgres=# \l # postgres=# CREATE DATABASE test; CREATE DATABASE postgres=# \c test postgres=# CREATE TABLE test ( id SERIAL NOT NULL PRIMARY KEY, data VARCHAR(128) DEFAULT NULL, ts TIMESTAMP NOT NULL ); test=# \dt+ List of relations Schema | Name | Type | Owner | Persistence | Access method | Size | Description --------+----------+-------+----------+-------------+---------------+---------+------------- public | test | table | postgres | permanent | heap | 0 bytes | # psql test < /tmp/test_dump.sql
The Query

The query we had problems with looked something like this:

WITH t AS ( SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data -- MariaDB 10.6: 0.66s -- MariaDB 10.9: 0.37s -- Competing product: 0.70s ) SELECT 'AAA' AS label, ((SELECT c FROM t WHERE pn = 'ab06d9ca0e5eb27e5bb2e5c0f345b782')+(SELECT c FROM t WHERE pn = 'ab0f8e861348995d1cbf9ec21d4bf52d'))/((SELECT c FROM t WHERE pn = 'ab134910f6808802cf44b587fb825ab5')+(SELECT c FROM t WHERE pn = 'ab1d6682feb41a97054e8c634ecd5552'))*100 AS value -- MariaDB 10.6: 2.04s -- MariaDB 10.9: 0.79s -- Competing product: 0.70s UNION SELECT 'BBB' AS label, (SELECT c FROM t WHERE pn = 'ab2516b576d7463f106b5671bdac6d32')/((SELECT c FROM t WHERE pn = 'ab28e05f4fa736f4738a2ff6cf494f04')+(SELECT c FROM t WHERE pn = 'ab31ab6a7d52fee49990d63dee58371b'))*100 AS value -- MariaDB 10.6: 3.54s -- MariaDB 10.9: 1.38s -- Competing product: 0.70s UNION SELECT 'CCC' AS label, (SELECT c FROM t WHERE pn = 'ab390b211d7bb3cc19fc1f808fb1bc3a')/(SELECT c FROM t WHERE pn = 'ab45af9e4ca10d3a618d40ec3f230349')*100 AS value -- MariaDB 10.6: 4.58s -- MariaDB 10.9: 1.76s -- Competing product: 0.70s UNION SELECT 'DDD' AS label, ((SELECT c FROM t WHERE pn = 'ab531756624d8287d38eb5fe5809bdb4')+(SELECT c FROM t WHERE pn = 'ab5aaa230cc64c15a61f908aea1c3f9c'))/((SELECT c FROM t WHERE pn = 'ab60f7712cd176b6ac26040d69be0e11')+(SELECT c FROM t WHERE pn = 'ab674e464997f4916e7ed23942c53d23'))*100 AS value UNION SELECT 'EEE' AS label, (SELECT c FROM t WHERE pn = 'ab6cc3781441a08d67eb8039521c63bb')/((SELECT c FROM t WHERE pn = 'ab7612bca85db4f6c4638a658fbee646')+(SELECT c FROM t WHERE pn = 'ab7811882f520c2051c6761edf4c306f'))*100 AS value UNION SELECT 'FFF' AS label, (SELECT c FROM t WHERE pn = 'ab7f0bd3758fbdac0ff6abb24db166ad')/(SELECT c FROM t WHERE pn = 'ab87d7d598ad47ca5db3bf9e6cbdee48')*100 AS value -- MariaDB 10.6: 9.15s -- MariaDB 10.9: 3.51s -- Competing product: 0.71s ;
The results

Timing is enabled in PostgreSQL as follows:

postgres=# \timing

Then we were running the different queries against the different databases and versions:

 Q1Q2Q3Q4Q5Q6 MariaDB 10.6.110.14 s0.32 s0.79 s1.39 s1.78 s3.56 s MariaDB 10.9.40.14 s0.33 s0.80 s1.40 s1.80 s3.60 s MariaDB 10.10.20.14 s0.33 s0.80 s1.39 s1.78 s3.55 s MySQL 8.0.310.03 s0.77 s0.77 s0.77 s0.77 s0.77 s PostgreSQL 15.10.03 s0.04 s0.04 s0.04 s0.04 s0.04 s
The Queries Query 1: SELECT COUNT(*) FROM test;
Query 2 SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data;
Query 3 WITH t AS ( SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data ) SELECT 'AAA' AS label, ((SELECT c FROM t WHERE pn = 'ab06d9ca0e5eb27e5bb2e5c0f345b782')+(SELECT c FROM t WHERE pn = 'ab0f8e861348995d1cbf9ec21d4bf52d'))/((SELECT c FROM t WHERE pn = 'ab134910f6808802cf44b587fb825ab5')+(SELECT c FROM t WHERE pn = 'ab1d6682feb41a97054e8c634ecd5552'))*100 AS value;
Query 4 WITH t AS ( SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data ) SELECT 'AAA' AS label, ((SELECT c FROM t WHERE pn = 'ab06d9ca0e5eb27e5bb2e5c0f345b782')+(SELECT c FROM t WHERE pn = 'ab0f8e861348995d1cbf9ec21d4bf52d'))/((SELECT c FROM t WHERE pn = 'ab134910f6808802cf44b587fb825ab5')+(SELECT c FROM t WHERE pn = 'ab1d6682feb41a97054e8c634ecd5552'))*100 AS value UNION SELECT 'BBB' AS label, (SELECT c FROM t WHERE pn = 'ab2516b576d7463f106b5671bdac6d32')/((SELECT c FROM t WHERE pn = 'ab28e05f4fa736f4738a2ff6cf494f04')+(SELECT c FROM t WHERE pn = 'ab31ab6a7d52fee49990d63dee58371b'))*100 AS value;
Query 5 WITH t AS ( SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data ) SELECT 'AAA' AS label, ((SELECT c FROM t WHERE pn = 'ab06d9ca0e5eb27e5bb2e5c0f345b782')+(SELECT c FROM t WHERE pn = 'ab0f8e861348995d1cbf9ec21d4bf52d'))/((SELECT c FROM t WHERE pn = 'ab134910f6808802cf44b587fb825ab5')+(SELECT c FROM t WHERE pn = 'ab1d6682feb41a97054e8c634ecd5552'))*100 AS value UNION SELECT 'BBB' AS label, (SELECT c FROM t WHERE pn = 'ab2516b576d7463f106b5671bdac6d32')/((SELECT c FROM t WHERE pn = 'ab28e05f4fa736f4738a2ff6cf494f04')+(SELECT c FROM t WHERE pn = 'ab31ab6a7d52fee49990d63dee58371b'))*100 AS value UNION SELECT 'CCC' AS label, (SELECT c FROM t WHERE pn = 'ab390b211d7bb3cc19fc1f808fb1bc3a')/(SELECT c FROM t WHERE pn = 'ab45af9e4ca10d3a618d40ec3f230349')*100 AS value;
Query 6 WITH t AS ( SELECT SUM(id)+0.0 AS c, data AS pn FROM test WHERE ts BETWEEN '2022-11-22 00:00:00' AND '2022-11-22 23:59:59' AND data LIKE 'ab%' GROUP BY data ORDER BY data ) SELECT 'AAA' AS label, ((SELECT c FROM t WHERE pn = 'ab06d9ca0e5eb27e5bb2e5c0f345b782')+(SELECT c FROM t WHERE pn = 'ab0f8e861348995d1cbf9ec21d4bf52d'))/((SELECT c FROM t WHERE pn = 'ab134910f6808802cf44b587fb825ab5')+(SELECT c FROM t WHERE pn = 'ab1d6682feb41a97054e8c634ecd5552'))*100 AS value UNION SELECT 'BBB' AS label, (SELECT c FROM t WHERE pn = 'ab2516b576d7463f106b5671bdac6d32')/((SELECT c FROM t WHERE pn = 'ab28e05f4fa736f4738a2ff6cf494f04')+(SELECT c FROM t WHERE pn = 'ab31ab6a7d52fee49990d63dee58371b'))*100 AS value UNION SELECT 'CCC' AS label, (SELECT c FROM t WHERE pn = 'ab390b211d7bb3cc19fc1f808fb1bc3a')/(SELECT c FROM t WHERE pn = 'ab45af9e4ca10d3a618d40ec3f230349')*100 AS value UNION SELECT 'DDD' AS label, ((SELECT c FROM t WHERE pn = 'ab531756624d8287d38eb5fe5809bdb4')+(SELECT c FROM t WHERE pn = 'ab5aaa230cc64c15a61f908aea1c3f9c'))/((SELECT c FROM t WHERE pn = 'ab60f7712cd176b6ac26040d69be0e11')+(SELECT c FROM t WHERE pn = 'ab674e464997f4916e7ed23942c53d23'))*100 AS value UNION SELECT 'EEE' AS label, (SELECT c FROM t WHERE pn = 'ab6cc3781441a08d67eb8039521c63bb')/((SELECT c FROM t WHERE pn = 'ab7612bca85db4f6c4638a658fbee646')+(SELECT c FROM t WHERE pn = 'ab7811882f520c2051c6761edf4c306f'))*100 AS value UNION SELECT 'FFF' AS label, (SELECT c FROM t WHERE pn = 'ab7f0bd3758fbdac0ff6abb24db166ad')/(SELECT c FROM t WHERE pn = 'ab87d7d598ad47ca5db3bf9e6cbdee48')*100 AS value;
Conclusion

This little tests showed some results I have not expected:

• MariaDB Optimizer does some bad things on this query. OK, this was expected. Bug is reported: MDEV-30017.
• The originally very bad execution times from MariaDB 10.6 (see timing notes in the first query) were not reproducible any more the next day. I should consider more to run ANALYZE TABLE. I should have know this because it is not the first time I experienced this.
• We have a lot of variations between different measurements. This I should investigate a bit more later...
• In the MariaDB Server Fest 2022 yesterday I heard with MariaDB 11 Optimizer everything will become much better... Wait! there is something more I do not understand and which I did not expect:
• PostgreSQL response time is constantly dramatic better for these queries. I checked the docu but did not find anything beside their normal cache which would explain this (something similar to the MariaDB Query Cache for example). If somebody has a clue why they manage to respond so fast or why we respond so slow I would be happy for a hint...

Taxonomy upgrade extras: postgresqlOptimizerperformanceresponse timelatency

Before you consider migrating your data from MySQL to another database you have to know which objects have to be migrated.

With this query you will find the objects to consider:

SELECT TABLE_SCHEMA AS `SCHEMA`, IF(TABLE_TYPE = 'BASE TABLE', 'TABLE', TABLE_TYPE) AS OBJECT_TYPE, TABLE_NAME AS `OBJECT_NAME` , IFNULL(ENGINE, '') AS ENGINE, IFNULL(TABLE_ROWS, '') AS `ROWS` , IFNULL(DATA_LENGTH, '') AS DATA_SIZE, IFNULL(INDEX_LENGTH, '') AS INDEX_SIZE FROM information_schema.tables WHERE TABLE_SCHEMA NOT IN ('information_schema', 'mysql', 'performance_schema', 'sys') UNION SELECT '', IF(is_role = 'Y', 'ROLE', 'USER'), CONCAT("'", user, "'", '@', "'", host, "'") AS OBJECT_TYPE, '', '', '', '' FROM mysql.user UNION SELECT db, type, name, '', '', '', '' FROM mysql.proc WHERE db NOT IN ('sys', 'information_schema', 'performance_schema', 'mysql') UNION SELECT db, 'EVENT', name, '', '', '', '' FROM mysql.event UNION SELECT trigger_schema, 'TRIGGER', trigger_name, '', '', '', '' FROM information_schema.triggers UNION SELECT db, 'EVENT', name, '', '', '', '' FROM mysql.event ORDER BY `SCHEMA` ASC, OBJECT_TYPE ASC, OBJECT_NAME ASC ; +--------+-------------+---------------------------+--------+---------+-----------+------------+ | SCHEMA | OBJECT_TYPE | OBJECT_NAME | ENGINE | ROWS | DATA_SIZE | INDEX_SIZE | +--------+-------------+---------------------------+--------+---------+-----------+------------+ | | ROLE | 'test_r'@'' | | | | | | | USER | 'app'@'%' | | | | | | | USER | 'app'@'127.0.0.1' | | | | | | | USER | 'focmm'@'127.0.0.1' | | | | | | | USER | 'test'@'localhost' | | | | | | sbtest | TABLE | sbtest1 | InnoDB | 9680 | 2637824 | 163840 | | test | EVENT | myevent | | | | | | test | FUNCTION | format_time | | | | | | test | PROCEDURE | diagnostics | | | | | | test | TABLE | dt | InnoDB | 6 | 16384 | 0 | | test | TABLE | test | InnoDB | 1045044 | 63520768 | 0 | | test | TRIGGER | test_trigger | | | | | | test | VIEW | test_v | | | | | +--------+-------------+---------------------------+--------+---------+-----------+------------+

An easy way to dump all the object definitions (except users and roles) is the following command:

mysqldump --user=root --no-data test > /tmp/test_structure_dump.sql

If you want to dump your data for importing them into another SQL database this command can help:

mysqldump --user=root --skip-extended-insert --skip-lock-tables --no-create-info \ --where='id = id ' --skip-add-locks --skip-comments --skip-quote-names test test \ | grep -v '^/\*' | grep -v ^$ > /tmp/test_dump.sql
Taxonomy upgrade extras: migrationdatabase

Currently we are in the testing phase of the next release of our Ops Center (focmm). This testing is quite complex because it includes testing the interaction of various different components like a MariaDB Galera Cluster, a Galera Load Balancer, a virtual IP, etc.

Recently I was in the Linuxhotel for a Galera Cluster Training and there one of the other trainers was romanticising about Linux Containers and LXD. So I had a short look, if LXD could be useful to simplify our unit testing. Some old knowledge from a previous Docker PoC was quite helpful to start with...

Prepare a LXC container for Galera Load Balancer

This did NOT work as expected because the Debian image was lacking IPv4 addresses and I did not find on the quick how to change that. So I used the Ubuntu 22.04 image.

shell> lxc remote list shell> lxc image list images: ubuntu/22.04 amd64 shell> INSTANCE='qa-glb' shell> lxc launch images:ubuntu/jammy ${INSTANCE} shell> lxc list *glb* +--------+---------+-----------------------+----------------------------------------------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +--------+---------+-----------------------+----------------------------------------------+------------+-----------+ | qa-glb | RUNNING | 10.139.158.183 (eth0) | fd42:1730:178f:78c:216:3eff:fe3f:2948 (eth0) | PERSISTENT | 0 | +--------+---------+-----------------------+----------------------------------------------+------------+-----------+

Now we should remember the IPv4 address for later use. I am sure this can be done more elegant but for now this is fine...

Install Galera Load Balancer in the LXC container shell> lxc exec ${INSTANCE} -- /bin/bash container> apt-get update container> apt-get install wget container> wget https://support.fromdual.com/admin/download/glb_1.0.1-Ubuntu12.04-x86_64.deb container> apt-get install ./glb_1.0.1-Ubuntu12.04-x86_64.deb container> rm -f glb_1.0.1-Ubuntu12.04-x86_64.deb

Then we have to add the unit file:

# # /etc/systemd/system/glb.service # [Unit] Description=Galera Load Balancer Service After=network.target [Service] EnvironmentFile=/etc/default/glbd Type=simple ExecStart=/usr/local/sbin/glbd --daemon --threads $THREADS --max_conn $MAX_CONN $OTHER_OPTIONS --control $CONTROL_ADDR $LISTEN_ADDR $DEFAULT_TARGETS [Install] WantedBy=multi-user.target

enable the unit file, configure the Galera Load Balancer and start it:

container> systemctl enable glb # Galera Load Balancer Configuration # Redhat: /etc/sysconfig/glbd # Debian: /etc/default/glbd LISTEN_ADDR="3306" CONTROL_ADDR="10.139.158.183:8011" CONTROL_FIFO="/var/run/glbd.fifo" THREADS="2" MAX_CONN=151 DEFAULT_TARGETS="10.139.158.1:3330:1 10.139.158.1:3331:1 10.139.158.1:3332:1" OTHER_OPTIONS="--round" container> systemctl start glb container> apt-get remove wget
Container testing

To be sure everything works fine we should do some basic tests:

shell> lxc stop ${INSTANCE} shell> lxc start ${INSTANCE} shell> echo getinfo | nc -q 1 10.139.158.183 8011
Unit testing focmm against Galera Load Balancer in the LXC container

And finally we did the unit testing of focmm against the Galera Local Balancer which is in the LXC container:

shell> ./tst/run_all_tests.php --instance=qamariadb106 --module=LoadBalancer shell>/dev/null Environment is: qamariadb106 Logfile is: /tmp/focmm_tst_qamariadb106.log Tests are: module=LoadBalancer and function=all OK stopLoadBalancerRemote OK startLoadBalancerRemote OK restartLoadBalancerRemote OK readLoadBalancerTypes OK createLoadBalancer OK updateLoadBalancer OK readLoadBalancers OK parseGlbConfiguration OK openSocket OK writeReadSocket OK parseGlbGetInfo OK parseGlbGetStats OK changeLoadBalancerBackendWeightRemote OK deleteLoadBalancer
Taxonomy upgrade extras: containerlxclxdtestingunit testinggaleraload balancer

Aufgrund der grossen Nachfrage bieten wir vom 19. bis 22. Dezember 2022 ein zusätzliches Galera Cluster Seminar an. Dies in Zusammenarbeit mit unserem Schulungspartner, der GfU Cyrus AG in Köln.

Dieses Seminar wird remote durchgeführt. Sie können an diesem Seminar also bequem von zuhause aus teilnehmen...

Eine Seminarübersicht finden Sie hier.

Sie können das Seminar einfach online buchen.

Falls Sie an diesen Terminen verhindert sind, haben wir bereits einige Termine für das Jahr 2023 für Sie in Planung.

Ihr FromDual Schulungsteam

Taxonomy upgrade extras: galeraschulunggfu20232022seminar

The newest release of the Performance Monitor for MariaDB and MySQL v2.0.0 (fpmmm) allows you to monitor your MariaDB and MySQL databases via SNMP (Simple Network Management Protocol).

This feature enables you to report your Database Metrics into Enterprise Monitoring solutions from Microsoft, IBM or into Oracle Cloud Control.

In this article we will show you the few simple steps it needs to send the data from your fpmmm Agent to your SNMP Manager.

Check if SNMP Manager can be reached

To make sure fpmmm can send data to the SNMP Manager at all you can test the connection with the following command to send SNMP Notifications to the SNMP Manager:

# COMMUNITY='public' # MANAGER='192.168.56.102' # PORT='162' # TRAP_OID='1.3.6.1.4.1.57800.1.1.2' # OID='1.3.6.1.4.1.57800.1.1.1' # TYPE='c' # VALUE=$(mariadb --user=root --execute="SELECT variable_value FROM information_schema.global_status WHERE variable_name LIKE 'threads_running'\G" | grep variable_value | cut -d' ' -f2) # snmptrap -v 2c -c ${COMMUNITY} ${MANAGER}:${PORT} '' ${TRAP_OID} ${OID} ${TYPE} "${VALUE}"

Then check in the SNMP Manager Logfile if the traps arrive.

Copy fpmmm MIBs to your fpmmm Agent Machine

You can see where your fpmmm MIBs are expected on your database machine (where your fpmmm Agent is located) with the following command:

# net-snmp-config --default-mibdirs ${HOME}/.snmp/mibs:/usr/share/snmp/mibs:/usr/share/snmp/mibs/iana:/usr/share/snmp/mibs/ietf:/usr/share/mibs/site:/usr/share/snmp/mibs:/usr/share/mibs/iana:/usr/share/mibs/ietf:/usr/share/mibs/netsnmp

Copy the fpmmm MIBs as follows (we assume that you work with the same user as your fpmmm Agent works (zabbix or mysql)):

# mkdir -p ${HOME}/.snmp/mibs # cp /opt/fpmmm/tpl/FromDual-fpmmm-MIB.mib ${HOME}/.snmp/mibs

or choose any other location you want to have them.

To check if the FromDual MIBs are recognized correctly run these commands:

# snmptranslate -DFromDual-fpmmm-MIB.mib -m +FromDual-fpmmm-MIB 1.3.6.1.4.1.57800.1.1.1 registered debug token FromDual-fpmmm-MIB.mib, 1 FromDual-fpmmm-MIB::fpmmmLastrun # snmptranslate -On FromDual-fpmmm-MIB::fpmmmLastrun .1.3.6.1.4.1.57800.1.1.1

Do the same for the SNMP Manager according to your installation instructions of your SNMP Manager.

Configure your FromDual Performance Monitor to send SNMP traps

The fpmmm Configuation File (/etc/fpmmm.conf) should contain at least the following lines to make fpmmm sending SNMP traps instead of Zabbix messages:

[default] # SNMP Server SnmpServer = 192.168.56.102 SnmpCommunity = public SnmpTrapPort = 162 OutputFormat = snmp

Do a test run of fpmmm with:

# /opt/fpmmm/bin/fpmmm --config=/etc/fpmmm.conf

Then check again on your SNMP Server machine if the SNMP traps arrive:

snmptrapd: Agent Address: 0.0.0.0 Agent Hostname: Date: 1 - 1 - 4 - 1 - 1 - 1970 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v2c, community public Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (1215419) 3:22:34.19#012SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.57800.1.1.2#012SNMPv2-SMI::enterprises.57800.1.1.1 = Counter32: 3 snmptrapd: Agent Address: 0.0.0.0 Agent Hostname: Date: 1 - 1 - 4 - 1 - 1 - 1970 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v2c, community public Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (60) 0:00:00.60#012SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.57800.1.1.2#012SNMPv2-SMI::enterprises.57800.2.7.1 = Counter32: 1 snmptrapd: Agent Address: 0.0.0.0 Agent Hostname: Date: 1 - 1 - 4 - 1 - 1 - 1970 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v2c, community public Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (60) 0:00:00.60#012SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.57800.1.1.2#012SNMPv2-SMI::enterprises.57800.2.7.2 = Counter32: 151 snmptrapd: Agent Address: 0.0.0.0 Agent Hostname: Date: 1 - 1 - 4 - 1 - 1 - 1970 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v2c, community public Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (60) 0:00:00.60#012SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.57800.1.1.2#012SNMPv2-SMI::enterprises.57800.2.7.1 = Counter32: 1 snmptrapd: Agent Address: 0.0.0.0 Agent Hostname: Date: 1 - 1 - 4 - 1 - 1 - 1970 Enterprise OID: . Trap Type: Cold Start Trap Sub-Type: 0 Community/Infosec Context: TRAP2, SNMP v2c, community public Uptime: 0 Description: Cold Start PDU Attribute/Value Pair Array:#012DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (60) 0:00:00.60#012SNMPv2-MIB::snmpTrapOID.0 = OID: SNMPv2-SMI::enterprises.57800.1.1.2#012SNMPv2-SMI::enterprises.57800.2.7.3 = Counter32: 1

This is already all about making fpmmm capable sending SNMP traps.

Sources

• Monitoring your MariaDB database with SNMP

Taxonomy upgrade extras: performanceperformance monitoringmonitorfpmmmSNMP

FromDual has the pleasure to announce the release of the new version 2.0.0 of its popular Database Performance Monitor for MariaDB and Galera Cluster fpmmm.

The FromDual Performance Monitor for MariaDB (fpmmm) enables DBAs and System Administrators to monitor and understand what is going on inside their MariaDB database instances and on the machines where the databases reside.

More detailed information you can find in the fpmmm Installation Guide.

Download

The new FromDual Performance Monitor for MariaDB (fpmmm) can be downloaded from here or you can use our FromDual repositories. How to install and use fpmmm is documented in the fpmmm Installation Guide.

In the inconceivable case that you find a bug in the FromDual Performance Monitor for MariaDB please report it to the FromDual Bug-tracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to us.

Monitoring as a Service (MaaS)

You do not want to set-up your Database monitoring yourself? No problem: Choose our MariaDB Monitoring as a Service (Maas) program to safe time and costs!

Installation of Performance Monitor 2.0.0

A complete guide on how to install FromDual Performance Monitor you can find in the fpmmm Installation Guide.

Upgrade of fpmmm tarball from 1.x to 2.0.0

There are some changes in the configuration file (fpmmm.conf):

• The access rights should be change as follows: chmod 600 /etc/fpmmm.conf
• The key Methode was spelled wrong in the configuration file. It was renamed to Method.
• The key PidFile is ambiguous which could lead to problems and bugs. Thus it was changed to either MyPidFile for fpmmm and DbPidFile for the database.

Upgrade with DEB/RPM packages should happen automatically. For tarballs follow this:

shell> cd /opt shell> tar xf /download/fpmmm-2.0.0.tar.gz shell> rm -f fpmmm shell> ln -s fpmmm-2.0.0 fpmmm
Changes in FromDual Performance Monitor for MariaDB 2.0.0

This release contains new features and various bug fixes.

You can verify your current FromDual Performance Monitor for MariaDB version with the following command:

shell> /opt/fpmmm/bin/fpmmm --version
General

• Some trigger url links were improved in various templates.
• Zabbix 5.0 templates were added.
• New MyEnv library added.
• oncePerHourRun and oncePerDay run implemented.
• Code clean-up in various functions: sendData, cacheFile, initValue, mr_version, mylog, extractVersioncomment. This should make fpmmm dramatically faster in some scenarios.
• Error handling improved, error messages improved, some Linux command replaced by PHP commands, typos fixed.
• Better handling of non available variables and status information.
• All old php5 stuff removed.
• Various fixed of minor bugs.
• Ambiguous PidFile replaced by MyPidFile for fpmmm and DbPidFile for database PID file. Caution during Upgrade!
• Lock location changed from /var/run to /run/lock.
• Several distribution related and new MariaDB privilege related bugs fixed.
• Variables default, configuration and configuration file sorted out.
• Lock file handling improved, Nagios functionality added.
• Config file is checked if it is readable for others.
• Own locking mechanism replaced by O/S flock.
• Option checks activated.
• Newer distros added (Debian 11, Ubuntu 22.04, Rocky Linux 8).
• PHP 8, MySQL 8 and MariaDB 10.6 tests passed.

Server

• Item entropy added.
• Trigger urls fixed.
• Items added to templates: Server Entropy.
• NUMA count fixed.
• Linux md drives are supported now as well.
• smartctl error handling improved.
• NUMA node memory balancing graph added and trigger set.
• CPU time values were far too high. This is fixed with the new CPU measuring variant.
• Broken md devices is added to server module + trigger.
• Server disk: minor device is only gathered in extensive, loop device is skipped.
• Filesystem locks item and trigger added.
• Path /usr/sbin for tool smartctl added for Debian.
• Skip loop and snap devices.
• Server does not wait 1 second any more for CPU info.
• New Feature GatherData = {normal|minimal|extensive} added.

Data

• Disabled binary log is caught in data.
• #innodb_temp schema size added.

Galera

• tx_isolation made MySQL 8 ready.
• Bug fixed in case when Galera is disabled but module is activated.
• Tables without Primary Key and non InnoDB table items and triggers added.
• Various triggers added to check correct Galera configuration.
• Change of cluster configuration id fixed.
• XA trigger fixed.
• Galera cache item + trigger fixed.
• Some more items added for Galera compatibility checks.
• Items wsrep_xa_statements, wsrep_log_bin, wsrep_log_slave_updates and gcache.size added.

User

• Code refactored and cleaned-up.
• Problem during max_connections reached fixed.

Agent

• Trigger links fixed.
• New fpmmm version items and triggers added.
• fpmmm version stuff fixed.
• Lay off of software-release function and implement check for fpmmm version.

InnoDB

• InnoDB template fixed on deadlock item.
• All locking related graphs moved together into same InnoDB screen.
• innodb_buffer_pool_max_dirty_pages_pct added.
• Fix division by zero bug.
• Bug in InnoDB status fixed.
• InnoDB buffer pool dirty pages trigger added and a small number of buffer pool instances trigger removed.
• Trigger for innodb_flush_log_at_trx_commit set from info to warning because this is security related.
• Innodb_log_occupancy, Innodb_checkpoint_age, innodb_log_group_capacity and trigger added.
• innodb_file_format item added.
• Items innodb_max_dirty_pages_pct added.
• InnoDB metrics added.
• Redo log occupancy increased.

MyISAM

• none

Aria

• none

Security

• none

Master

• Sync_binlog warning change into the opposite. Security first!
• BINLOG CLIENT privilege replaced by BINLOG MONITOR for MariaDB 10.5.

Slave

• Template link fixed.
• Table with missing Primary Key item added.
• Link to new graphs fixed for 5.0.

Backup

• Restore items moved from backup to restore template.
• Items for restore added for automatized restore testing.
• Trigger links in backup template fixed.
• Trigger is fired if backup did not happen within last 36 hours.

MySQL

• Trigger for transaction_cache_size and statement_cache_size was improved.
• Trigger urls fixed.
• Table Open Cache and Table Definition Cache triggers link changed to item values.
• Storage_engine added for compatibility reasons with default_storage_engine.
• Processlist total threads added.
• Error log parser added.
• com_change_db and queries added.
• Refactoring of code.
• Processlist graph and items added, code made better.
• P_S metadata lock warning improved.
• Metadata lock info plugin installation message added.
• Storage_engine removed.

Nagios

• FromDual Nagios plug-ins implemented in fpmmm.

SNMP

• SNMP implemented for most important MariaDB and Galera metrics.

For subscriptions of commercial use of fpmmm please get in contact with us.

Taxonomy upgrade extras: performancemonitormonitoringfpmmmmaasreleasegraph

For this how-to we were working with a MariaDB 10.6.5 database server and the newest MariaDB MaxScale 6.3.0 (aka 2.6) load balancer/proxy.

As a starting point we had a working MariaDB Master/Master replication with one active Master (Ma) and one passive Master (Mp). Whereas the passive Master was set to read_only. As a replication user we used the user replication.

Creating database accounts for the MaxScale Load Balancer

The MaxScale load balancer connects itself with the application users to the database. To do this it needs to retrieve the available users from the database first. To get these users the MaxScale account needs some privileges:

CREATE USER 'maxscale'@'%' IDENTIFIED BY 'secret'; GRANT SELECT ON mysql.user TO 'maxscale'@'%'; GRANT SELECT ON mysql.db TO 'maxscale'@'%'; GRANT SELECT ON mysql.tables_priv TO 'maxscale'@'%'; GRANT SELECT ON mysql.columns_priv TO 'maxscale'@'%'; GRANT SELECT ON mysql.procs_priv TO 'maxscale'@'%'; GRANT SELECT ON mysql.proxies_priv TO 'maxscale'@'%'; GRANT SELECT ON mysql.roles_mapping TO 'maxscale'@'%'; GRANT SHOW DATABASES ON *.* TO 'maxscale'@'%';

For monitoring the replication and doing proper switchover and failover MaxScale further needs a monitoring account:

CREATE USER 'maxscale_monitor'@'%' IDENTIFIED BY 'secret'; GRANT REPLICATION CLIENT on *.* to 'maxscale_monitor'@'%'; GRANT REPLICATION SLAVE on *.* to 'maxscale_monitor'@'%'; GRANT SUPER, RELOAD on *.* to 'maxscale_monitor'@'%';

And last we need an application account for our test application:

CREATE USER 'app'@'%' IDENTIFIED BY 'secret'; GRANT ALL on test.* to 'app'@'%';
Starting MariaDB MaxScale Load Balancer

Because we do not use the provided DEB/RPM packages on our systems but generic binary tarballs, which are not available, we have to start the MaxScale Load Balancer a bit over-complicated:

# export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/home/mysql/product/maxscale/usr/lib/x86_64-linux-gnu/maxscale # ldd ./maxscale # ./maxscale --help # ./maxscale --config=/home/mysql/etc/maxscale_mm.cnf --datadir=/home/mysql/tmp --basedir=/home/mysql/product/maxscale --logdir=/home/mysql/log --config-check # ./maxscale --config=/home/mysql/etc/maxscale_mm.cnf --datadir=/home/mysql/tmp --basedir=/home/mysql/product/maxscale --logdir=/home/mysql/log

The MaxScale configuration file we used in these tests looks as follows:

# # maxscale_mm.cnf # [maxscale] threads = auto [master1] type = server address = 192.168.1.11 port = 3306 protocol = MariaDBBackend [master2] type = server address = 192.168.1.12 port = 3306 protocol = MariaDBBackend [MultiMasterMonitor] type = monitor module = mariadbmon servers = master1,master2 user = maxscale_monitor password = secret enforce_read_only_slaves = true auto_rejoin = true # auto_failover = true [WriteService] type = service router = readconnroute router_options = master servers = master1,master2 user = maxscale password = secret [ReadService] type = service router = readconnroute router_options = slave servers = master1,master2 user = maxscale password = secret [WriteListener] type = listener service = WriteService protocol = MariaDBClient port = 3306 [ReadListener] type = listener service = ReadService protocol = MariaDBClient port = 3307

If the start was successful can be seen for example with:

# ps -ef | grep maxscale

If you start the MariaDB MaxScale the first time no user/password is needed. So we can connect the the MaxScale Load Balancer with the maxctrl client:

# ./maxctrl maxctrl> list listeners ┌──────────────────┬──────┬──────┬─────────┬──────────────┐ │ Name │ Port │ Host │ State │ Service │ ├──────────────────┼──────┼──────┼─────────┼──────────────┤ │ WriteListener │ 3306 │ :: │ Running │ WriteService │ ├──────────────────┼──────┼──────┼─────────┼──────────────┤ │ MaxAdminListener │ 3307 │ :: │ Running │ ReadService │ └──────────────────┴──────┴──────┴─────────┴──────────────┘ maxctrl> list services ┌──────────────┬───────────────┬─────────────┬───────────────────┬──────────────────┐ │ Service │ Router │ Connections │ Total Connections │ Servers │ ├──────────────┼───────────────┼─────────────┼───────────────────┼──────────────────┤ │ WriteService │ readconnroute │ 0 │ 0 │ master1, master2 │ ├──────────────┼───────────────┼─────────────┼───────────────────┼──────────────────┤ │ ReadService │ readconnroute │ 0 │ 0 │ master1, master2 │ └──────────────┴───────────────┴─────────────┴───────────────────┴──────────────────┘ maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬──────────────────────────────┬────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────────────┼────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 0 │ Master, Running │ 0-3308-134 │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────────────┼────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 0 │ Relay Master, Slave, Running │ 0-3308-134 │ └─────────┴──────────────┴──────┴─────────────┴──────────────────────────────┴────────────┘ maxctrl> list sessions ┌────────┬──────┬─────────────────────┬──────────────────────────┬──────┬─────────────┐ │ Id │ User │ Host │ Connected │ Idle │ Service │ ├────────┼──────┼─────────────────────┼──────────────────────────┼──────┼─────────────┤ │ 746468 │ app │ ::ffff:192.168.1.99 │ Fri May 13 15:28:55 2022 │ 0.4 │ ReadService │ └────────┴──────┴─────────────────────┴──────────────────────────┴──────┴─────────────┘

To not allow everybody to access MariaDB MaxScale we set a password for the admin user:

maxctrl> alter user admin secret OK

Then you can connect with username and password:

# ./maxctrl --user=admin --password=secret maxctrl> list users ┌───────┬──────┬────────────┐ │ Name │ Type │ Privileges │ ├───────┼──────┼────────────┤ │ admin │ inet │ admin │ └───────┴──────┴────────────┘
Testing connections over the MaxScale Load Balancer

To test if connections are ending up on the correct database instances we used the following commands:

# mariadb --user=app --host=192.168.1.1 --port=3306 --password=secret test --execute='SELECT @@hostname, @@port' # mariadb --user=app --host=192.168.1.1 --port=3307 --password=secret test --execute='SELECT @@hostname, @@port'
Monitoring of nodes in MariaDB MaxScale

To see what is going on inside the MariaDB MaxScale Load Balancer we used the following command to create a simple real time monitor:

# watch -d -n 1 ./maxctrl --user=admin --password=secret list servers
Switchover and Failover with MariaDB MaxScale

To use MariaDB MaxScales switchover and failover capabilities the Master/Master Replication must be configured to use Global Transaction IDs (GTID). To change to GTID based replication you can use the following commands on both masters:

SQL> STOP SLAVE; SQL> CHANGE MASTER TO MASTER_USE_GTID = SLAVE_POS; SQL> START SLAVE;
Switchover in MariaDB MaxScale

A graceful switchover is used in a controlled situation if we want to switch the roles between the active master and the passive master and vice versa... This can be used for example before a maintenance operation: We do the maintenance operation on the passive Master first, then we switch the roles and then we can do the maintenance operation on the now new passive Master:

maxctrl> call command mariadbmon switchover MultiMasterMonitor

After switching for and back a few times we found out, that the MariaDB MaxScale Monitor has replaced our replication user replication by his own user maxscale_monitor. This is not documented? and I do really not like it, especially if it is done silently...

Failover in MariaDB MaxScale

To provoke/simulate a failover situation we stopped the active Master. Then we see in our monitor that the active Master is down and the passive Master is still running:

maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬────────────────┬────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼────────────────┼────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 0 │ Down │ 0-3309-142 │ ├─────────┼──────────────┼──────┼─────────────┼────────────────┼────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 1 │ Slave, Running │ 0-3309-142 │ └─────────┴──────────────┴──────┴─────────────┴────────────────┴────────────┘

We further observe, that our application (the famous insert_test.sh) is not working any more but throwing errors... Then we trigger a manual failover:

maxctrl> call command mariadbmon failover MultiMasterMonitor

It looks like an automatic failover is possible (auto_failover = true) but we do not recommend this set-up and thus we did not further investigate in this feature.

After the manual failover the former passive Master becomes active Master:

maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬─────────────────┬───────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼─────────────────┼───────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 0 │ Down │ 0-3308-365829 │ ├─────────┼──────────────┼──────┼─────────────┼─────────────────┼───────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 0 │ Master, Running │ 0-3308-370235 │ └─────────┴──────────────┴──────┴─────────────┴─────────────────┴───────────────┘

Now we simulate the repair of the former failed active Master by just restarting it:

maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬─────────────────┬───────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼─────────────────┼───────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 0 │ Slave, Running │ 0-3308-401309 │ ├─────────┼──────────────┼──────┼─────────────┼─────────────────┼───────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 1 │ Master, Running │ 0-3308-401309 │ └─────────┴──────────────┴──────┴─────────────┴─────────────────┴───────────────┘

It comes back into the MariaDB MaxScale but just as a simple Slave. So a failover seems to break our Master/Master replication for some reasons. This must be repaired afterwards. Whereas a simple switchover seems to work properly. If this is a bug or intended behaviour I do not know... With the following command on master2 the loop is closed again:

SQL> CHANGE MASTER TO master_host='192.168.1.11', master_port=3306, master_user='maxscale_monitor', master_password='secret', master_use_gtid=slave_pos; SQL> START SLAVE;

Now everything is fine and working as expected again:

maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬──────────────────────────────┬───────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────────────┼───────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 1 │ Relay Master, Slave, Running │ 0-3308-440194 │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────────────┼───────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 0 │ Master, Running │ 0-3308-440194 │ └─────────┴──────────────┴──────┴─────────────┴──────────────────────────────┴───────────────┘

Stopping and starting the passive Master had not effect at all.

Switchover with a lagging passive master

To test this scenario we created an artificial lag of the passive Master by setting innodb_flush_log_at_trx_commit = 1 and sync_binlog = 1. When the passive master was lagging far enough (about 30 seconds) we tried an switchover:

maxctrl> call command mariadbmon switchover MultiMasterMonitor Error: timeout of 10000ms exceeded

So it is not really clear what happens in this case and the error message is not really telling us about the problem. Further this operations somehow breaks Master/Master replication again.

Failover with a lagging passive master

If we try a failover instead of a switchover we get at least a bit a more meaningful error message:

Error: Server at http://127.0.0.1:8989 responded with 403 Forbidden to `POST maxscale/modules/mariadbmon/failover?MultiMasterMonitor` { "links": { "self": "http://127.0.0.1:8989/v1/maxscale/modules/mariadbmon/failover/" }, "meta": { "errors": [ { "detail": "Can not select 'master1' as a demotion target for failover because it is a running master." }, { "detail": "Failover cancelled." } ] } }
Draining a node with MariaDB MaxScalse

If we try to drain a passive Master we get some warnings but it seems like the result is the expected:

maxctrl> drain server master2 Warning: Saving runtime modifications to 'ReadService' in '/home/mysql/product/maxscale/var/lib/maxscale/maxscale.cnf.d/ReadService.cnf'. The modified values will override the values found in the static configuration files.;Saving runtime modifications to 'WriteService' in '/home/mysql/product/maxscale/var/lib/maxscale/maxscale.cnf.d/WriteService.cnf'. The modified values will override the values found in the static configuration files. To hide these warnings, run: export MAXCTRL_WARNINGS=0 Warning: Saving runtime modifications to 'ReadService' in '/home/mysql/product/maxscale/var/lib/maxscale/maxscale.cnf.d/ReadService.cnf'. The modified values will override the values found in the static configuration files.;Saving runtime modifications to 'WriteService' in '/home/mysql/product/maxscale/var/lib/maxscale/maxscale.cnf.d/WriteService.cnf'. The modified values will override the values found in the static configuration files. To hide these warnings, run: export MAXCTRL_WARNINGS=0 OK maxctrl> list servers ┌─────────┬──────────────┬──────┬─────────────┬──────────────────────┬───────────────┐ │ Server │ Address │ Port │ Connections │ State │ GTID │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────┼───────────────┤ │ master1 │ 192.168.1.11 │ 3306 │ 0 │ Master, Running │ 0-3308-631119 │ ├─────────┼──────────────┼──────┼─────────────┼──────────────────────┼───────────────┤ │ master2 │ 192.168.1.12 │ 3306 │ 0 │ Maintenance, Running │ 0-3308-631119 │ └─────────┴──────────────┴──────┴─────────────┴──────────────────────┴───────────────┘
Un-drain a node with MariaDB MaxScale

For un-draining the node we tried:

maxctrl> clear server master2 drain OK

On the first look everything seems to be OK. But the State of master2 was still in Maintenance. The command:

maxctrl> clear server master2 maintenance

did the job... If this is a bug or should be considered as an operator error I do not know...

Draining the active Master caused a switchover.

Sources

• Multi-Master Monitor
• Common Monitor Parameters
• Automatic Failover With MariaDB Monitor
• MariaDB Monitor
• MariaDB MaxScale Configuration Guide
• MaxScale 6 MaxCtrl - Drain Server

Taxonomy upgrade extras: replicationmaxscaleload balancerproxyactive-activemaster-mastermariadb

Because the original MariaDB documentation is a little bit sparse about this topic here are some notes about how to do a streaming MariaDB backup:

# mariadb-backup --user=root --backup --parallel=4 --stream=mbstream | ssh root@target_machine "/usr/bin/mbstream -x -C /mnt/backup/" # mariadb-backup --user=root --prepare --target-dir=/mnt/backup

FromDual has the pleasure to announce the release of the new version 2.2.5 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.2.5 shell> cd ${HOME}/product shell> tar xf /download/brman-2.2.5.tar.gz shell> rm -f brman shell> ln -s brman-2.2.5 brman
Changes in FromDual Backup and Recovery Manager 2.2.5

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0 and 2.1 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version shell> bman --version shell> rman --version
General

• Newest myEnv library copied from myEnv project.
• Some code clean-up.

FromDual Backup Manager (bman>

• Option --source-data which was introduce in MySQL 8.0.26 caused a problem in the mysqldump command. Fixed.
• Tool XtraBackup 8.0 does not support --no-timestamp option any more. Fixed.
• Library bman.inc restructured into smaller files.
• Requirement checks for myenv and brman separated.
• Start backup message move much further up in output.
• Connection handling of bman made more OO style to nicer cope with some special cases (DB down).
• Directory for fpmmm is created automatically now if it does not exist.
• Clean-up file size returned to bman and can now be reported correctly to fpmmm.
• Output of configuration files read improved.
• Configuration files of brman are now checked if group or other can read the files.
• Hostname is converted to IP first now before we check if it is a remote location in doBinlogBackup.
• Check in doSchemabackup for missing RELOAD privilege (rc=4198) was done in a wrong way. Fixed.
• Error message improved in doBinlogBackup.
• Various typos fixed.
• Various bugs in bman fixed found with testing and tests improved.

FromDual Recovery Manager (rman)

• Rman also checks configuration file for readability for group and others now.
• Option --target in rman is accepted now as configuration parameter.

FromDual brman Catalog

• Some bugs bman_catalog fixed found with testing.

Testing

• XtraBackup version check bug fixed in full physical backup test.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from from us.

Taxonomy upgrade extras: BackupRestoreRecoverypitrbrmanreleasebmanrmanFromDual Backup and Recovery Manager

As already mentioned earlier, we roughly have a hand full of customers which are playing with the though of sharding solutions. They typically have many different customers (clients, tenants) and the number of customers becomes so huge (thousands to millions) that one machine cannot cope with the load any more.

So splitting the load by customers to different machines makes sense. This is quite easy when customers are separated per schema. In the good old times of Open Source our customers have implemented those solutions themselves. But nowadays it looks like do-it-yourself is not sexy any more. It seems like this core competence of a business advantage must be outsourced. So some vendors have already made some solutions available to solve this need: Sharding Solutions.

My question here is: Can a generic sharding solution build exactly what your business needs? Are you still capable to optimize your business process in the way you need it when you buy a 3rd party solution?

And: If you use another product, you have also to build up the know-how how to use it correctly. So I am really wondering if it is worth the effort? Buy or make is the question here. So we made a little Proof-of-Concept of a sharding solution. And it did not take too long...

The concept

First of all we have 2 different kinds of components:

• The Fabric Node - this is the database where all the meta information about the shards are stored.
• The Shards - these are the databases where all the customer data are stored.

And we need all this more or less in a highly available fashion.

The fabric table can look as simple as this:

CREATE TABLE `tenant` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `tenant_id` varchar(128) NOT NULL, `schema` varchar(128) NOT NULL, `machine` varchar(128) NOT NULL, `port` smallint(5) unsigned NOT NULL, `locked` enum('no','yes') NOT NULL DEFAULT 'no', PRIMARY KEY (`id`), UNIQUE KEY `tenant_id` (`tenant_id`), UNIQUE KEY `schema` (`schema`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ; INSERT INTO tenant VALUES (NULL, 'Customer 1', 'customer_1', '192.168.33.21', 3306, 'no') , (null, 'Customer 2', 'customer_2', '192.168.33.22', 3306, 'yes') , (null, 'Customer 3', 'customer_3', '192.168.33.23', 3306, 'no') ; SQL> SELECT * FROM tenant; +----+------------+------------+---------------+------+--------+ | id | tenant_id | schema | machine | port | locked | +----+------------+------------+---------------+------+--------+ | 4 | Customer 1 | customer_1 | 192.168.33.21 | 3306 | no | | 5 | Customer 2 | customer_2 | 192.168.33.22 | 3306 | yes | | 6 | Customer 3 | customer_3 | 192.168.33.23 | 3306 | no | +----+------------+------------+---------------+------+--------+
Connection

Now our application needs to know on which shard the data for a specific customer is stored. This will be found in the fabric. So our application has to do first a connect to the fabric and then a connect to the shard. To make it more transparent for your application you can encapsulate everything in one method. And if you want to optimize the connecting you can store the sharding information in a local cache.

$dbhFabric = getFabricConnection($aFabricConnection); $aShardConnection = readShardConnection($dbhFabric, $aTenant); $dbhShard = getShardConnection($aShardConnection); // Do everything needed for the tenant... $sql = sprintf("SELECT * FROM `customer_data` limit 3"); $result = $dbhShard->query($sql);

For the PoC we create 3 different schemas (customer_1, customer_2 and customer_3) with some customer data in it:

CREATE TABLE `customer_data` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `customer_name` varchar(128) NOT NULL, `customer_data` varchar(128) NOT NULL, `customer_number` INT UNSIGNED NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ; INSERT INTO `customer_data` VALUES (NULL, 'Customer 3', 'Some data', ROUND(RAND()*1000000, 0)) , (NULL, 'Customer 3', 'Some data', ROUND(RAND()*1000000, 0)) , (NULL, 'Customer 3', 'Some data', ROUND(RAND()*1000000, 0)) ;
Resharding

One of the challenges of such a construct is that the intensity of usage in the shards changes over time. All new customers are possibly placed in the same new shard. The new customers start playing around with your application. They do more and more. So a shard becomes overloaded sooner or later. Or the other way around: You have a lot of customers which were very enthusiastic about your product in the beginning and not so much any more now. So you loose customers on the older shards. Or you have to replace an old shard by newer hardware. All this leads to the situation, that your solution is not balanced any more and must be re-balanced. We call this resharding:

This can be done quite easily with a few commands:

-- Lock tenant UPDATE `tenant` SET `locked` = 'yes' WHERE `tenant_id` = 'Customer 3'; -- Wait some time until cache expires or invalidate cache mariadb-dump --user=root --host=192.168.33.23 --port=3306 customer_3 | mariadb --user=root --host=192.168.33.21 --port=3306 # Check error codes very well. Possibly do some other additional checks! -- Update tenant to new shard UPDATE `tenant` SET `machine` = '192.168.33.21', `port` = 3306 WHERE `tenant_id` = 'Customer 3'; -- DROP tenant in old shard mariadb --user=root --host=192.168.33.23 --port=3306 --execute="DROP SCHEMA `customer_3`" -- Unlock tenant UPDATE `tenant` SET `locked` = 'no' WHERE `tenant_id` = 'Customer 3';
Monitoring

To find out which customers cause a lot of load and which shards are over- or under-loaded you need some kind of sophisticated monitoring. You possibly want to know things like:

• CPU usage (mpstat)
• I/O usage (iostat)
• NW usage
• RAM usage (free)
• Number of connections per shard and per customer
• Number of queries per shard and per customer
• etc.

Those metrics can be found with some queries:

SHOW GLOBAL STATUS LIKE 'Bytes%'; SELECT * FROM information_schema.global_status WHERE variable_name IN ('Threads_connected', 'Threads_running', 'Max_used_connections') ; SELECT processlist_db, COUNT(*) FROM performance_schema.threads WHERE type = 'FOREGROUND' GROUP BY processlist_db ; SELECT * FROM sys.schema_table_lock_waits WHERE object_schema LIKE 'customer%'; SELECT * FROM sys.schema_table_statistics WHERE table_schema LIKE 'customer%'; SELECT * FROM sys.schema_table_statistics_with_buffer WHERE table_schema LIKE 'customer%';; SELECT table_schema, SUM(data_length) + SUM(index_length) AS schema_size FROM information_schema.tables WHERE table_schema LIKE 'customer%' GROUP BY table_schema ;
Missing features

If you really think you need it, you can also make a nice GUI to show all those metrics. But be prepared: This will cost you most of your time!

We assume that all the shards are accessed with the same user as the fabric is accessed. In reality this is possibly not the case. But the tenant table can be easily extended.

To make the whole concept much easier we omitted the idea of number of replicas which is known from other solutions. We think having every shard redundant by a Master/Slave replication is sufficient.

If you find anything which is missing in this concept study or if you experience some problems we did not thought about, we would be happy hearing from you.

Taxonomy upgrade extras: shardingmulti-tenant

In the last months we got more and more requests for supporting MariaDB/MySQL/Galera Cluster in (mostly Docker) containers.

Because of its additional layer and added complexity I do not like containers much. Containers are more complicated during troubleshooting and debugging problems.

Other people have already written more than enough about the advantages of containers. What is more difficult to find are the disadvantages of technologies. Thus I focus on those:

Wrong technology?

Container solutions were designed to deal with stateless applications that have ephemeral data. Containers spin up a quick microservice and then destroy it. This includes all the components of that container (including its cache and data). The transient nature of containers is because all of the components and services of that container are considered to be part of the container (essentially it is all or nothing). Serving the container a data volume owned by the underlying O/S by punching a hole through the container can be very challenging.
Most of the development efforts put into the various solutions had one goal in mind: Statelessness. There are solutions that can help keep your data persistent, but they are very quickly evolving. They require a high level of complexity, that negate any efficiency gains due to increased operational complexity (and risk). [1]

Stability

These container solutions are meant for quick development and deployment of applications that are broken into tiny components: microservices. Normally, these applications evolve very quickly in organizations that are very software/developer-driven. That seems to be how these container solutions (again, especially Docker) are developed as well. New features are pushed out with little testing and design. The main focus seems to be the latest feature set and being first to market. They “beg for forgiveness” instead of “ask for permission.” On top of that, backward compatibility is a distant concern (and even that might be an overstatement). This means that you are going to have to have a mature Continuous Delivery and testing environment as well as a known and tested image repository for your containers. [1]

We have seen complaints about Galera Cluster stability issues inside Docker containers. The signs were pointing without doubt to network issues. If these were real network issues or just container network issues we could not find out yet.

Networking can be tricky in containers world when you want to limit the access within containers and also have proper network communications where required. [3]

Docker might even make your application slower. If you are working with it, you should set limits on how much memory, CPU, or block I/O the container can use. Otherwise, if the kernel detects that the host machine’s memory is running too low to perform important system functions, it could start killing important processes. If the wrong process is killed (including the Docker itself), the system will be unstable.

Performance

We have heard reports that performance overhead of Docker containers can be up to 10%. If this is still true with the right configuration and and recent version must be shown. [2]

You should not expect Docker to speed up an application in any way. [5]

Security

Since there is no full operating system people tend to overlook the security aspect of containers, but if you look up online, you will see that hackers are targeting systems that are hosted in containers and not secured properly.
Since the containers use the same kernel, they are not 100 isolated, so you should be aware of the risks if you are using multiple containers in one server, and make sure you know what you are doing and which containers are running on the same kernel along with your stuff! [3]

All containers share access to a single host operating system. You risk running Docker containers with incomplete isolation. Any malicious code can get access to your computer memory. [5]

Running applications with Docker implies running the Docker daemon with root privileges. Any processes that break out of Docker container will have the same privileges on the host as it did in the container. Running your processes inside the containers as a non-privileged user cannot guarantee security. It depends on the capabilities you add or remove. To mitigate the risks of Docker container breakout, you should not download ready-to-use containers from untrusted sources. [5]

Data storage is intricate – By design, all of the data inside a container leaves forever when it closes down except you save it somewhere else first. There are ways to store data tenaciously in Docker, such as Docker Data Capacities, but this is arguably a test that still has yet to be approached in a seamless manner. [6]

Container O/S is the same as host O/S. If the host O/S is upgraded all the containers get also a new O/S.

Popular docker images have many vulnerabilities. So build and harden your images yourself. [10, 11, 12]

One of the most famous Docker security vulnerabilities can be found here: Alpine Linux Docker Images Shipped for 3 Years with Root Accounts Unlocked and Alpine Linux Docker Image Vulnerability CVE-2019-5021 and Docker Image Vulnerability (CVE-2019-5021).

Complexity

Containerization also means consolidation. And as in consolidated systems usually you can have side effects or effects caused by someone you did not expect.

Debugging

Debugging problems in a container environment becomes more complex because the many additional layers added. Then the necessary information and metrics are not there or not available in the way as expected. This makes troubleshooting more complicated.

Policy

Docker implementation is quite complex. A load of technological supports are necessary for Docker implementation including orchestration, container management, app stack, data screenshots, networking of containers, and so on.

The container ecosystem is split – But the core Docker platform is open source, some container products do not work with other ones. [6]

Features

Container technologies require kernel features which were not present in earlier kernels. This made system maintenance more complicated. This problem may have been solved in the meanwhile?

If you are aware of any other disadvantage not mentioned above please let us know. Some of these disadvantages might have been reduced in the recent years.

Literature

• [1] Are Docker Containers Good for Your Database? (2016-11)
• [2] Testing Docker multi-host network performance (2016-08)
• [3] Why Docker? Pros and Cons (2017-11)
• [4] Why is Docker so Popular - Good and Bad of Docker
• [5] 7 Cases When You Should Not Use Docker (2019-11)
• [6] What are the advantages and disadvantages of Docker? (2018-09)
• [7] Advantages and Disadvantages of Docker – Learn Docker
• [8] Docker Downsides: Container Cons to Consider before Adopting Docker (2017-05)
• [9] Containers: The pros and cons you may not know about (2019-02)
• [10] Docker security
• [11] 10 Docker Security Best Practices (2019-03)
• [12] 80% of developers are not addressing Docker security
• [13] Top Docker Security Vulnerabilities, Best Practices, & Insights (2020-04)
• [14] Docker » Docker : Security Vulnerabilities

Taxonomy upgrade extras: containerdockerkubernetes

This week I came across an old known issue reported in May 2010: Master/Slave Replication with binlog_format = ROW and tables without a Primary Key is a bad idea! Especially if these tables are huge.

Why this is a bad idea is described in the bug report #53375:

if one runs DML on a table that has no indexes, a full table scan is done. with RBR, the slave might need to scan the full table for *each* row changed.

The consequence of this behaviour is that the Slave starts lagging. It was further mentioned:

Worst part is that PROCESSLIST, etc provide absolutely NO obvious indication what is going on, for something that may take 12 hours, 3 days or even more...

Symptoms of this problem are described as follows:

Observe 78,278 row locks but only 10,045 undo log entries, so many more rows being scanned than changed. Also observer 16 row deletes per second but 600,754 row reads per second, same mismatch between counts suggesting unindexed accesses are happening.

You may also see "invalidating query cache entries (table)" as a symptom in the processlist. If you see that, check to see whether this is the possible root cause instead of giving full blame to only the query cache."

The suggested workaround is: add a primary key to the table.

But some user complain:

in my case, my only decent primary key is a surrogate key - and that's untenable because of the locking and lost concurrency (even with lock_mode = 2). Even if I solved that, I'd have to use the surrogate in partitioning - which more or less defeats the purpose of partitioning by lopsiding the partitions.

and others claim:

Adding an "otherwise usable (i.e. to improve query times)" PK is not really an option for them since there are no short unique columns.

A long composite key is also not an option because:

• In InnoDB tables, having a long PRIMARY KEY wastes a lot of space.
• In InnoDB, the records in nonclustered indexes (also called secondary indexes) contain the primary key columns for the row that are not in the secondary index. InnoDB uses this primary key value to search for the row in the clustered index. If the primary key is long, the secondary indexes use more space, so it is advantageous to have a short primary key.

And then comes a first suggestion for solving the issue:

So, we can create a normal short/auto-increment PK, but this is more or less the same as having the internal/hidden InnoDB PK (which does not seem to be used properly for RBR replication purposes).

As mentioned before, possibly the internal/hidden InnoDB PK can be used to resolve this bug.

Short after we get an important information and the learning of the day:

there's nothing that makes the InnoDB internal key consistent between a master and a slave or before and after backup and restore. Row with internal ID 1 can have completely different end user data values on different servers, so it's useless for the purpose being considered here, unfortunately.

Nor is there any prohibition on a slave having a unique key, which will be promoted to PK, even if there is no unique key on the master. They can even have different PKs and there can be good application reasons for doing that. Though we could require at least a unique key on all slaves that matches a master's PK without it hurting unduly.

It is possible to recommend at least _a_ key (not necessarily unique) on the slave and have replication try key-based lookups to narrow down the number of rows that must examined. That combined with batch processing should cut the pain a lot because we can reasonably ask for at least some non-unique but at least reasonably selective key to use. But this is only recommend, not require. If people want no key, we should let them have no key and be slow.

Then we got some further information why moving back to SBR is a bad idea:

It is my opinion that switching to SBR has way too many trade offs (if even for one table) to call it an acceptable workaround. The main crux for this argument being that just about the only time you run into this bug is when you have tables with a massive amount of rows - which is exactly where you start paying heavy penalties for SBR (Locking)."

And a new potential problem rises up:

As far as how the server should know which key to use - am I correct in assuming that it will use the optimizer to determine the index, and you are asking what would happen if the optimizer picked the wrong one?"

Another suggestion for improvement:

Batching looks promising, at least it would reduce the number of scans. But it would still be very painful if no key could be used. While using a key would be very painful if a high percentage of the rows in the table were being touched. So maybe some mixed solution that depends on the count of rows being touched might be best.

In about 2012 they had an implementation for batch jobs.

And you can force a Primary Key now in MySQL 8.0 since 2021 with sql_require_primary_key.

How MariaDB solves the problem you can find here: Row-based Replication With No Primary Key.

Taxonomy upgrade extras: primary keyreplicationRow Based Replication (RBR)Statement Based Replication (SBR)

We get ever and ever again customer requests concerning Deadlocks. First of all, Deadlocks are usually an application problem, not a database problem! The database itself manifests the application problem with the following message which is sent to the application as an error:

ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

If your application receives this error message you know where in your application you have a problem. But a deadlock is always a problem between 2 different connections. So how you can find the other part of the problem?

The full Deadlock situation is shown with the following command:

SQL> SHOW ENGINE InnoDB STATUS\G ... ------------------------ LATEST DETECTED DEADLOCK ------------------------ 2019-12-23 18:55:18 0x7f51045e3700 *** (1) TRANSACTION: TRANSACTION 847, ACTIVE 10 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 4 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 1 MySQL thread id 46, OS thread handle 139985942054656, query id 839 localhost root Updating delete from t where id = 10 *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 40 page no 3 n bits 80 index PRIMARY of table `test`.`t` trx id 847 lock_mode X locks rec but not gap waiting Record lock, heap no 2 PHYSICAL RECORD: n_fields 3; compact format; info bits 32 0: len 4; hex 8000000a; asc ;; 1: len 6; hex 00000000034e; asc N;; 2: len 7; hex 760000019c0495; asc v ;; *** (2) TRANSACTION: TRANSACTION 846, ACTIVE 25 sec starting index read mysql tables in use 1, locked 1 3 lock struct(s), heap size 1136, 2 row lock(s), undo log entries 1 MySQL thread id 39, OS thread handle 139985942361856, query id 840 localhost root Updating delete from t where id = 11 *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 40 page no 3 n bits 80 index PRIMARY of table `test`.`t` trx id 846 lock_mode X locks rec but not gap Record lock, heap no 2 PHYSICAL RECORD: n_fields 3; compact format; info bits 32 0: len 4; hex 8000000a; asc ;; 1: len 6; hex 00000000034e; asc N;; 2: len 7; hex 760000019c0495; asc v ;; *** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 40 page no 3 n bits 80 index PRIMARY of table `test`.`t` trx id 846 lock_mode X locks rec but not gap waiting Record lock, heap no 3 PHYSICAL RECORD: n_fields 3; compact format; info bits 32 0: len 4; hex 8000000b; asc ;; 1: len 6; hex 00000000034f; asc O;; 2: len 7; hex 770000019d031d; asc w ;; *** WE ROLL BACK TRANSACTION (2) ...

Unfortunately with the SHOW ENGINE INNODB STATUS command you can only show the last Deadlock error. If you want to log all the Deadlock errors in the MariaDB error log you can activate the Deadlock error reporting with the following configuration option in your MariaDB configuration file (my.cnf):

innodb_print_all_deadlocks = ON
How to solve Deadlocks?

Deadlocks always can occur in a transactional database system with fine-grained locking. So your application must be aware of Deadlocks and catch the Deadlock error and retry the transaction. The error message already indicates this. It is an advice for YOU as a developer:

try restarting transaction
How to avoid deadlocks?

Deadlocks are an hot spot on some rows. Hot spot means: Many (at least 2) connections do something (INSERT, UPDATE, REPLACE, DELETE, SELECT FOR UPDATE) on the same rows (which then are locked).

So how can you solve a Deadlock problem? Solving in the meaning of "reduce the probability of a Deadlock close to zero". Zero Deadlocks guarantee you can only achieve in no-concurrency scenarios.

• Look at the design of your application: Is it necessary that the 2 conflicting transactions are done in the way you are doing it right now? Can you change the behaviour of your application?
• Reduce locking time: As shorter the locks are hold as smaller the probability is that you run into a Deadlock situation. So make sure your conflicting transactions are running as fast as possible and the locks are hold as short as possible. Check that all the transactions are using perfect indexes.
• Reduce the amount of locks: The fewer rows your conflicting transactions lock the smaller is the chance for a Deadlock. For example: Several small INSERTs in several transactions instead of one huge INSERT in one big transaction should cause less locks and thus a smaller probability of Deadlocks. Only change (REPLACE, INSERT ON DUPLICATE KEY UPDATE) what is really needed to change. Check before you change, that rows need to be changed.
• Do your tasks less often. For example polling every 10 seconds instead of every second, or do refreshes of data less often.
• Timely defer concurrent transactions. For example 2 batch jobs: Do they really need to run at the exact same time?
• Reduce concurrency (parallelism), if possibly.
• Catch the Deadlock error and retry the transaction again a few milliseconds later. If this does not happen too often it is not a problem for your database.
• Change transaction isolation level to a level causing less locks. You can do that per session (SET SESSION ...) to reduce the impact on the whole system or globally (SET GLOBAL ...). But changing the isolation level globally potentially can have also an impact on other parts of your application or even completely different applications running on the same database.

And to be aware: You cannot completely avoid (100%) Deadlocks. They always can happen and you have to cope with it!

Do not mix up Deadlocks with Galera cluster conflicts which look similar at the first look but are not.

Literature

• Deadlocks, indexing and Primary Key's
• MariaDB Deadlocks
• MySQL Deadlocks in InnoDB

Taxonomy upgrade extras: deadlock

Also in 2022 there will be a FOSDEM (Free and Open source Software Developers' European Meeting) on 5 and 6 February 2022. This time again online from Brussels (Belgium).

MariaDB has again its own Devroom and the Call for Papers (CfP) is now open for your submissions. We are looking for interesting topics about your daily business, technical presentations, war stories, point of views of management, etc. The deadline for the CfP is before 21 December 2021.

For further information about how to submit a presentation please look here: https://mariadb.org/cfp-for-the-mariadb-devroom-fosdem-2022-now-open/

If you need any help or if you have any question please let us know...

Taxonomy upgrade extras: fosdem2022developersocial event

The MariaDB Connection ID exists since long ago. So why bother about the Connection ID? Because it is interesting and you can do some interesting things with the Connection ID like tracking statements in your connections and find where they come from your application code.

The MariaDB Connection ID is a strictly monotonic increasing number starting with 1 at server restart:

shell> mariadb --user=root --execute='SELECT CONNECTION_ID()\G' | grep CONNECTION_ID CONNECTION_ID(): 2372 shell> mariadb --user=root --execute='SELECT CONNECTION_ID()\G' | grep CONNECTION_ID CONNECTION_ID(): 2373 shell> mariadb --user=root --execute='SELECT CONNECTION_ID()\G' | grep CONNECTION_ID CONNECTION_ID(): 2374

The MariaDB documentation states [1]:

Returns the connection ID (thread ID) for the connection. Every thread (including events) has an ID that is unique among the set of currently connected clients.

The MariaDB documentation is only partly correct because the thread ID is something different and the term is used ambiguous. See further down.

The maximum number of connections created can be shown with:

SQL> SHOW GLOBAL STATUS LIKE 'Connections'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | Connections | 2322 | +---------------+-------+

But where else can I find the Connection ID?

Processlist

The Connection ID is shown in many different places inside your MariaDB database server. First of all you will find it in the process list:

SQL> SELECT id AS connection_id, user, host, IFNULL(CONCAT(SUBSTR(info, 1, 32), '...'), '') AS query FROM information_schema.processlist; +---------------+-------------+-----------+-------------------------------------+ | connection_id | user | host | query | +---------------+-------------+-----------+-------------------------------------+ | 2383 | root | localhost | SELECT id AS processlist_id, use... | | 6 | system user | | | +---------------+-------------+-----------+-------------------------------------+

Unfortunately it is named there just id so you have to know what it means.

You can easily show what all connections are doing while filtering out your own connection:

SQL> SELECT * FROM information_schema.processlist WHERE id != CONNECTION_ID(); +------+-------------+-----------------+------+-----------+--------+-------+------+---------------+-------+-----------+----------+-------------+-----------------+---------------+----------+-------------+------+ | ID | USER | HOST | DB | COMMAND | TIME | STATE | INFO | TIME_MS | STAGE | MAX_STAGE | PROGRESS | MEMORY_USED | MAX_MEMORY_USED | EXAMINED_ROWS | QUERY_ID | INFO_BINARY | TID | +------+-------------+-----------------+------+-----------+--------+-------+------+---------------+-------+-----------+----------+-------------+-----------------+---------------+----------+-------------+------+ | 2398 | app | localhost:35512 | NULL | Sleep | 4 | | NULL | 4771.722 | 0 | 0 | 0.000 | 81784 | 81784 | 0 | 14030 | NULL | 6946 | | 2392 | root | localhost | NULL | Sleep | 36 | | NULL | 36510.867 | 0 | 0 | 0.000 | 81784 | 81784 | 0 | 14021 | NULL | 3850 | +------+-------------+-----------------+------+-----------+--------+-------+------+---------------+-------+-----------+----------+-------------+-----------------+---------------+----------+-------------+------+

Note: The information_schema.processlist view is a superset of the command SHOW FULL PROCESSLIST and can be used with SQL means. So it has some advantages to retrieve the data from there...

InnoDB Monitor

Also in the InnoDB Monitor you can see the MariaDB Connection ID in at least 3 different places. The InnoDB Monitor is called as follows:

SQL> SHOW ENGINE INNODB STATUS\G

In the following 3 sections you will find the Connection ID:

------------------------ LATEST DETECTED DEADLOCK ------------------------ 2021-06-26 11:05:21 13c0 *** (1) TRANSACTION: TRANSACTION 17064867, ACTIVE 17 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 8 lock struct(s), heap size 3112, 7 row lock(s), undo log entries 2 MySQL thread id 7778, OS thread handle 0xb9c, query id 23386973 10.0.0.6 WPFieldUser updating Update wp_schema.trackfield_table Set reasonForrejection=4 where track_id='1406250843353122' *** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 241 page no 1541 n bits 680 index `track_id_FK` of table `wp_schema`.`trackfield_table` trx id 170648 67 lock_mode X waiting Record lock, heap no 139 PHYSICAL RECORD: n_fields 2; compact format; info bits 0 0: len 16; hex 31343036323530383433333533313232; asc 1406250843353122;; 1: len 4; hex 8000a4d8; asc ;; *** (2) TRANSACTION: TRANSACTION 17063837, ACTIVE 173 sec starting index read, thread declared inside InnoDB 5000 mysql tables in use 1, locked 1 14 lock struct(s), heap size 3112, 22 row lock(s), undo log entries 10 MySQL thread id 7765, OS thread handle 0x13c0, query id 23387033 10.0.0.6 WPFieldUser updating Update wp_schema.trackfield_table Set reasonForrejection=4 where track_id='1406251613333122' *** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 241 page no 1541 n bits 680 index `track_id_FK` of table `wp_schema`.`trackfield_table` trx id 170638 37 lock_mode X Record lock, heap no 139 PHYSICAL RECORD: n_fields 2; compact format; info bits 0 0: len 16; hex 31343036323530383433333533313232; asc 1406250843353122;; 1: len 4; hex 8000a4d8; asc ;; ------------------------ LATEST FOREIGN KEY ERROR ------------------------ 2021-08-19 15:09:19 7fbb6c328700 Transaction: TRANSACTION 543875059, ACTIVE 0 sec inserting mysql tables in use 1, locked 14 lock struct(s), heap size 1184, 2 row lock(s), undo log entries 1 MySQL thread id 124441421, OS thread handle 0x7fbb6c328700, query id 7822461590 192.168.1.42 fronmdual update INSERT INTO contact (user_id,kontact_id) VALUES (62486, 63130) Foreign key constraint fails for table `test`.`contact`: , CONSTRAINT `FK_contact_user_2` FOREIGN KEY (`contact_id`) REFERENCES `user` (`id`) ON DELETE CASCADE ON UPDATE CASCADE Trying to add in child table, in index `contact_id` tuple: DATA TUPLE: 2 fields; ... But in parent table `test`.`user`, in index `PRIMARY`, the closest match we can find is record: PHYSICAL RECORD: n_fields 41; compact format; info bits 0 ... ------------ TRANSACTIONS ------------ Trx id counter 2499 Purge done for trx's n:o < 2486 undo n:o < 0 state: running History list length 12 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION (0x7f70d6b93330), ACTIVE 3 sec mysql tables in use 1, locked 0 0 lock struct(s), heap size 1128, 0 row lock(s) MariaDB thread id 2398, OS thread handle 140122589112064, query id 14075 localhost 127.0.0.1 app Sending data select * from test.test Trx read view will not see trx with id >= 2499, sees < 2499

Unfortunately the Connection ID here is called thread id in all 3 different places. But we know at least which connection is causing Deadlock errors, Foreign Key errors or long running transactions.

PERFORMANCE_SCHEMA threads view

In the PERFORMANCE_SCHEMA.threads view you can also find the MariaDB Connection ID, called processlist_id. But here starts confusion again. MySQL introduced a new column thread_id which is mostly used in the PERFORMANCE_SCHEMA and contains the ID of a thread (not a connection):

SQL> SELECT thread_id, processlist_id FROM performance_schema.threads WHERE processlist_id = CONNECTION_ID(); +-----------+----------------+ | thread_id | processlist_id | +-----------+----------------+ | 2369 | 2321 | +-----------+----------------+

The processlist_id can be found also in the views: session_account_connect_attrs and session_connect_attrs in the PERFORMANCE_SCHEMA.

Via the thread_id you can match now your connection to various other views in the PERFORMANCE_SCHEMA: events_stages_*, events_statements_*, events_transactions_*, events_waits_*, memory_summary_by_thread_by_event_name, socket_instances, status_by_thread and user_variables_by_thread

There are related thread_ids in: threads.PARENT_THREAD_ID, metadata_locks.OWNER_THREAD_ID, mutex_instances.LOCKED_BY_THREAD_ID, prepared_statements_instances.OWNER_THREAD_ID, table_handles.OWNER_THREAD_ID and rwlock_instances.WRITE_LOCKED_BY_THREAD_ID.

sys Schema

Also in the sys Schema you find the thread_id (but not the connection_id) in the following views: io_by_thread_by_latency, latest_file_io, memory_by_thread_by_current_bytes, processlist, schema_table_lock_waits and session_ssl_status.

MariaDB Log files MariaDB Error Log file

Also in your MariaDB Error Log file you find the Connection ID. The Connection ID is the 2nd position in the Log. And sometimes you see the Connection ID also in the error message itself:

2021-11-25 16:22:34 1796 [Warning] Hostname 'chef' does not resolve to '192.168.56.1'. 2021-11-25 16:22:34 1796 [Note] Hostname 'chef' has the following IP addresses: 2021-11-25 16:22:34 1796 [Note] - 192.168.1.142 2021-11-25 16:22:34 1796 [Warning] Aborted connection 1796 to db: 'unconnected' user: 'unauthenticated' host: '192.168.56.1' (This connection closed normally without authentication) 2021-11-26 16:09:14 2397 [Warning] Access denied for user 'app'@'localhost' (using password: YES)
MariaDB General Error Log

The most important use of the Connection ID I see in the MariaDB General Query Log. Here you can find ALL the queries sent through connections to the database. You can easily search for a specific Connection ID and you will see exactly what a connection does or did:

211108 22:18:02 4568 Connect fpmmm_agent@localhost on using TCP/IP 4568 Query SET NAMES utf8 4568 Query SHOW GRANTS 4568 Query SELECT "focmm" 4568 Query SHOW GLOBAL VARIABLES 4568 Query SHOW /*!50000 GLOBAL */ STATUS 4568 Quit
MariaDB Slow Query Log

You can also find the Connection ID in the MariaDB Slow Query Log. But here again it is called Thread_id:

# Time: 210729 9:22:57 # User@Host: root[root] @ localhost [] # Thread_id: 34 Schema: test QC_hit: No # Query_time: 0.003995 Lock_time: 0.000114 Rows_sent: 2048 Rows_examined: 2048 # Rows_affected: 0 Bytes_sent: 79445 SET timestamp=1627543377; select * from test;
MariaDB SQL Error Log

Unfortunately the Connection ID is missing in the MariaDB SQL Error Log output:

shell> tail sql_errors.log 2021-11-26 16:58:36 app[app] @ localhost [127.0.0.1] ERROR 1046: No database selected : select * from test limt 10

We opened a feature request for this: SQL Error Log plug-in lacks Connection ID (MDEV-27129).

MariaDB Binary Log

Also in the MariaDB Binary Log the Connection ID is missing.

MariaDB Audit Plugin

But in the MariaDB Audit Plugin we will find again the Connection ID in the 5th column:

20211126 17:19:01,chef,app,localhost,2477,18407,QUERY,,'SELECT DATABASE()',0 20211126 17:19:01,chef,app,localhost,2477,18409,QUERY,test,'show databases',0 20211126 17:19:01,chef,app,localhost,2477,18410,QUERY,test,'show tables',0 20211126 17:19:02,chef,app,localhost,2477,18423,QUERY,test,'select * from test limt 10',1064 20211126 17:19:05,chef,app,localhost,2477,18424,READ,test,test, 20211126 17:19:05,chef,app,localhost,2477,18424,QUERY,test,'select * from test limit 10',0 20211126 17:19:38,chef,app,localhost,2477,18426,QUERY,test,'select connection_id()',0 20211126 17:19:52,chef,app,localhost,2477,0,DISCONNECT,test,,0
INFORMATION_SCHEMA

Also in some INFORMATION_SCHEMA views we will find the Connection ID: In THREAD_POOL_QUEUES.CONNECTION_ID, METADATA_LOCK_INFO.THREAD_ID and INNODB_TRX.trx_mysql_thread_id.

Other related topics to MariaDB Connection ID Thread Cache

When using the MariaDB Thread Cache it looks like the thread_id (and also the Connection ID) is changed each time a new connection is created. This is not what I expected, at least for the Thread ID. If I take a thread from the pool I would expect the same or at least another old thread_id again.

Connection Pooling

If you are using application side Connection Pooling different application connection handles will share the same DB connection. So you have to expect traffic from different application parts under the same Connection ID on the database side.

Pseudo thread ID

Since MySQL 8.0.14 there is a variable called pseudo_thread_id. It is for internal server use. Changing the variable on session level also changes the value of the function CONNECTION_ID(). I have no idea what this variable is used for.

Other related information

• Abbrechende MariaDB/MySQL Verbindungen
• Failed MySQL DDL commands and Galera replication

Taxonomy upgrade extras: connectionmax_used_connections

Das FromDual Seminarprogramm für 2022 steht. Mit unseren Schulungspartnern Linuxhotel in Essen, GfU Cyrus in Köln sowie der Heinlein Akademie in Berlin bieten wir auch 2022 wieder zahlreiche MariaDB und MySQL Schulungen an:

• MariaDB und MySQL für Fortgeschrittene
• Galera Cluster für MariaDB und MySQL

Im Seminar für Fortgeschrittene nehmen wir uns Themen wie Backup/Restore, Master/Slave Replikation, Galera Cluster sowie Datenbank-Konfigurations-Tuning und SQL Query Tuning vor und üben die einzelnen Punkte praktisch.

Im Galera Cluster Seminar nehmen wir alle Aspekte des Aufbaus und des Betriebs eines Galera Clusters durch und praktizieren das Ganz anschliessend ausführlich.

Den Umständen entsprechend werden die Seminare entweder vor Ort, remote oder hybrid durchgeführt.

Bei allfälligen Fragen bitten wir Sie, mit uns oder unseren Schulungspartnern Kontakt aufzunehmen und diese zu klären.

Taxonomy upgrade extras: trainingschulungseminarmysqlmariadbgalera2022replikation

From 25 to 29 October 2021 (calendar week 43) we will have another MariaDB / MySQL advanced training in the Linuxhotel in Essen (Germany). The training is in German and will take place on-site (3G!). There are still some places free!

More details about the training you can find here.

Nearly everybody does backups. But nobody needs backups! What everybody wants and needs is a working restore not a working backup...

So how to make sure that your backup is working for the restore? There are a few things you can do already during your backup:

• Check that your backup was running fine. For example by checking the return code of your backup.
• Check the runtime of your backup. If the runtime of your backup significantly changed, it is worth to have a closer look at the backup.
• Check the size of your backup. If the size of your backup significantly changed, it is worth to have a closer look at your backup.
• And finally make your monitoring system aware if the backup was NOT running at all and if you are sure your backup is really triggered...

Backup test with FromDual Enterprise Tools

All this functionality is integrated in the newest releases of FromDual Backup and Recovery Manager for MariaDB and MySQL (brman) and the FromDual Performance Monitor for MariaDB and MySQL (fpmmm) in combination with the great monitoring solution Zabbix.

You have to run Backup Manager with the options --fpmmm-hostname and --fpmmm-cache-file:

shell> bman --target=brman:secret@127.0.0.1:3306 --type=full --policy=daily \ --fpmmm-hostname=mariadb-106 --fpmmm-cache-file=/var/cache/fpmmm/fpmmm.FromDual.mariadb-106.cache

then Backup Manager knows that it has to collect the metrics: return code, backup time and backup size and deposits them for the next Performance Monitor run. The next fpmmm run then will automatically gather these data and send it to the Zabbix Server.

In the Zabbix Monitor then you can see how your backup behaves:

and you get alerts if backups failed or did not even happen:

Restore Tests with FromDual Enterprise Tools

If your backup was running fine, returned with a zero return code, had the usual size and was running in a normal amount of time it does not necessarily mean that you also can do a restore with this backup!

For the sissies among us we also should do a restore test. To justify restore tests we have worked out 2 different concepts:

Staging systems restore test

Many of us have different stages of systems: Production, Quality Assurance, Integration, Testing and Development. All those systems need to be provisioned from time to time with new and fresh data from production. These systems would be ideal candidates for your daily restore tests. Also here the FromDual Enterprise Tools can help you: The backup, done with Backup Manager is shipped automatically to the other system where the Recovery Manager is restoring the database. Booth tools report their results to the Performance Monitor which sends the results to Zabbix. And Zabbix sends complains to the administrators if the backup or the restore did not happen or failed:

Spider system restore test

If you have many systems to do restore test or if you do not want to expose your precious data to other stages than production we recommend you the restore spider concept: Every production database does its daily backup on its local backup store. And additionally you have a centralized restore system which has access to the backup of each database. So this central spider restore system grabs now the backup of each database one after the other and does a restore on its restore system. The result will be reported to the Performance Monitor again and you get notice by Zabbix is the Restore did not happen, not work or was talking longer than a predefined time:

Taxonomy upgrade extras: BackupRestorebrmanfpmmmmonitoring

FromDual has the pleasure to announce the release of the new version 2.2.4 of its popular Backup and Recovery Manager for MariaDB and MySQL (brman).

The new FromDual Backup and Recovery Manager can be downloaded from here. The FromDual Repositories were updated. How to install and use the Backup and Recovery Manager is described in FromDual Backup and Recovery Manager (brman) installation guide.

In the inconceivable case that you find a bug in the FromDual Backup and Recovery Manager please report it to the FromDual Bugtracker or just send us an email.

Any feedback, statements and testimonials are welcome as well! Please send them to feedback@fromdual.com.

Upgrade from 2.x to 2.2.4 shell> cd ${HOME}/product shell> tar xf /download/brman-2.2.4.tar.gz shell> rm -f brman shell> ln -s brman-2.2.4 brman
Changes in FromDual Backup and Recovery Manager 2.2.4

This release is a new minor release. It contains mainly bug fixes. We have tried to maintain backward-compatibility with the 1.2, 2.0 and 2.1 release series. But you should test the new release seriously!

You can verify your current FromDual Backup Manager version with the following command:

shell> fromdual_bman --version shell> bman --version shell> rman --version
General

• Redhat 8 RPM package added to FromDual repository.
• Stripped unnecessary files from tarball.
• From now on we fully support MySQL 5.7 to 8.0.
• From now on we fully support PHP 8.
• From now on we fully support MariaDB 10.3 to 10.6.
• All old PHP 5 stuff was removed.
• Library from myEnv updated.
• Distribution distinguishing code was cleaned-up and Ubuntu, Rocky Linux, AlmaLinux and CloudLinux should pass checks correctly now.

FromDual Backup Manager

• Schema backup was not ignoring SYS schema when doing test for non-transactional (Aria) tables. This test is done correct now.

FromDual Recovery Manager

• Restore return code and restore time hook added for integration into fpmmm monitoring. Automated restore tests are better supported now.

FromDual brman Catalog

• none

FromDual brman Data masking / data obfuscating

• none

Testing

• All tests passed for MySQL 8.0.
• All tests passed with PHP 8.
• Various tests added.
• All tests passed for MariaDB 10.6.

Subscriptions for commercial use of FromDual Backup and Recovery Manager you can get from from us.

Taxonomy upgrade extras: BackupRestoreRecoverypitrbrmanreleasebmanrmanFromDual Backup and Recovery Manager