FromDual GmbH

How the Lack of a Primary Key May Effectively Stop the Slave

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 02 May 2017 09:50:52 +0000

Most (relational) DBAs and DB application developers know the concept of a primary key (“PK”) and what it is good for. However, much too often one still encounters table definitions without a PK. True, the relational theory based on sets does not need a PK, and all operations (insert, select, update, delete) can also be done on tables for which no PK was defined. If performance doesn’t matter (or the data volume is small, a typical situation in tests), the lack of a PK does not immediately cause negative consequences.

But recently, we had several customers who (independent of each other) had big tables without a PK in a replication setup which they wanted to delete, and they all suffered severely from that: Their replication did not progress, the slave lag grew larger and larger, and all this without reporting any error. When I say “larger and larger”, I mean it: I’m not talking about minutes or even hours, I’m talking about days!

Observations

In all cases, the situation could be summarized as:

They were running a traditional, asynchronous replication using the “row” format.
They had a table without PK with many entries, say a million rows.
This table wasn’t needed any more, and someone issued a “delete from T” statement on the master.
From this moment, the slave did not show any sign of progress: The output of “show slave status
G”
- listed both the IO and the SQL slave thread as running,
- reported the same log positions without any change,
- did not mention any slave error,
- but the slave lag grew in sync with the passing time.

Looking at the slave’s machine load, say using “vmstat”, you could see a certain amount of CPU load in user mode, possibly some “waiting for IO”, and some read IO (group “io”, column “bi” = “block input”). Checking with “top”, you could see that the MySQL server process was the only significant CPU load, and closer inspection …

Deleted files still (growing) on disk

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 25 Aug 2016 10:34:16 +0200

For this phenomenon, the relevant documentation is here: deleting files. Close to the top, see this sentence:

The unlink function deletes the file name filename.
If this is a file’s sole name, the file itself is also deleted.
(Actually, if any process has the file open when this happens,
deletion is postponed until all processes have closed the file.)

This is the standard programming technique to ensure that temporary files are automatically cleaned up on a program or machine crash. Often this subject comes up as the difference in the output of "du" (follows directory entries) and "df" (checks file system space information).

Jörg Brühe

Multiple MySQL Instances on a Single Machine

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 28 Jul 2016 13:11:03 +0000

Typically, on a single machine (be it a physical or a virtual one) only a single MySQL instance (process) is running. This is perfectly ok for all those situations where a single instance is sufficient, like for storing small amounts of data (RedHat using MySQL for postfix, KDE using it for akonadi, …), as well as those where a dedicated machine per MySQL instance is appropriate (high CPU load, memory fully loaded, availability requirements).

But there are also those users who want to (or would like to) have multiple instances which would still fit into a single machine. Even among them, a single instance per machine is typical. For this, there are good reasons:

MySQL comes with defaults for files (config file, error log, …) and directories (data directory, binlogs, …) which would cause conflicts between multiple instances (unless they are changed).
The scripts coming with MySQL, especially the automated start/stop with machine reboot/shutdown, are written for a single instance only.
Last but not least: The instructions. those in the manual as well as the many “How to setup …” in the Web, cover a single instance only.

Because of this, users often either restrict themselves to a single instance, or they set up several virtual machines (or containers) holding a single instance each.

But that overhead (both in software and in labour) isn’t necessary: There is a way out, supporting easy handling of multiple MySQL instances on a single machine directly, without containers or VMs. This is our “MyEnv” package, available for download here, licensed under the GPL.

What Does MyEnv Do?

MyEnv cares about two aspects which in combination provide easy use of multiple instances:

It helps to configure multiple MySQL instances without overlap, so they won’t collide with each other.
It maintains separate environments, each to manage and access one specific instance.

Each environment contains the path to the binaries (so the instances can use different versions), …

Past and Future Conferences, and Talks Around MySQL

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 11 Apr 2016 15:25:25 +0000

Time flies, and my blogging frequency is quite low. More frequent would be better, but knowing myself I’ll rather not promise anything ;-)

Still, it is appropriate to write some notes about CeBIT, the “Chemnitzer Linuxtage 2016”, and future events.

CeBIT

CeBIT was running from March 14 to 18 (Monday till Friday) in Hannover, Germany, and I will leave the general assessment to the various marketing departments as well as to the regular visitors (to which I do not belong).

In order to meet our current customers as well as potential future ones, FromDual had a booth in the “Open Source Forum”. We displayed a Galera Cluster, running on three tiny headless single-board Linux machines, and showed how it reacts to node failures and then recovers all by itself, without any administrator intervention. Many of our visitors were fascinated, because a HA solution would be a good fit in their solution architecture. We had got several stuffed dolphins “Sakila”, the traditional MySQL symbol, and all of them found new homes (typically with the words “for my grandchild”). :-)

IMHO, the “Open Source Forum” had deserved a better visitor attraction than it really got - placing it into one hall with document management systems was no good fit, research and development might have been more appropriate. The forum had an area for talks which were running all five days, I consider John “Maddog” Hall (who had provided an Alpha machine to Linus Torvalds decades ago) and Prof. Klaus Knopper (who is maintaining the “Knoppix” live distribution) the most prominent speakers. FromDual’s Oli Sennhauser talked about the new features of MySQL 5.7, you can get the slides via the FromDual download page.

Chemnitzer Linux-Tage

The weekend following CeBIT, March 19 and 20, had been selected for the Chemnitzer Linux-Tage. Like in the previous years, the conference attracted many visitors from all over Germany as well as from some neighbouring countries, and …

On Files, the Space They Need, and the Space They Take

oli.sennhauser@fromdual.com (Oli Sennhauser) — Tue, 09 Feb 2016 14:55:09 +0000

xfs Users, Take Care!

Recently, we had a customer ask: Why do many files holding my data take up vastly more space than their size is? That question may sound weird to you, but it is for real, and the customer’s observation was correct. For a start, let’s make sure we are using the same terms.

The size of a file is the number of bytes it will deliver if it is read sequentially from start to end.
The space it takes up is the sum of all disk pages which are used to hold the file’s data, or to locate those data pages (“indirect” blocks in Unix/Linux terminology).

Every Unix/Linux admin knows (or at least should know) that a file may take up less disk space than its size is. This happens when not all bytes of the file were really written, but the write pointer was advanced via “seek()”, leaving a gap. Disk pages which are completely contained in such a gap will not be written, and reading these positions will produce bytes containing zero. This is called a “sparse file”. You will find some remarks about them in our blog at https://fromdual.com/mysql-cluster-sparse-files, or search the net for that term.

The Customer’s Message

Now that we have brought those basics into active memory again, let’s return to the original question: Can there be files which take up vastly more space than their size is? We will not consider potential administrative overhead (pointers to pages), because to the customer a file of slightly more than 4 GB was reported to take up 8.1 GB disk space - see this quote from his mail (file name changed):

$ ls -l some_table#P#p01.ibd
-rw-rw---- 1 mysql mysql 4307550208 Jan 4 01:06 some_table#P#p01.ibd

$ du -hs some_table#P#p01.ibd
8,1G some_table#P#p01.ibd

Luckily, the customer’s mail mentioned the file system: It was not one of the “ext” family (ext2, ext3, or etx4), but rather they are using xfs. This gave me a hint to search for information, and Google provided several pointers, IMO the most helpful ones where …

How to Get a Galera Cluster Into Split Brain

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 23 Oct 2015 17:02:07 +0000

“Split Brain” is the term commonly used for a cluster whose nodes have different contents, rather than identical as they should have. Typically, a “split brain” situation is the DBA’s nightmare, and the Galera software is designed to avoid it. Galera is very successful in that avoidance, and it needs some special steps by the DBA to achieve “split brain”. Here is how to do it - or, for most DBAs, what to avoid doing to not get a split-brain cluster.

Galera’s Design

First, let’s remember how Galera is operating:

The Galera software ensures that all nodes participating in a cluster will start from identical contents, by doing a “snapshot state transfer” (SST) of all current data to a newly joining node.
When the cluster is running, Galera transfers all changes (transactions) to all cluster nodes and applies them (or rolls back and ignores, in the case of a conflict).
If some connections get lost, all nodes check whether they “have quorum” (belong to a majority), and stop serving requests if they don’t.
When a disconnected node re-joins the cluster, it gets all meantime changes transferred (“incremental state transfer” IST) and so makes its contents current.
Should that be impossible, because some of those changes have become unavailable (log purge), a full transfer (SST) is done.

By this design, the Galera software successfully avoids getting into a “split brain” situation.

Of course, the quorum is a well-known concept. The old term for it is “majority consensus”, and the approach is built on a simple principle:
In any set (of cluster nodes), there cannot be two (or more) non-overlapping subsets which both contain a majority of the elements.
So if some loss of connectivity splits the cluster into subsets, at most one of them can “have quorum”, all others will stop serving requests, and there cannot be two (or more) different directions in which the contents (data) changes.

What …

Change: Port needed

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 05 Aug 2015 16:42:44 +0200

With Galera version 25.3.10, the “garbd” daemon would not start unless I appended the port number (default: 4567) to the IP addresses of the cluster nodes.
The failure symptom was that an exception “gu::NotSet” was thrown, without any helpful information what it was really missing.

The Upcoming Leap Second

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 29 Jun 2015 17:56:45 +0000

The press, be it the general daily newspaper or the computer magazines, is currently informing the public about an upcoming leap second, which will be taken in the night from June 30 to July 1 at 00:00:00 UTC. While we Europeans will enjoy our well-deserved sleep then, this will be at 5 PM (17:00) local time on June 30 for Califormia people, and during the morning of July 1 for people in China, Japan, Korea, or Australia. (Other countries not mentioned for the sake of brevity.) This is different from last time, when the leap second was taken in the night from Saturday to Sunday (2012-July-1 00:00:00 UTC), so it was a weekend everywhere on the globe.

We have got several requests from our customers about this upcoming leap second, whether they need to take any special precautions or whether they “are safe”. Well, obviously nobody is “safe” from the leap second in the sense that it would circumvent them, everybody will encounter it on their systems. The concern is whether they have to expect any trouble.

For the people operating MySQL (or any other DBMS), this issue is threefold:

How will the operating system behave?

How will the database system behave?

How will the applications behave?

Let us look at the operating system first, dealing with Linux only. (All other operating systems don’t show up with significant numbers in our customer base.)
Linux measures the time in seconds (since Jan 1, 1970, 00:00:00 UTC), and it does not include the leap seconds in this counting. When the leap second is taken, the timestamp value (those seconds) will simply not be advanced at the end of the regular second, but it will re-use its current value in the leap second. As a result, the conversion of timestamps into common time reckoning will still produce values from 0 to 59 for the minute as well as for the second, there will not be a 60. Another consequence is that there is no way to tell the leap second from the preceding regular one.

MySQL always takes the time information from the Linux kernel, so …

Linuxtag: Knowledge and People - and New Colleagues?

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 04 Feb 2015 10:49:29 +0000

At FromDual, we are currently preparing for our participation in the “Chemnitzer Linux-Tage” in March.
While we don’t yet know whether the programme committee accepted our proposed talks, we will have a booth and hope for interesting exchanges with others from the MySQL, database, Linux, … world. Of course, we will also mention that we are looking for additional colleagues - there are so many tasks that we need more people to handle them all. (In case you got curious, look here: http://www.fromdual.com/mysql-dba-2014-12-de )

While I attended the Chemnitzer Linux-Tage already last year (thank you, Frank Hofmann, for recommending them!), it will be the first time there for FromDual as a company. I enjoyed the informal atmosphere, which made it easy to get into contact with new people, to learn, and to discuss both facts and opinions.

To those of you who didn’t yet attend such events, I can only say: You are missing some great experiences! It is good to get stimulated by others, by their problems and their experiences, so that one’s thoughts are broadened and start considering new fields. Often, they lead to insights that are helpful for one’s own daily work.

In the opposite direction, we at FromDual hope that during the Linux-Tage we can help others to make (better) use of MySQL in all its variants, with its surrounding tools and products, so that everybody will profit.

The Chemnitzer Linux-Tage will take place on March 21 and 22, 2015, in Chemnitz, Germany. Parts of the programme will probably be delivered in English, so even those of you who don’t understand German might enjoy coming there. If you now consider a visit, you will find further information here:
https://chemnitzer.linux-tage.de/2015/de

We will be happy to meet you in Chemnitz - to talk about whatever interests both you and us, maybe even about you joining us?

Later Findings ...

oli.sennhauser@fromdual.com (Oli Sennhauser) — Wed, 21 Jan 2015 17:49:44 +0200

… show that this is expected behavior:

Galera will replicate a DDL statement first, before it gets executed locally.
Local execution will fail, because the "super" privilege is a global one, it cannot be granted for a specific database "db.*" but only for all of them "*.*"
Of course, execution on the other nodes will fail in the same way.

So: While it looks ugly, it is nothing to worry about.

Introducing Myself: Jörg Brühe

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 19 Jan 2015 20:56:19 +0000

For some time already, FromDual’s “Our Team” page lists me, and it even reveals that I joined in September, 2014. Also for some time, the list of FromDual blogs contains an entry “Jörg’s Blog”, but it doesn’t lead to any entries. It is high time to fix this and create entries, starting with an introduction of myself.

Often, in such introductions people use the phrase of “the new kid on the block”. I won’t. If I am to use those words, I will arrange them as “the kid on the new block”. The reason is that I don’t feel as a new kid in the MySQL village (or is it a city?), let alone in DBMS country.

Ever since I left university (Technical University of Berlin, Germany), I have been involved in SQL DBMS development. After my previous product’s team had been dissolved in Berlin and maintenance moved to Riga, Latvia, as a cost-cutting measure, I joined MySQL AB in 2004 as a member of the Build Team. Some of you will remember my name from MySQL release announcements, others may have seen it in bug reports, newsgroup entries, or other MySQL communication. Together with the other MySQL colleagues, I got acquired by Sun in 2008 and then by Oracle in 2010.

In 2012, I decided I wanted a culture change, from a highly (and very strictly) organized corporation to a smaller, more flexible entity. That brought me to an internet site, where I worked as a DBA, concentrating on their (more than 100) MySQL installations. It was very interesting to view MySQL from the DBA side now, I got a big load of new experiences and learned a lot about using, running, and automating MySQL (rather than building and testing it).

That might have continued, but as usual the world is changing, and my DBA job was no exception: Company strategy was to shift those tasks more to the various application teams and to do without a central point for MySQL. It is a sign how far MySQL knowledge has spread, and how little effort may be sufficient to run MySQL, that this worked. …

oli.sennhauser@fromdual.com (Oli Sennhauser) — Mon, 01 Jan 0001 00:00:00 +0000