A customer rose a support case with a problem on his Master-Master set-up. The 2^nd Master claims to have a problem:

master2> SHOW SLAVE STATUS\G
                ...
    Master_Log_File: master1-bin.000014
Read_Master_Log_Pos: 97975045
   Slave_IO_Running: No
  Slave_SQL_Running: Yes
Exec_Master_Log_Pos: 97975045
      Last_IO_Errno: 1236
      Last_IO_Error: Got fatal error 1236 from master when reading data from binary log:
                     'Client requested master to start replication from impossible position'

What has happened?

Be careful when using SAN (Storage Area Networks) or similar shard storage solutions (and any other virtualization, consolidation or cloud solutions).

This week it happened again: A customer called us because he had some troubles with his on-line shop (note the date!). Everybody in his company complained that the databases are answering slowly.

Vorsicht bei der Nutzung von SAN (Storage Area Networks) oder ähnlichen Shard Storage Lösungen (und allen anderen Virtualisierungs-, Konsolidierungs- oder Cloud-Lösungen).

The following tools we use sometimes for our consulting engagements...

Caution: These tools are NOT for production use! Use with care!

Or an active-active fail-over cluster à la VMware.

Today I have learned about a totally crazy/cool looking architecture where the expensive VMware ESX server was replace by a free/cheap VMware Workstation version in combination with DRBD.

Basically DRBD we name "the little man's SAN" and that is exactly what this customer is doing. He replaced the SAN with DRBD and now he can easily move one VMware instance to the other host.