It is about performance on FromDual GmbH

Thanks, I appreciate the

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 07 Jan 2011 23:44:29 +0200

Thanks, I appreciate the effort you are taking to find the truth. -Bradley

I didn't mean to be flaming,

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 07 Jan 2011 15:11:49 +0200

I didn’t mean to be flaming, I’m sorry if it seemed that way. And I apologize for not noticing that the response to my comment wasn’t from the original author.

I guess I got a different message from the original posting. The original posting observed that indexes cost something. But what’s the point of that observation? The point is to do something to your database to fix the slowness. The original posting implied that dropping some indexes would be one fix to the problem. The chart says fewer indexes means higher performance. So far, so good.

However, there are other things you could do. For example, switch to InnoDB or TokuDB. (There may be other approaches such as using some NoSQL solution, but let’s put those aside for now.)

I guess I misread the original posting. Upon re-reading, it seems like the data in the original posting doesn’t look so bad. After all, inserting 500K rows in 3 minutes instead of 90 seconds to get 5 indexes doesn’t seem so bad. Sure, indexes cost something, but it’s not much on this measurement. (In fact, for such a small database, perhaps MyISAM is faster than InnoDB. But the problems I’ve seen don’t become difficult until the databases get larger.)

Here’s a much more interesting experiment (at least more interesting to me): What happens if a) the tables are bigger than main memory, and b) the indexes are random (or nearly random) values (things like UUID-PKs). Check out Mark Callaghan’s postings on his MySQL at Facebook page to see the advantage that InnoDB offers. (And check out the graphs on Tokutek.com to see the additional advantages that TokuDB offers. Or don’t if you really don’t want to know about a non-GPL solution.)

I guess my point is that if you are finding that the cost of maintaining indexes is slowing down your DML, dropping the indexes isn’t your only option. You may like the indexes because they speed up queries. Given a fixed hardware budget and a fixed …

It is about performance

oli.sennhauser@fromdual.com (Oli Sennhauser) — Fri, 07 Jan 2011 13:13:53 +0200

You are correct that indexes cost something in InnoDB. However, if you measure InnoDB you will likely find it is much faster than MyISAM for insertions when maintaining an index on a large table. My measurements (and others, such as Mark Callaghan at Facebook) say that InnoDB is faster (maybe 4x) for this kind of load.

Also you are correct that some people will not consider using a non-GPL storage engine, and that’s fine. That’s part of why I am pointing out that InnoDB is far better suited than MyISAM for the load you measured.

Some people are willing to use a non-GPL storage engine if it offers some other advantage, however. In the case of TokuDB, one advantage is more than a 10x performance improvement for this kind of load. (There are other advantages such as compression.)

I read your home page, which says you offer “vendor neutral” consulting, and that you are in the business of recommending the best solution to meet your clients needs. I hope that there are some situations where TokuDB would be the best solution to meet your clients needs, and if you find such a client, I hope we can work together.

-Bradley

Use a better indexer

oli.sennhauser@fromdual.com (Oli Sennhauser) — Thu, 06 Jan 2011 02:30:22 +0200

Using MyISAM is just asking for poor performance. Try using a storage engine that is actually good at indexing.

For example, InnoDB offers a substantial improvement (perhaps a factor of 4) over MyISAM for maintaining indexes. InnoDB uses an change buffer to achieve that performance.

For really high DML operations, try TokuDB (tokutek.com), which is about ten times faster than InnoDB. TokuDB uses fractal tree indexes to achieve its performance.

Both InnoDB and TokuDB are transactional and can recover from crashes. MyISAM will corrupt your data on a crash.