Use dconf-editor (from package dconf-tools) and go to org.gnome.desktop.wm.keybindings. There you have it.

Why would I want to change keyboard shortcuts? Some of them (namely Alt+F1, Alt+F7 and Alt+F8) clash with IntelliJ IDEA’s shortcuts. That’s why!

But it is hard to formulate the right assertions. What if the test runs on a beefier machine or in another environment?

Did performance degrade?

I just want to answer a simple question: Did performance degrade? (And if, when?)

Lately I came across junit-benchmarks. This framework allows you to visualize the performance history of select unit tests.
This is great, but I want my tests to fail if the performance degraded by some factor. A test run should fail, if it performs X% worse than the run before or Y% worse than the average of the last Z runs.

Extending junit-benchmarks

So I forked the junit-benchmarks, and created a project over at github.

Now you can assert that performance did not degrade:

public class MyTest {
 
    /** This is needed to enable junit-benchmark. */
    @Rule
    public MethodRule benchmarkRun = new BenchmarkRule(h2consumer);
 
    /**
     * This test fails, if it performs 20% worse than the last test run or 10% worse than the average of the last 10 runs.
     */
    @Test
    @BenchmarkOptions(perfDiffToLastRun = 0.2d, perfDiffToAverage = 0.1d, perfAverageOverRuns = 10)
    public void test() throws Exception {
        // Do something ...
    }
}

Note: Consumer H2 must be enabled. See junit-benchmark documentation.
As for all junit-benchmark settings, you can set defaults via JVM command line options.

But new features might cost performance

The approach above works if performance assumptions never change. But they do. So, performance tests will fail after adding a new feature that decreases performance on purpose. What should we do? Delete all the performance history? You could do that. Another way is to tell junit-benchmark that it should ignore all performance data up to a certain test run:

/**
 * This test fails, if it performs 20% worse than the last test run or 10% worse than the average of the last 10 runs.
 * But it ignores the 96 runs.
 */
@Test
@BenchmarkOptions(perfDiffToLastRun = 0.2d, perfDiffToAverage = 0.1d, perfAverageOverRuns = 10, perfIgnoreUpToRun = 96)
public void test() throws Exception {
    // Do something ...
}

Get the code

http://github.com/optivo/junit-benchmarks

The problem

Look at this simple example: Let’s say we have a table customer:

CREATE TABLE customer {
 
    id UNSIGNED INT NOT NULL,
    firstname VARCHAR(100) NOT NULL,
    lastname VARCHAR(100) NOT NULL,
    city VARCHAR(100) NOT NULL,
 
    PRIMARY KEY(id)
}

In addition to that we store orders customers made in Hadoop. An order includes: customerId, date, itemId, price. Note that these structures serve as a very simplified example.

Let’s say we want to find the first 50 customers, that placed at least one order sorted by firstname ascending. If both tables were in MySQL we could use a single SQL statement like:

SELECT DISTINCT c.id, c.firstname FROM customer c JOIN ORDER o ON c.id = o.customerId ORDER BY c.firstname ASC LIMIT 50

Having the orders in Hadoop we have basically two options:

1. We write a Map-Reduce job that reads all customers from MySQL and joins them with the orders stored in Hadoop’s HDFS. The output is sorted by firstname ascending. From the result we use only the first 50 entries.
2. We write a Map-Reduce job to extract all distinct customerIds, write them to a table in MySQL and use a SELECT with a JOIN.

In most cases option 2 will be the better choice if we have a non-trivial number of rows in our customer table. And that’s for three reasons:

1. MySQL is not optimized for streaming rows. As our Map-Reduce job would always have to read the whole table, we would stream a lot.
2. You cannot easily write something like a LIMIT clause in Map-Reduce. Even if you could, you’d likely have to read through all customer entries anyway. So the amount of data processed by the Map-Reduce job is significant higher if you use aproach 1.
3. If you just started to move to Hadoop, most of the data structures like categories, product information etc. are still kept in MySQL and most of the business logic relies on SQL. In most application you would not move all your data to Hadoop anyway. So storing Hadoop’s result in MySQL simply integrates better with your existing application.

So, storing Map-Reduce results in MySQL seems to be the better option most of the time. But you still have to write all customerIds extracted by the Map-Reduce job into a table. And that is a performance killer. Even if you use HEAP tables it puts a lot of pressure on MySQL. Other options like CSV storage engine are not feasible since they do not provide any keys. And joining without a key is never a good idea.

Introducing: MySQL UDF csv_find() and csv_get()

One of the big advantages of Map-Reduce is that it produces output sorted by whatever we want. So we could output sorted CSV files. And we could perform binary search on these sorted CSV files. Great!

I have written two MySQL User Defined Functions (UDF) that provide find and get functionality on sorted CSV files.

How to use it?

Taking our example from above we transfer the resulting CSV file from HDFS to the local filesystem of our MySQL server and write a query like this:

SELECT * FROM customer WHERE csv_find('/tmp/myHadoopResult.csv', customer.id) = 1 ORDER BY firstname ASC LIMIT 50

And this is a lot faster than inserting the Map-Reduce result into a table. It might even be faster than our original SELECT statement where we assumed both tables customer and order are in MySQL since we are not using a JOIN at all. More on performance later on.

How does it work?

On initialization of csv_find the CSV file will be loaded into memory using mmap. And since the first column of the CSV file is sorted in ascending order we can simply use binary search on each call to csv_find.

If you need to access other columns of a CSV use csv_get(<file expression>, <key expression>, <column expression>). Example:

SELECT customer.lastname, csv_get('purchases.csv', customer.id, 2) AS price FROM customer

assuming that column 2 contains the price of a product purchased.

Prerequisites

The following assumptions are made and must be met by your CSV files:

• Column delimiter is ‘\t’ and row delimiter is ‘\n’. You can change this at compile time.
• The first column must be sorted in UTF-8 binary ascending order. “binary” means that it has to be sorted by byte value and not by a specific collation. For example ‘ä’ (0xc3 0xb6) comes after ‘z’ (0x7a). In bash you would sort a file like this:

  LC_ALL=C sort < input.csv > ordered.csv

  Remember that sorting comes for free in Map-Reduce.

• No escaping is done. If you need it, you could do the following: First, escape everything in your CSV, say by replacing ‘\n’ with ‘\\n’ and then use csv_find or csv_get like this:

  csv_find(<file expression>, REPLACE(<key expression>, '\n', '\\n'))

• Some MySQL APIs (at least JDBC) treat results of an UDF as binary data. You have to explicitly cast the return value of csv_get like this:

  SELECT CAST(csv_get(<file expression>, <key expression>, <column expression>) AS CHAR)

For more information take look into the source code documentation.

Usage patterns other than integrating with Hadoop

We use csv_find and csv_get not only to integrate with Hadoop but to integrate multiple MySQL servers. To make data from one MySQL server available in another you could export it like this:

SELECT * FROM customer WHERE  <some condition> ORDER BY BINARY id ASC INTO OUTFILE '/tmp/customer.csv'

Then copy the file over to the other MySQL server (or use NFS). Of course you could use FEDERATED storage engine. We decided not to because it has/had some glitches.

Another useful application is to replace complicated JOINs or SUBSELECTs. MySQL is good at performing some JOINs but really poor at a lot others, especially SUBSELECTs.

A brief performance evaluation

First we create a test CSV file:

#> for a in $(seq 1000000 2000000); do echo $a >> /tmp/random.csv; done

Then we load it into a table:

mysql> CREATE TABLE rand (id VARCHAR(255) NOT NULL, PRIMARY KEY(id));
Query OK, 0 ROWS affected (0.00 sec)
 
mysql> LOAD DATA INFILE '/tmp/random.csv' INTO TABLE rand;
Query OK, 1000001 ROWS affected (5.60 sec)

To test performance of JOIN vs. csv_find we create a second table containing the same rows:

mysql> CREATE TABLE rand2 (id VARCHAR(255) NOT NULL, PRIMARY KEY(id));
Query OK, 0 ROWS affected (0.01 sec)
 
mysql> LOAD DATA INFILE '/tmp/random.csv' INTO TABLE rand2;
Query OK, 1000001 ROWS affected (5.75 sec)

We see that importing 1 million rows already took 5.75 seconds.

Now lets compare the actual JOIN and csv_find:

mysql> SELECT COUNT(*) FROM rand JOIN rand2 ON rand.id = rand2.id;
1 ROW IN SET (2.37 sec)
 
mysql> SELECT COUNT(*) FROM rand WHERE csv_find('/tmp/random.csv', id) = 1;
1 ROW IN SET (1.83 sec)

We see 1.83 seconds for csv_find vs. 2.37 seconds for a JOIN.

Taking the time spent in LOAD DATA into account we even have 1.83 seconds vs. 8.12 seconds meaning csv_find is 4 times faster.

Since most Map-Reduce jobs do not use LOAD DATA but a ton of INSERT statements the real performance might be even worse. Not to mention the load massive INSERTs put on the MySQL server.

rand2 is an InnoDB table. Let’s retry with a memory table:

mysql> SET max_heap_table_size = 64 * 1024 * 1024;
Query OK, 0 ROWS affected (0.00 sec)
 
mysql> CREATE TABLE rand3 (id VARCHAR(10) NOT NULL, PRIMARY KEY(id)) ENGINE=HEAP;
Query OK, 0 ROWS affected (0.00 sec)
 
mysql> LOAD DATA INFILE '/tmp/random.csv' INTO TABLE rand3;
Query OK, 1000001 ROWS affected (1.94 sec)
Records: 1000001  Deleted: 0  Skipped: 0  Warnings: 0
 
mysql> SELECT COUNT(*) FROM rand JOIN rand3 ON rand.id = rand3.id;
1 ROW IN SET (1.80 sec)

As you can see, execution time of both queries is nearly equal, but we still need 1.94 seconds to load the data into table. Thus csv_find is still twice as fast compared to a JOIN on a HEAP table.

But, did you notice this statement?

SET max_heap_table_size = 64 * 1024 * 1024

We had to raise the maximum heap table size since the contents of our 7.7 MB test file would not fit into the default 16 megabytes.

Actually the HEAP table uses about 50 MB of RAM compared to just exactly 7.7 MB for csv_find.

And because RAM is a limiting factor you cannot use HEAP all that often anyway. csv_find and csv_get allocate as much memory as the file size. You can limit the maximum allowed file size at compile time.

Where to download?

Download mysql_udf_csv_binary_search-0.1.tar.gz.

This package includes instructions on how to install (see README) as well as a comprehensive test suite containing both, unit and integration tests (make test).

A note on Windows: Since I don’t use Windows there are no build instructions for this OS. I tried to write portable code but since I cannot test it, I don’t know if it is working. It would be great if someone out there could contribute a Windows version.

Code has been tested on MySQL version 5.1.41 as well as 5.0.83.

Final words

This package provides fast and simple integration of sorted CSV files coming from any source.

Comments and improvements are welcome.


java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)
com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160)
com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188)
- locked com.mysql.jdbc.util.ReadAheadInputStream@cb9a81c
com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:2494)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2949)
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2938)
com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3481)
com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2109)
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2642)
- locked java.lang.Object@70cbccca
com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2571)
com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:782)
- locked java.lang.Object@70cbccca
com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:625)
org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:260)
org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:260)


Whenever this happened we just restarted the Tomcat server and everything was fine again for some days or weeks. But today it struck us very hard so we finally took the time to hunt this down. It seems to be related to this bug report. Some comments suggested to use SQL_NO_CACHE with your queries.

A lot of people (including me) suggest to disable the MySQL query cache since it may cause severe problems. To disable the query cache at server startup, set the query_cache_size system variable to 0.

This is what we usually do, but one of our servers had query cache turned on. Disabling it solved this problem.

We managed to improve performance of our 10-node-test-cluster by almost 100% by adjusting the heartbeat intervals. Namenode and jobtracker use heartbeats to communicate with their workers (datanodes and tasktrackers).
We concentrate on jobtracker heartbeats. To reliably manage huge cluster the minimum interval is 3 seconds. Every 10 nodes the interval is increased by a second. If you have lots of fast running map- or reduce-tasks this implies a noticeable overhead.

What we did was to patch Hadoop and lower the minimum heartbeat interval to as low as 500ms and the increment to 10ms per node. This way we got our MapReduce-jobs run almost twice as fast. If you want to try it, you could take a look at our github branch (view commit). Please note that the git-branch contains our adopted version of Hadoop, so use it only for testing purposes.

There is a fix (HADOOP-5784) in the upcoming version 0.21 which allows you to lower the heartbeat increment per node.

To make a long story short (you probably read a lot of forum threads about this): Our Karmic Koala uses IPv6 for DNS queries and only if this fails it falls back to IPv4. A lot of home routers do not support IPv6 DNS queries. DOH!

Resolutions:

1. Firefox only: Disable IPv6 support by typing “about:config” into your location bar, then search for ipv6 and disable it by clicking on the line.

2. Disable IPv6 entirely: If you do not need IPv6-Support (I don’t) you could disable it completely and everything is up to speed again. How do I do this?

Otherwise you will get the following error:

<command>: cannot execute binary file


Took me 2 hours to find out.

In the mindset of a “traditional” database engineer one of the biggest concerns about Hadoop, or MapReduce in conjunction with a distributed file system in general, is the lack of indexes. Set aside that the debate “(R)DBMS vs MapReduce” is most of the time superfluous and sometimes almost leads to religious debates, the absence of a thing like an index is one the biggest hurdles you face when migrating data from a traditional DBMS.
Even though you will love the ability to view your data in any way you want without caring about its structure, at some point you feel that it is not right to always scan you 45TB of log files. (Even though it is soooo easy…).

Brute force is easy. Brute force is bad.

When we began migrating all those TBs of log-style data from our huge MySQL installations to Hadoop we did a lot of testing. We tested everything from Hadoop and MapReduce settings to different MapReduce abstractions like PIG, Cascading, Hive and others. There was this huge mass of data grinning at us and waited to be analysed in multiple ways, from “online” real-time access to “offline” decision making analysis. Due to our multiple views on the same data we came to this conclusion quite quickly: “Brute force is easy. Brute force is bad.” Yes, we can optimize our Hadoop installations and we can choose the really best query mechanism (actually we ended up writing our own), but it will not make things noticeably faster if you continue scanning all of our data all of the time.

Partitions are (sometimes) the better indexes

So, why are you using indexes (in the context of data retrieval)? I know why we did and do. It is all about primary key lookup and data clustering. Say you have the following table (MySQL):

CREATE TABLE order (
id INT NOT NULL,
product INT NOT NULL,
customer INT NOT NULL,
amount FLOAT NOT NULL,
orderDate DATE NOT NULL,

PRIMARY KEY(id),
INDEX idx_product_customer(product, customer),
INDEX idx_customer(customer)
)

Just a simple order log with an unique identifier (id) and a single associated product and customer. Since we want to view our data from different perspectives we added two additional indexes on product and customer. (In this example we need two indexes because MySQL can only use the leftmost prefix of an index.)
Dumping the whole table as a single CSV-file into your Hadoop cluster would mean that you always have to use what (R)DBMS call a “full table scan”. It would be pretty much the same like removing all indexes from your MySQL-table. Try to search for all products a customer ordered without the index idx_product_customer. (In fact Hadoop would perform this full table scan an order of magnitude faster.) But it would be ridiculous to remove all indexes from your table. But that is actually what you did when you exported the whole table into a flat-file!
What you should do, and what we did with great success, is to split up your flat-file CSV and arrange the data so that you can decide beforehand which part of the data needs to be accessed. So let’s split up the data and simulate all of the indexes (besides the primary key, more on that later on). A file-system-layout could look like this:

orders/
product_A/
customer_1.csv
customer_2.csv
product_B/
customer_1.csv
customer_3.csv

So when searching all orders customer_1 placed, we just use this file-pattern orders/*/customer_1.csv. Remember: HDFS and MapReduce’s inputs (like FileInputFormat) support globbing.

Now we actually simulated indexes by partitioning the data!

From here on you can go into more detail depending on your data structure. As an example you could add the date- and id-range to the file name like this:


orders/product_A/customer_1.2009-06-04.2009-06-05.1000.2000.csv
orders/product_A/customer_1.2009-06-06.2009-06-07.5000.7000.csv


This comes handy if you keep adding data to your cluster.
To make thinks even easier you could write your own InputFormat that encapsulates the building of the paths that match your query.

The small file problem

Since Hadoop has been designed to work on quite huge blocks of data it is not efficient when using a lot of small files. To prevent the creation of millions of very small files take a closer look at your data. Say your average customer places 50 orders. It would be a waste of resources to store multiple files for a single customer, each filling only a few KBs. A possible solution: group customers together.

orders/
product_A/
customer_1_to_1000.csv
customer_1001_to_2000.csv


You have to find the right balance between file-size and access pattern.

Some final words

To make it clear: Even though we have found a way to partition our data we have not gained the same flexibility as we have in any descent (R)DBMS (with enough of disk space, processing power and – most of all – RAM!). Querying for all orders made by a single customer may still take 0.01s in a (R)DBMS vs. 10s (or more) in Hadoop.

Never try to simply replace your (R)DBMS with Hadoop! Eventually you will end up writing a blog post saying that MapReduce and Hadoop are hopelessly worse than your favourite (R)DBMS. Hadoop is not a database!

Real-time lookups

You can still accomplish real-time lookup performance using Hadoop. One thing you could do is to take a look at HBase, a Google BigTable implementation.
Some times it is enough to use MapFiles which are simply a huge disk-based Hashtable.
In our application we implemented primary key and secondary keys directly on CSV files using MapFiles and distribute the lookup and local in-memory-caches over several machines. To speed things up even more we use a memcached cluster. (Eventually we will release all of this along with our MapReduce-abstraction as open-source once we feel it is mature and stable enough.)
One way or the other: Data redundancy will most likely become your best friend in these situations.

Regardless the techniques you are actually using you still have to think about your data in another way. You always have to when moving from a traditional (R)DBMS to any other kind of data storage and retrieval system!