Automated performance degradation tests with JUnit4

2010/09/11 pero java, 0

There are several extension to JUnit that provide means to test performance like JUnitPerf or p-unit.

But it is hard to formulate the right assertions. What if the test runs on a beefier machine or in another environment?

Did performance degrade?

I just want to answer a simple question: Did performance degrade? (And if, when?)

Lately I came across junit-benchmarks. This framework allows you to visualize the performance history of select unit tests.
This is great, but I want my tests to fail if the performance degraded by some factor. A test run should fail, if it performs X% worse than the run before or Y% worse than the average of the last Z runs.

Extending junit-benchmarks

So I forked the junit-benchmarks, and created a project over at github.

Now you can assert that performance did not degrade:

public class MyTest {
 
    /** This is needed to enable junit-benchmark. */
    @Rule
    public MethodRule benchmarkRun = new BenchmarkRule(h2consumer);
 
    /**
     * This test fails, if it performs 20% worse than the last test run or 10% worse than the average of the last 10 runs.
     */
    @Test
    @BenchmarkOptions(perfDiffToLastRun = 0.2d, perfDiffToAverage = 0.1d, perfAverageOverRuns = 10)
    public void test() throws Exception {
        // Do something ...
    }
}

Note: Consumer H2 must be enabled. See junit-benchmark documentation.
As for all junit-benchmark settings, you can set defaults via JVM command line options.

But new features might cost performance

The approach above works if performance assumptions never change. But they do. So, performance tests will fail after adding a new feature that decreases performance on purpose. What should we do? Delete all the performance history? You could do that. Another way is to tell junit-benchmark that it should ignore all performance data up to a certain test run:

/**
 * This test fails, if it performs 20% worse than the last test run or 10% worse than the average of the last 10 runs.
 * But it ignores the 96 runs.
 */
@Test
@BenchmarkOptions(perfDiffToLastRun = 0.2d, perfDiffToAverage = 0.1d, perfAverageOverRuns = 10, perfIgnoreUpToRun = 96)
public void test() throws Exception {
    // Do something ...
}

Get the code

http://github.com/optivo/junit-benchmarks

Integrating MySQL and Hadoop – or – A different approach on using CSV files in MySQL Back online