karma-benchmark

A Karma plugin to run
Benchmark.js over multiple browsers with CI
compatible output.

☁️ Installation

npm install --save-dev benchmark karma-benchmark

🕵🏾‍♀️ Reporters

• karma-benchmark-json-reporter
  by @etpinard
• karma-benchmark-plotly-reporter
  by @etpinard
• karma-benchmarkjs-reporter
  by @FormidableLabs
• karma-benchmark-reporter
  by @lazd

⚖️ Configuration

In karma.conf.js, add 'benchmark' to the list of frameworks:

 module.exports = (config) => {
  config.set({
    autoWatch: false,
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
    singleRun: true
  });
};

Terminal Reporting

Now let's add
karma-benchmarkjs-reporter
by @FormidableLabs to report results to
the Terminal:

npm install --save-dev karma-benchmarkjs-reporter

In karma.conf.js, add 'benchmark' to the list of reporters:

 module.exports = (config) => {
  config.set({
    autoWatch: false,
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
+   reporters: ['benchmark'],
    singleRun: true
  });
};

JUnit Reporting

To feed our data into Continuous Integration, we can use the
karma-junit-reporter.

npm install --save-dev karma-junit-reporter

In karma.conf.js, add junit to the list of reporters and configure the
reporter accordingly:

 module.exports = (config) => {
  config.set({
    autoWatch: false,
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
+   junitReporter: {
+     outputDir: 'reports',
+     outputFile: 'benchmark.xml'
+   },
-   reporters: ['benchmark'],
+   reporters: ['benchmark', 'junit'],
    singleRun: true
  });
};

Data Visualisation Reporting

With a free plot.ly account, we can generate visualisations
using the
karma-benchmark-plotly-reporter
by @etpinard.

npm install --save-dev karma-benchmark-plotly-reporter

In karma.conf.js, add benchmark-plotly to the list of reporters and
configure the reporter accordingly:

 module.exports = (config) => {
  config.set({
    autoWatch: false,
+   benchmarkPlotlyReporter: {
+     username: '<your username>',
+     apiKey: '<your api key>',
+     cloudFilename: 'plotly-example',
+     imageFilename: 'plotly-example.png'
+   },
    browsers: ['Chrome'],
    concurrency: 1,
    files: ['bench/**/*.bench.js'],
    frameworks: ['benchmark'],
    junitReporter: {
      outputDir: 'reports',
      outputFile: 'benchmark.xml'
    },
-   reporters: ['benchmark', 'junit'],
+   reporters: ['benchmark', 'benchmark-plotly', 'junit'],
    singleRun: true
  });
};

👩🏻‍🔬 Writing Benchmarks

Benchmarks can be written by using the original
Benchmark.js API, but a wrapper API is also provided
by karma-benchmark in the form of the suite and benchmark globals. The
karma-benchmark API aims to make the process of writing Benchmarks feel familiar
to users of Jasmine or Jest.

In this example, a suite is defined that pits _.each against the native
Array.forEach method:

 suite('Array iteration', () => {
  benchmark('_.each', () => {
    _.each([1, 2, 3], (el) => {
      return el;
    });
  });

  benchmark('native forEach', () => {
    [1, 2, 3].forEach((el) => {
      return el;
    });
  });
});

Suite options

Suite options are the same as in Benchmark.js with one exception: onStart and
onComplete can be set at the suite level.

See the
Benchmark.js Suite constructor API docs for
a full list of options.

 suite(
  'Array iteration',
  () => {
    benchmark('_.each', () => {
      _.each(this.list, (number) => {
        return number;
      });
    });

    benchmark('native forEach', () => {
      this.list.forEach((number) => {
        return number;
      });
    });
  },
  {
    onCycle(event) {
      var suite = this;
      var benchmark = event.target;
      console.log('Cycle completed for ' + suite.name + ': ' + benchmark.name);
    },
    onStart() {
      this.list = [5, 4, 3];
    },
    onComplete() {
      this.list = null;
    }
  }
);

Benchmark options

Benchmark options are the same as in Benchmark.js. If setup and teardown are
passed to benchmark(), they will override setup and teardown from the
suite. Pass null or undefined to remove them.

See the
Benchmark.js Benchmark constructor API docs
for a full list of options.

 suite('Iteration', () => {
  benchmark(
    '_.each with array',
    () => {
      _.each(this.list, (number) => {
        return number;
      });
    },
    {
      setup() {
        this.list = ['a', 'b', 'c'];
      },
      teardown() {
        delete this.list;
      }
    }
  );

  benchmark(
    '_.each with object',
    () => {
      _.each(this.list, (number) => {
        return number;
      });
    },
    {
      setup() {
        this.list = { 0: 'a', 1: 'b', 2: 'c' };
      },
      teardown() {
        delete this.list;
      }
    }
  );
});

Running only a specific benchmark or suite

To run only a specific benchmark, use benchmark.only() or bbenchmark()
instead of benchmark():

 benchmark.only(() => {
  // Only this benchmark will run
  // bbenchmark() does the same thing
});

benchmark(() => {
  // This benchmark won't run
});

The same applies to suites with suite.only() and ssuite().

Skipping benchmarks & suites

To skip a benchmark, use benchmark.skip() or xbenchmark() instead of
benchmark():

 benchmark.skip(() => {
  // This benchmark won't run
  // xbenchmark() does the same thing
});

benchmark(() => {
  // This and all other benchmarks will run
});

The same applies to suites with suite.skip() and xsuite().

🙋🏿‍♂️ Get Help

There are few ways to get help:

1. For bug reports and feature requests, open issues :bug:
2. For direct and quick help, you can use Gitter :rocket: