results-interpreter

A Node.js writable
stream for
interpreting streaming test results in accordance with a whitelist file.

Installation

npm install --save-dev results-interpreter

Usage

 var TestInterpreter = require('results-interpreter');

// See the following section, "Input: results stream"
var testResultStream = runMyTests();
// See the following section, "Input: whitelist file"
var interpreter = new TestInterpreter('path/to/a-whitelist-file.txt', {
  // (optional) See the following section, "Output: whitelist file"
  outputFile: 'path/to/another-whitelist-file.txt'
});

testResultStream.pipe(interpreter)
  .on('error', function(error) {
    console.error(error);
    process.exitCode = 1;
  })
  .on('finish', function() {
    // See the following section: "Output: `summary` object"
    console.log(JSON.stringify(this.summary));
    process.exitCode = this.summary.passed ? 0 : 1;
  });

Input: test results stream

Users of this library should provide a Readable object
stream which emits
data describing test results. Each object must define the following properties:

• id - type: string; unique identifier describing the test; must be stable
  across test executions
• expected - type: string; the outcome that was expected; either "pass" or
  "fail"
• actual - type: string; the outcome that was observed; either "pass" or
  "fail"

Input: whitelist file

The whitelist file is read as a UTF-8-formatted text file. Test IDs must be
separated by a newline character. Any text following the "number sign"
character (#) will be interpreted as a comment and ignored.

Output: summary object

Once results interpretation is complete (and after a new whitelist file has
been created--see below), the stream will define a summary property with an
object value. This object contains information about the test results.

• The following properties contain arrays of testIDs which satisfy
  expectations:
  • the actual and expected values match, and there is no corresponding
    entry in the whitelist file
    • summary.allowed.success
    • summary.allowed.failure
  • the actual and expected values do not match, but there is a
    corresponding entry in the whitelist file
    • summary.allowed.falsePositive
    • summary.allowed.falseNegative
• The following properties contain arrays of test IDs which violate
  expectations. If any of these arrays are non-empty, the results are
  considered "failing", and the summary.passed attribute referenced below
  will be false
  • the actual and expected values match, but there is a corresponding
    entry in the whitelist file
    • `summary.disallowed.success
    • `summary.disallowed.failure
  • the actual and expected values do not match, and there is no
    corresponding entry in the whitelist file
    • summary.disallowed.falsePositive
    • summary.disallowed.falseNegative
  • the test ID was included in the whitelist file, but the provided stream did
    not emit an object describing a corresponding result
    • summary.unrecognized
• summary.passed - a boolean attribute describing whether the results
  completely meet expectations

Output: whitelist file

The stream may optionally output a new version of the whitelist file based on
the provided whitelist file. The contents of the output file will be based on
the input whitelist, modified to satisfy the behavior of the test run.
Specifically:

• Lines referencing tests which no longer violate expectations will be removed
  (including trailing comments, if present)
• The identifiers for tests which violated expectations will be appended
• Lines referencing tests which had no corresponding object in the provided
  results stream will be removed (including trailing comments, if present)

All other lines (including comment lines) will be persisted in the output file.

License

Copyright 2017 Mike Pennisi under the GNU General Public License
v3.0