Commoner

Commoner makes it easy to write scripts that flexibly and efficiently
transpile any dialect of JavaScript into a directory structure of
Node-compatible CommonJS module files.

This task is made possible by

1. a declarative syntax for defining how module source code should be
   found and processed,
2. the use of promises to manage an
   asynchronous build pipeline, and
3. never rebuilding modules that have already been built.

The output files can be required seamlessly by Node, or served by any
static file server, or bundled together using a tool such as
Browserify,
WrapUp, or
Stitch for delivery to a web
browser.

If you pass the --relativize option, Commoner also takes care to rewrite
all require calls to use relative module
identifiers,
so that the output files can be installed into any subdirectory of a
larger project, and external tools do not have to give special treatment
to top-level modules (or even know which modules are top-level and which
are nested).

Commoner was derived from an earlier, more opinionated tool called
Brigade that provided additional
support for packaging modules together into multiple non-overlapping
bundles. Commoner grew out of the realization that many tools already
exist for bundling CommonJS modules, but that fewer tools focus on getting
to that point.

Installation

From NPM:

npm install commoner

From GitHub:

cd path/to/node_modules
git clone git://github.com/reactjs/commoner.git
cd commoner
npm install .

Usage

Here's the output of bin/commonize --help:

Usage: commonize [options] <source directory> <output directory> [<module ID> [<module ID> ...]]

Options:

  -h, --help                               output usage information
  -V, --version                            output the version number
  -c, --config [file]                      JSON configuration file (no file means STDIN)
  -w, --watch                              Continually rebuild
  -x, --extension <js | coffee | ...>      File extension to assume when resolving module identifiers
  --relativize                             Rewrite all module identifiers to be relative
  --follow-requires                        Scan modules for required dependencies
  --cache-dir <directory>                  Alternate directory to use for disk cache
  --no-cache-dir                           Disable the disk cache
  --source-charset <utf8 | win1252 | ...>  Charset of source (default: utf8)
  --output-charset <utf8 | win1252 | ...>  Charset of output (default: utf8)

In a single sentence: the commonize command finds modules with the given
module identifiers in the source directory and places a processed copy of
each module into the output directory, along with processed copies of all
required modules.

If you do not provide any module identifiers, commonize will process all
files that it can find under the source directory that have the preferred
file extension (.js by default). If your source files have a file
extension other than .js, use the -x or --extension option to
specify it. For example, --extension coffee to find .coffee files.

Output

Commoner prints various status messages to STDERR, so that you can see
what it's doing, or figure out why it's not doing what you thought it
would do.

The only information it prints to STDOUT is a JSON array of module
identifiers, which includes the identifiers passed on the command line and
all their dependencies. This array contains no duplicates.

Internally, each module that Commoner generates has a hash computed from
the module's identifier, source code, and processing steps. Since this
hash can be computed before processing takes place, Commoner is able to
avoid processing a module if it has ever previously processed the same
module in the same way.

If you dig into the
code,
you'll find that Commoner maintains a cache directory (by default,
~/.commoner/module-cache/) containing files with names like
9ffc5c853aac07bc106da1dc1b2486903ca688bf.js. When Commoner is about to
process a module, it checks its hash against the file names in this
directory. If no match is found, processing procedes and the resulting
file is written to the cache directory with a new hash. If the appropriate
hash file is already present in the cache directory, however, Commoner
merely creates a hard link between the hash file and a file with a more
meaningful name in the output directory.

When you pass the --watch flag to bin/commonize, Commoner avoids
exiting after the first build and instead watches for changes to
previously read files, printing a new JSON array of module identifiers to
STDOUT each time rebuilding finishes. Thanks to the caching of processed
modules, the time taken to rebuild is roughly proportional to the number
of modified files.

Customization

The bin/commonize script is actually quite simple, and you can write
similar scripts yourself. Let's have a look:

 #!/usr/bin/env node

require("commoner").resolve(function(id) {
    var context = this;

    return context.getProvidedP().then(function(idToPath) {
        // If a module declares its own identifier using @providesModule
        // then that identifier will be a key in the idToPath object.
        if (idToPath.hasOwnProperty(id))
            return context.readFileP(idToPath[id]);
    });

}, function(id) {
    // Otherwise assume the identifier maps directly to a filesystem path.
    // The readModuleP method simply appends the preferred file extension
    // (usually .js) to the given module identifier and opens that file.
    return this.readModuleP(id);
});

The scriptable interface of the commoner module abstracts away many of
the annoyances of writing a command-line script. In particular, you don't
have to do any parsing of command-line arguments, and you don't have to
worry about installing any dependencies other than commoner in your
$NODE_PATH.

What you are responsible for, at a minimum, is telling Commoner how to
find the source of a module given a module identifier, and you do this by
passing callback functions to require("commoner").resolve. The script
above uses two strategies that will be tried in sequence: first, it calls
the helper function this.getProvidedP to retrieve an object mapping
identifiers to file paths (more about this below); and, if that doesn't
work, it falls back to interpreting the identifier as a path relative to
the source directory.

Now, you might not care about this.getProvidedP. It's really just a
proof of concept that Commoner can support modules that declare their own
identifiers using the // @providesModule <identifier> syntax, and I
included it by default because it doesn't make a difference unless you
decide to use @providesModule. If you don't like it, you could write an
even simpler script:

 #!/usr/bin/env node

require("commoner").resolve(function(id) {
    return this.readModuleP(id);
});

The point is, it's entirely up to you to define how module identifiers are
interpreted. In fact, the source you return doesn't even have to be valid
JavaScript. It could be CoffeeScript, or
LESS, or whatever language you prefer to write by
hand. Commoner doesn't care what your source code looks like, because
Commoner allows you to define arbitrary build steps to turn that source
code into plain old CommonJS.

Let's consider the example of using LESS to write dynamic CSS
modules. First, let's apply what we already know to give special meaning
to .less files:

 #!/usr/bin/env node

require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
});

function isLess(id) {
    return /\.less$/i.test(id);
}

All this really accomplishes is to avoid appending the .js file
extension to identifiers that already have the .less extension.

Now we need to make sure the contents of .less files somehow get
transformed into plain old CommonJS, and for that we need
require("commoner").process:

 require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    if (isLess(id))
        return compileLessToJs(source);
    return source;
});

How should compileLessToJs be implemented? At a high level, I propose
that we generate a CommonJS module that will append a new <style> tag to
the <head> the first time the module is required. This suggests to me
that we need to take the CSS generated by LESS and somehow embed it as a
string in a CommonJS module with a small amount of boilerplate JS.

Here's a first attempt:

 function compileLessToJs(less) {
    var css = require("less").render(less);
    return 'require("css").add(' + JSON.stringify(css) + ");";
}

Implementing a css module with an appropriate add method is an
exercise that I will leave to the reader (hint: you may find this
StackOverflow answer useful).

This almost works, but there's one problem: require("less").render does
not actually return a string! For better or worse, it passes the compiled
CSS to a callback function, which would make our task extremely painful
if Commoner were not deeply committed to supporting asynchronous
processing.

Commoner uses promises for asynchronous control flow, so we need to return
a promise if we can't return a string immediately. The easiest way to make
a promise is to call this.makePromise in the following style:

 #!/usr/bin/env node

require("commoner").resolve(function(id) {
    if (isLess(id))
        return this.readFileP(id);
}, function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    if (isLess(id)) {
        return this.makePromise(function(nodeStyleCallback) {
            compileLessToJs(source, nodeStyleCallback);
        });
    }
    return source;
});

function compileLessToJs(less, callback) {
    require("less").render(less, function(err, css) {
        callback(err, 'require("css").add(' + JSON.stringify(css) + ");")
    });
}

And we're done! This example was admittedly pretty involved, but if you
followed it to the end you now have all the knowledge you need to write
source files like sidebar.less and require them from other modules by
invoking require("sidebar.less"). (And, by the way, embedding dynamic
CSS modules in your JavaScript turns out to be an excellent idea.)

Generating multiple files from one source module

Commoner is not limited to generating just one output file from each
source module. For example, if you want to follow best practices for
producing source maps, you probably want to create a .map.json file
corresponding to every .js file that you compile.

Recall that normally your .process callback returns a string (or a
promise for a string) whose contents will be written as a .js file in
the output directory. To write more than one file, just return an object
whose keys are the file extensions of the files you want to write, and
whose values are either strings or promises for strings representing the
desired contents of those files.

Here's an example of generating two different files for every source
module, one called <id>.map.json and the other called <id>.js:

 require("commoner").resolve(function(id) {
    return this.readModuleP(id);
}).process(function(id, source) {
    var result = compile(source);
    return {
        ".map.json": JSON.stringify(result.sourceMap),
        ".js": [
            result.code,
            "//# sourceMappingURL=" + id + ".map.json"
        ].join("\n")
    };
});

Note that

 return {
    ".js": source
};

would be equivalent to

 return source;

so you only have to return an object when you want to generate multiple
files. However, the .js key is mandatory when returning an object.

For your convenience, if you have a sequence of multiple processing
functions, the values of the object returned from each step will be
resolved before the object is passed along to the next processing
function, so you can be sure all the values are strings (instead of
promises) at the beginning of the next processing function.

Configuration

Of course, not all customization requires modifying code. Most of the
time, in fact, configuration has more to do with providing different
dynamic values to the same code.

For that kind of configuration, you don't need to modify your Commoner
script at all, because Commoner scripts accept a flag called --config
that can either specify a JSON file or (if --config is given without a
file name) read a string of JSON from STDIN.

Examples:

bin/commonize source/ output/ main --config release.json
bin/commonize source/ output/ main --config debug.json
echo '{"debug":false}' | bin/commonize source/ output/ main --config
echo '{"debug":true}' | bin/commonize source/ output/ main --config /dev/stdin

This configuration object is exposed to the .resolve and .process
callbacks as this.config. So, for example, if you wanted to implement
minification as a processing step, you might do it like this:

 require("commoner").resolve(function(id) {
    return this.readModule(id);
}).process(function(id, source) {
    if (this.config.debug)
        return source;
    return minify(source);
});

Perhaps the coolest thing about the configuration object is that Commoner
generates a recursive hash of all its properties and their values which is
then incorporated into every module hash. This means that two modules with
the same identifier and identical source code and processing steps will
have distinct hashes if built using different configuration objects.

Custom Options

You can define custom options for your script by using the option function.

 require("commoner").resolve(function(id) {
    return this.readModule(id);
}).option(
    '--custom-option',
    'This is a custom option.'
).process(function(id, source) {
    if (this.options.customOption) {
        source = doCustomThing(source);
    }
    return source;
});

For more information of the options object available inside the process function see Commander.