Monday, July 18, 2016

Generating Dependencies Automatically with GNU Make & Browserify

Abstract

For any one to be required to use more force than is absolutely necessary for the job in hand is waste.
— Henry Ford

In the previous post about the example of a build system for JavaScript SPAs, we didn’t cover the topic of auto-discovering dependencies. While not being the most complex one, it oftentimes leads to a rather frustrating expirience for the novice user.

In this post we’ll examine several ways of dependency management to aid Make to properly construct its dependency trees.

We’ll use a simple “app” consisting of 3 .js files:

foobar
├── bar.js
├── foo.js
└── main.js

where we’ll compile them from ES2015 to ES5 w/ Babel & will combine them in 1 bundle w/ Browserify. The dependency tree for main.js looks very simple:

i.e., foo.js & bar.js are commonjs modules, main.js requires bar that in turn requres foo.

The makefile that we’ll write will do 2 things:

  1. compile all .js files into a separate tree directory;
  2. create a bundle from the files in the separate tree directory.

The dependency problem arises when we modify, say, foo.js. Our build system should automatically recognize that the bundle from the step 2 became outdated & needs to be recreated.

The compilation

As usual we want to support a single source three with multiple builds (development & production). Thus it’s inconvinient to put the results of the compilation in the source directory. The simplest way of achieving this is to run Make from the output directory that != source directory. For example:

the-plan
├── foobar/
│   ├── bar.js
│   ├── foo.js
│   ├── main.js
│   └── main.mk
└── _out/
    └── development/
        ├── .ccache/
        │   ├── bar.js
        │   ├── foo.js
        │   └── main.js
        └── main.js

where foobar is out source directory, _out is the output directory where we run Make, _out/development/main.js is the bundle.

Let’s start with compiling .js files first. For simplicity we’ll assume that all the npm packages we need are installed in the global mode.

# npm -g i babel-cli babel-preset-es2015 browserify
$ cat ../foobar/main.mk
.DELETE_ON_ERROR:

src := $(dir $(lastword $(MAKEFILE_LIST)))
NODE_ENV ?= development
out := $(NODE_ENV)

.PHONY: compile
compile:

js.src := $(shell find $(src) -name '*.js' -type f)
js.dest := $(patsubst $(src)%.js, $(out)/.ccache/%.js, $(js.src))

ifeq ($(NODE_ENV), development)
BABEL_OPT := -s inline
endif
_BABEL_OPT := --preset $(shell npm -g root)/babel-preset-es2015 $(BABEL_OPT)

$(js.dest): $(out)/.ccache/%.js: $(src)/%.js
»   @mkdir -p $(dir $@)
»   babel $(_BABEL_OPT) $< -o $@

compile: $(js.dest)

If we run it in _out directory:

$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//foo.js -o development/.ccache/foo.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//main.js -o development/.ccache/main.js

$ make -f ../foobar/main.mk
make: Nothing to be done for 'compile'.

To recap what we wrote here:

  • The empty .DELETE_ON_ERROR: target tells Make to remove the produced target, for example, development/.ccache/foo.js in case of the compilation failure. You should always include this line into your makefiles, otherwise, in our case, it’s possible to end up with invalid development/.ccache/foo.js if Babel terminates unexpectedly due to a bug, user signal, etc. Recall that Make thinks about the success in terms of the exit status of a shell command.

  • We collected the names of our source files in js.src; js.dest contains the transformed paths so that

      ../foobar//foo.js
    

    becomes

      development/.ccache/foo.js
    
  • Notice how we wrote the header of the patter rule:

      $(js.dest): $(out)/.ccache/%.js: $(src)/%.js
    

    by prepending it with $(js.dest) we limited the scope of it.

  • The default output build is ‘development’. We make sure that in the development mode we include source maps for the output .js files. I do not discuss here the command line options for Babel (& the kludge to force Babel pick up a globaly installed preset), for they are irrelevant to the topic.

Bundling

As we transpile the .js files into a mundane ES5, the bundle should be created from the results of the compilation, not from the original files.

$ awk '/bundle/,0' ../foobar/main.mk
bundles.src := $(filter %/main.js, $(js.dest))
bundles.dest := $(patsubst $(out)/.ccache/%.js, $(out)/%.js, $(bundles.src))

ifeq ($(NODE_ENV), development)
BROWSERIFY_OPT := -d
endif
$(bundles.dest): $(out)/%.js: $(out)/.ccache/%.js
»   @mkdir -p $(dir $@)
»   browserify $(BROWSERIFY_OPT) $< -o $@

compile: $(bundles.dest)

Again, if we run it in the output directory, the expected development/main.js appears:

$ make -f ../foobar/main.mk
browserify -d development/.ccache/main.js -o development/main.js

but the makefile falls short of detecting whether the bundle needs to be updated:

$ touch ../foobar/foo.js

$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//foo.js -o development/.ccache/foo.js

Despite of the fact that foo.js was indeed recompiled, our bundle remained intact because we didn’t specify any additional dependency relationships except a forlorn $(out)/main.js$(out)/.cache/main.js in the pattern rule.

There are several ways to ameliorate this. We’ll start with

Method 1: The Manual

The addition of a single line to main.mk:

$(out)/main.js: $(js.src)

seems to be able to solve the problem. If you run Make again it sees that one of the prerequisites (foo.js) is newer than the bundle target.

Pros Cons
Fast
Easy to maintain in small projects Unmanageable in projects w/ a lot of small modules
No dependencies on external tools

The biggest impediment here is that the method doesn’t scale. Essentially you resort yourself to doubling the amount of work of the dependency management: the 1st time you do it when you write your code, the 2nd time–during the reconstruction of the same dependency graph in the Makefile. This is waste.

It’s also prone to errors. For example, if you have several bundles:

example02/many-foobars
├── one/
│   └── main.js
├── two/
│   └── main.js
├── bar.js
├── foo.js
└── main.mk

then adding the same naïve lines:

$(out)/one/main.js: $(js.src)
$(out)/two/main.js: $(js.src)

to main.mk will lead you to the recompilation of 2 bundles even if you make a change only to 1 of them:

$ make -f ../many-foobars/main.mk
[...]

$ make -f ../many-foobars/main.mk -W ../many-foobars/one/main.js -tn
touch development/one/main.js
touch development/two/main.js

(-W options means “pretend that the target has been modified”.)

Method 2: Automatic make depend

Instead of specifying prerequisites manually we can use an external tool that returns the dependency list, in the Make-compatible format, for each file. One of such tools is make-commonjs-depend.

# npm -g i make-commonjs-depend
[...]
$ make-commonjs-depend development/.ccache/main.js
development/.ccache/main.js: \
  development/.ccache/bar.js
development/.ccache/bar.js: \
  development/.ccache/foo.js
development/.ccache/foo.js:
Pros Cons
Could be slow
Easy to maintain
Requires an external tool
May rebuilt already up to date targets

We can write a phony target “depend” & run make depend every time after we add/remove/rename any .js file & include the generated file into our Makefile.

We can also write a special target $(out)/.ccache/depend.mk, the recipe of which creates its target by running make-commonjs-depend command. In this case, if we include $(out)/.ccache/depend.mk & Make sees that the target is out of date, it remakes $(out)/.ccache/depend.mk & then immidiately restarts itself.

$ awk '/depend/,0' ../foobar/main.mk
$(out)/.ccache/depend.mk: $(js.dest)
»   make-commonjs-depend $^ > $@
»   @echo ========== RESTARTING MAKE ==========

include $(out)/.ccache/depend.mk

Here depend.mk file has all compiled .js files as prerequisites thus when any of them needs to be updated Make recompiles such .js files & reruns make-commonjs-depend.

$ rm -rf development
$ make -f ../foobar/main.mk
../foobar/main.makedepend.mk:41: development/.ccache/depend.mk: No such file or directory
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//foo.js -o development/.ccache/foo.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//main.js -o development/.ccache/main.js
make-commonjs-depend development/.ccache/bar.js development/.ccache/foo.js development/.ccache/main.js > development/.ccache/depend.mk
========== RESTARTING MAKE ==========
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//main.js -o development/.ccache/main.js
make-commonjs-depend development/.ccache/bar.js development/.ccache/foo.js development/.ccache/main.js > development/.ccache/depend.mk
========== RESTARTING MAKE ==========
browserify -d development/.ccache/main.js -o development/main.js

Although it works fine the unnecessary rebuilds could be a pain in big projects. For example, Make doesn’t understand that transpiling main.js in not needed in case of bar.js update, but because make-commonjs-depend gives Make a preconfigured graph which states that main.jsbar.js, it dutifully rebuilds main.js.

$ touch ../foobar/bar.js

$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//main.js -o development/.ccache/main.js
make-commonjs-depend development/.ccache/bar.js development/.ccache/foo.js development/.ccache/main.js > development/.ccache/depend.mk
========== RESTARTING MAKE ==========
browserify -d development/.ccache/main.js -o development/main.js

On the other hand, if you don’t mind such remakes you may think it’s a small price to pay for having a fully automated dependency graph available after adding only 5 lines of code to the makefile.

Method 3: Variation of Tromey’s Way

The invention of another, more clever way of auto-discovering dependencies is generally attributed to Tom Tromey, who invented it while working on automake project in the second half of the 90s.

Instead of having name.mk targets that Make uses to restart itself, every file that needs dependency traction writes its dependency tree after the compilation step, as a side effect of it.

Pros Cons
Fast
Easy to maintain
No dependencies on external tools (it uses Browserify)

For example,

$(out)/%.js: $(out)/.ccache/%.js
»   mkdir -p $(dir $@)
»   browserify $< -o $@
»   a-magic-command-to-generate-a-dependency-list > $(basename $<).d

The key here is to generate the prerequisite lists only for the bundles, not for every .js file & keep those prerequisite lists in .d files alongside the main.js file in $(out)/.ccache directory. (.d extension means nothing special, it’s just a name convention.)

During the 1st run when there is no .d files, Make knows nothing about them so it compiles .js files, then compiles bundles. The rule that creates a bundle also produces a corresponding .d file with the list of all the dependencies the bundle depends on.

At this stage we’re as at the point as if we didn’t have any dependencies for the bundles at all, but we can instruct Make to read those .d files at startup later on. In the next run, Make scans .d files, looks into the provided dependency lists & sees if any of the bundles needs to be updated. After each update the corresponding .d file updates as well.

The beauty of the method is that it doesn’t care if we reshuffle our code into a completely different set of .js files as long as we don’t remove any files in $(out)/.ccache directory & if we do remove that directory completely–it still doesn’t matter, for it’ll be the same as doing the clean build from the scratch.

$ awk '/bundle/,0' ../foobar/main.mk
bundles.src := $(filter %/main.js, $(js.dest))
bundles.dest := $(patsubst $(out)/.ccache/%.js, $(out)/%.js, $(bundles.src))

define make-depend
@echo Generating $(basename $<).d
@printf '%s: ' $@ > $(basename $<).d
@browserify --no-bundle-external --list $< \
»   | sed s%.\*$<%% | sed s%$(CURDIR)/%% | tr '\n' ' ' \
»   >> $(basename $<).d
endef

ifeq ($(NODE_ENV), development)
BROWSERIFY_OPT := -d
endif
$(bundles.dest): $(out)/%.js: $(out)/.ccache/%.js
»   @mkdir -p $(dir $@)
»   browserify $(BROWSERIFY_OPT) $< -o $@
»   $(make-depend)

compile: $(bundles.dest)

-include $(bundles.src:.js=.d)

Before explaining the new code, let’s see it in action. We clean up $(out) & run make:

$ rm -rf development
$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//foo.js -o development/.ccache/foo.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//main.js -o development/.ccache/main.js
browserify -d development/.ccache/main.js -o development/main.js
Generating development/.ccache/main.d

The generated file development/.ccache/main.d should contain a new rule (a oneliner, w/o a recipe):

$ cat development/.ccache/main.d
development/main.js: development/.ccache/foo.js development/.ccache/bar.js  

Now if we update bar.js:

$ touch ../foobar/bar.js
$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
browserify -d development/.ccache/main.js -o development/main.js
Generating development/.ccache/main.d

Volia! Make accurately recompiles only those files that needs to be recompiled: bar.js & the bundle.

Looking into the body of the pattern rule we see a line that contains $(make-depend) string. It looks like we’re injecting a value of the variable make-depend into the recipe. This trick is called a canned recipe. make-depend is a multi-line REV (recursively expanded variable) which means that Make expands it value every time it has a need to. You may think of make-depend variable as a macro or a function with a dynamic scope.

The purpose of the make-depend REV is to write a .d file that should contain a valid Make syntax.

If we run Browserify by hand on a compiled main.js file with --list command line option, Browserify prints a newline-separated list of main.js dependencies:

$ browserify --no-bundle-external --list development/.ccache/main.js
/home/alex/lib/writing/articles/data/gmake-autodeps/_out.blogger/s06/example01/_out/development/.ccache/foo.js
/home/alex/lib/writing/articles/data/gmake-autodeps/_out.blogger/s06/example01/_out/development/.ccache/bar.js
/home/alex/lib/writing/articles/data/gmake-autodeps/_out.blogger/s06/example01/_out/development/.ccache/main.js

This is obviously not a valid Make syntax. We ought to:

  1. remove main.js from the list, otherwise we get a circular dependency problem;

  2. transform absolute paths to relative ones, for our pattern rules expect the latter.

This is what make-depend macro does, not counting a pattern rule header generation.

Of course nothing prevents you from writing a small script that runs Browserify by internally & formats the output accordingly. You can even take make-commonjs-depend & write a custom printer for it if you’re feeling brave.

Finally, as we’re generating .d files we should give Make a chance to read them in the next run. This is what

-include $(bundles.src:.js=.d)

line does. :.js=.d suffix means “in every file name substitute .js extension with .d”, e.g. the expanded result looks like

-include development/.ccache/main.d

A minus sign prevents Make from printing a warning if development/.ccache/main.d is not found.

What if we rename foo.js into fool.js (& do the corresponding changes in the code)? In a poorly written build system it could break the build & could require users manually remove .d files.

$ mv ../foobar/foo.js ../foobar/fool.js
$ sed -i "s,'./foo','./fool'," ../foobar/bar.js
$ tree ../foobar/ --noreport
../foobar/
├── bar.js
├── fool.js
├── main.js
└── main.mk

$ make -f ../foobar/main.mk
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//bar.js -o development/.ccache/bar.js
babel --preset /opt/lib/node_modules/babel-preset-es2015 -s inline ../foobar//fool.js -o development/.ccache/fool.js
browserify -d development/.ccache/main.js -o development/main.js
Generating development/.ccache/main.d

There was no errors of any kind because foo.js leftover happily resides in $(out)/.ccache directory.

PS. Here is an alternate version of this post that can be more readable on your phone.

No comments:

Post a Comment