[novapost] Multiple inheritance in buildout configuration files

Configuration reuse

#1 use case for "extend" is configuration reuse:

• write directives you always use in some "base.cfg" file.
• make your "buildout.cfg" extend "base.cfg".

If you do this, you are certainly going to copy-paste-adapt your favorite base configuration file. So you'll appreciate:

• generating buildout configuration from templates (with Jinja, PasteScript, diecutter...).
• or extending from URL.

Also consider the following advices:

• extend bases that have distinct roles, i.e. bases which don't overlap each other.
• use explicit overrides instead of += operator (which performs implicit overrides).

The motivations for the two advices above are detailed in buildout's inheritance mechanism below.

Components

#2 use case for "extend" is management of multiple components:

• you have some "frontend.cfg" that contains buildout directives to install some frontend component.
• you have some "backend.cfg" that contains buildout directives to install some backend component.
• you make your "buildout.cfg" extend both "frontend.cfg" and "backend.cfg" to install both components in a row.
• but you could use "frontend.cfg" and "backend.cfg" separately.

Well, you can extend several components, but, as of december 2012, I feel buildout is not designed for this purpose:

• buildout is made to deploy consistent environments. As an example, items in the environment share a set of dependencies, i.e. within the environment, there is one and only one version of each dependency.
• buildout is not made to deploy independant components in a same location. As an example, if a frontend depends on Django 1.4 and an backend requires version 1.3, then buildout can't deploy both within a single run. You will need to run two deployments.

Deeper explanations below...

Beware the += bug

For buildout 1.6.3, I reported a bug with multiple inheritance and +=.

As a summary, you may encounter issues if you extend multiple files and make intensive use of += operator in sections.

Several solutions:

• submit a patch for the zc.buildout.buildout._update_section() function.
• avoid += in bases, see other examples in this article.
• optionally use and promote a revised implementation of the += feature, which introduces backward incompatibilities.

Avoid += in bases

Yes, you can use += in bases. But don't you feel it is a bit strange? I mean, why would you use += in a file that don't inherit from other files?

The only reason I know is that the buildout inheritance mechanism makes bases extend each other. But I believe this is a bad reason.

So, as of version 1.6.3, my recommendation is to avoid += operator in bases, unless themselves have bases (multi-level inheritance).

Here are two designs I'd prefer, but they introduce backward incompatibilities...

Given:

• A extends (B and C)
• A sets "ac=A"
• B sets "bc=B"
• C sets "bc=C" and "ac=C"

I would find it more comprehensive if:

• bases were parsed separately, then inherited values were computed at the end, i.e. if C doesn't inherit from B. Result would be (ac=A, bc=C).
• or multiple bases were be concatenated, i.e. result is (ac=A, bc=BC).

Beware implicit overrides

Here is something that could happen if you use one of the popular version manager in buildout (buildout.dumppickedversions or buildout-versions).

buildout.cfg:

[buildout]
extends = frontend.cfg backend.cfg
extensions = buildout-versions
versions = versions
parts = frontend backend

frontend.cfg:

[frontend]
recipe = z3c.recipe.scripts
eggs = Django frontend

[versions]
Django == 1.3

backend.cfg:

[backend]
recipe = z3c.recipe.scripts
eggs = Django backend

[versions]
Django == 1.2

Here, buildout performs implicit overrides, and you get Django==1.2 as a result. It means that the "frontend" won't work as expected. You don't get warnings or errors about it, so it's quite hard to debug if you aren't aware that implicit overrides occur.

Explicit is better than implicit

When extending multiple bases (and because the "strange" inheritance implementation), you should use explicit overrides.

As an example, in some "a.cfg", you'd better write:

[buildout]
extends = b.cfg c.cfg
parts = b1 b2 c1 c2

... instead of using += in c[parts] and += again in a[parts].

Completely separated directories

Use directory options and -c argument to deploy each component in its own directory, i.e. alter buildout:directory option.

As an example:

bin/buildout -c frontend.cfg buildout:directory=frontend
bin/buildout -c backend.cfg buildout:directory=backend

Some shared resources

But you can also share some resources, like the eggs cache or the "bin/" folder.

The minimum thing you can do is to setup distinct buildout:installed option in configuration files, so that parts, eggs, ... are shared. It works well if configuration files don't overlap.

As an example, frontend.cfg:

[buildout]
extensions = buildout-versions
installed = .frontend-installed.cfg
parts = frontend-django

[frontend-django]
recipe = z3c.recipe.scripts
eggs =
    Django
interpreter = frontend

[versions]
Django = 1.3.4

And backend.cfg:

[buildout]
extensions = buildout-versions
installed = .backend-installed.cfg
parts = backend-django

[backend-django]
recipe = z3c.recipe.scripts
eggs = Django
interpreter = backend

[versions]
Django = 1.4.2

With this example, components don't overlap, so buildout -c frontend.cfg && buildout -c backend.cfg works well.