[unlish] pip install -e / setup.py develop on a cube

The current layout of CubicWeb cubes is not setuptools friendly, and does not permit to use the "develop" mode. The CWEP 004, currently under discussion, aims to solve that problem, and once adopted make life easier for the developers.

Meanwhile, some people (like us at Unlish) would like to use the develop mode anyway, so that we can work on a cube without cloning the whole tree of CubicWeb requirements.

After a few experiments, it turns out we can tweak the virtualenv after pip install -e is run on the cube, and have the benefit of it.

The solution I describe below allows to work on both the cube and the unittests, importing from "cubes.thecube" and "cubes.anothercube" in a virtualenv. It can be applied to several cubes in the same virtualenv. It does *not* apply to the other cubicweb packages.

Problem 1 - the cubes path

Once pip install -e has been run the cube on which it was launched is not usable because it is not present in the cube path.

We do not want to add the parent directory of our cube to CW_CUBES_PATH because:

• When importing from cubes.acube, and because "cubes" is not a proper namespace, we have issues
• The spirit of virtualenv is that where the project was checked out does not matter -> we do not want to put restriction on where it can be put in the filesystem.

Solution

From within the virtualenv :

ln -s $(pwd) $VIRTUAL_ENV/share/cubicweb/cubes/thecube

Problem 2 - the python path

1. pip install -e add our directory to the interpreter path. It is a problem because it pollutes the root namespace, it not masks some modules when the names match (for example in unlish we have a 'markdown.py' file).
2. for some reason I did not identified, I could not import from cubes.xxx because the virtualenv cubes path is not added to the syspath like I expected it to (I may have done something wrong on this matter).

Solution

1. Remove our directory from the virtualenv path

   PTH=$VIRTUAL_ENV/lib/python2.7/site-packages/easy-install.pth
   grep -v $(pwd) $PTH > .pth.tmp
   mv .pth.tmp $PTH
   

2. Add the virtualenv cubes path to the python path

   export PYTHONPATH=$VIRTUAL_ENV/share/cubicweb
   

   This last command has to be done each time the virtualenv is activated, it is recommended to add it to the 'activate' script, OR postactivate script if you are using virtualenvwrapper (which I recommend too). 

Put it all together

The following script does all that for you, and has to be run after each run of pip install -e.
Note that it will overwrite your postactivate script, set CW_MODE=user and define a 'ctl' alias, so you may need to modify it before using it.

#!/bin/bash

MYPATH=$(pwd)
MODNAME=$(python -c "import __pkginfo__; print __pkginfo__.modname")

if [ -z "$VIRTUAL_ENV" ]; then
    exit 1
fi

if [ -n "$VIRTUAL_ENV/bin/postactivate" ]; then
    cat >>$VIRTUAL_ENV/bin/activate <<EOF
. $VIRTUAL_ENV/bin/postactivate
EOF
fi

cat >$VIRTUAL_ENV/bin/postactivate <<EOF
export CW_MODE=user
export PYTHONPATH=$VIRTUAL_ENV/share/cubicweb

alias ctl=cubicweb-ctl
EOF

rm -rf $VIRTUAL_ENV/share/cubicweb/cubes/$MODNAME
ln -s $MYPATH $VIRTUAL_ENV/share/cubicweb/cubes/$MODNAME

PTH=$VIRTUAL_ENV/lib/python2.7/site-packages/easy-install.pth
grep -v $MYPATH $PTH > .pth.tmp
mv .pth.tmp $PTH

echo "Tweaked virtualenv $VIRTUAL_ENV"
echo "Don't forget to re-activate it:"
echo ""
echo "   workon $(basename $VIRTUAL_ENV)"

To conclude, this is only a hackish workaround to wait for the real solution: reforming the cubes in cubicweb.

(this post was initially a mail on the cubicweb mailing-list)