Python’s module search path

Upgrading software on an Ubuntu 12.04 machine, I hit a snag. A daemon wouldn’t start. The error log showed the following:

from util.pystone import pystones
ImportError: No module named util.pystone

Reading a little, pystone is used in benchmarking and comes with Python 2.7. Yep, it’s there.

$ ls /usr/lib/python2.7/test
__init__.py __init__.pyc pystone.py pystone.pyc regrtest.py regrtest.pyc test_support.py test_support.pyc

Strange. Firing up python and importing this from the python prompt failed.

I tried only specifying this path to see if it would import:

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path=['/usr/lib/python2.7']
>>> print sys.path
['/usr/lib/python2.7']
>>> import test.pystone
>>>

At this point I suspected something in Python’s module search path was fucking me.

A little more messing around I found a good one.

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.sort()
>>> import test.pystone
>>>

and

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.reverse()
>>> import test.pystone
>>>

Okay something is matching in the search path before it get’s to /usr/lib/python2.7 which is about half way through the sys.path. So, I did what anybody in my shoes would do and removed them one by one.

>>> sys.path.reverse()
>>> print sys.path.pop()
/usr/lib/pymodules/python2.7
>>> sys.path.reverse()
>>> import test.pystone Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

Until I got to an empty list.

>>> print sys.path.pop()
/usr/lib/python2.7
>>> print sys.path
[]
>>> sys.path.append('/usr/lib/python2.7')
>>> print sys.path
['/usr/lib/python2.7']
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

Immediately, I assumed I was losing my mind. Of course, I should check that’s the case.

$ python
Python 2.7.3 (default, Feb 27 2014, 19:58:35)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone
>>> sys.path=['/usr/lib/python2.7']
>>> import test.pystone
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named pystone

Around this point I thought I was crazy and/or had a flaw in my testing process. Then, it dawned on me.

If I call test.pystone before zeroing out the sys.path it WON’T find it ever.

Well, shit.

I wrote a BASH script to iterate over the list removing one at a time and testing the import:

#!/bin/bash
FUCK=" /usr/local/lib/python2.7/dist-packages/pyserial-2.6-py2.7.egg /usr/local/lib/python2.7/dist-packages/simplejson-3.3.0-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages/gdata-2.0.18-py2.7.egg /usr/local/lib/python2.7/dist-packages/todoist-0.0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/python_twitter-1.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/requests_oauthlib-0.4.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauthlib-0.6.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/gmusicapi-3.1.1_dev-py2.7.egg /usr/local/lib/python2.7/dist-packages/appdirs-1.3.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/mock-1.0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauth2client-1.2-py2.7.egg /usr/local/lib/python2.7/dist-packages/proboscis-1.2.6.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/validictory-0.9.3-py2.7.egg /usr/local/lib/python2.7/dist-packages/httplib2-0.9-py2.7.egg /usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg /usr/local/lib/python2.7/dist-packages/walter-0.1-py2.7.egg /usr/local/lib/python2.7/dist-packages/geeknote-0.2a-py2.7.egg /usr/local/lib/python2.7/dist-packages/thrift-0.9.1-py2.7-linux-x86_64.egg /usr/local/lib/python2.7/dist-packages/markdown2-2.3.0-py2.7.egg /usr/local/lib/python2.7/dist-packages/html2text-2014.9.25-py2.7.egg /usr/local/lib/python2.7/dist-packages/oauth2-1.5.211-py2.7.egg /usr/lib/python2.7 /usr/lib/python2.7/plat-linux2 /usr/lib/python2.7/lib-tk /usr/lib/python2.7/lib-old /usr/lib/python2.7/lib-dynload /usr/local/lib/python2.7/dist-packages /usr/local/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages /usr/lib/python2.7/dist-packages/PIL /usr/lib/python2.7/dist-packages/gst-0.10 /usr/lib/python2.7/dist-packages/gtk-2.0 /usr/lib/pymodules/python2.7 /usr/lib/python2.7/dist-packages/ubuntu-sso-client"

for ITEM in $FUCK
do
echo $ITEM
python -c "import sys; sys.path.remove(';$ITEM'); import test.pystone"
done

This did it. I found the package: html2text-2014.9.25-py2.7.egg

$ cd /usr/local/lib/python2.7/dist-packages/html2text-2014.9.25-py2.7.egg
$ ls
EGG-INFO html2text test

It has a test directory and was earlier in the search path.

pip uninstall html2text

And away we go! What a lovely way to spend an evening…