Cleaning out the inbox: Updated Import pseudo code, filesystems and workingenv

by jesse in ,


In between bugs(bees) and pre-natal exams and other "bodies of agitated fun" I figured I'd continue the cleaning out of the inbox. A little while ago, Brett Cannon wrote up the pseudo code version of python's import system for his importlib project. He updated it on 2007-06-12 to include how modules are searched for, which, for me, is fantastic given the various ways I have to *cough* abuse the import system sometimes ((see PyMOTW: os (Part 2) - and popen2 isn’t thread safe)).

Of special note (for me) is this: # If the module is already cached in sys.modules then move along. if name in sys.modules: continue I sometimes wish that A> I had robot eyes, and B> Python's import system would take the last module in a path as the module of that name to use. Mind you, this is a small wish and easily forgotten. Also, reading the rest of it I also (as others do) wish that Python did not have to compile .pyc files in the same directory as the .py files. This is all sorts of bad for security (as a coworker recently admonished me about). I really wish I could do something like: export PYTHON_PYC_CACHE=/tmp/._pythonCache ... dance all night Or something along those lines. I see a future coming closer where layered filesystems ((see: Mike Fletcher's "I want an auto-journalling overlay file-system", "Bash in Bitfrost" and "Unioning File Systems for Fun and Profit")) (users/runtime space hard separated from binary/os space) and some level of virtualization will become the status quo rather than the exception.

(Note: Know what would be crazy? Using a no-bullshit object-store style filesystem with layering and having the python byte code put in the right place. But that's just because I think it would be cool, the object-store part is optional, and just because combining things is cool)

Now, something people might forget, your current working directory (i.e: the directory of the script) is always position 0 on sys.path: woot:~/subversion jesse$ python Python 2.5 (r25:51918, Sep 19 2006, 08:49:13) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path ['', ...wall of text

This of course means that if you have an OS module in your $CWD and one in the stdlib, due to the import system's method of first-in-is-cached/used, this means your local os.py module overrides the builtin os module. So dutifully call it something other than that, or invest in a namespace. For example, a demo of the sitecustomize.py I use when working on a side project:

#!/usr/bin/python
import os
try:
    import site
except: pass

if 'MYPACKAGE' in os.environ:
    site.addsitedir(os.environ['MYPACKAGE'])

Generally speaking, MYPACKAGE resolves to /Users/jesse/projects/python/some_package/ which has a magical __init__.py sitting in it. Or, if I am feeling particularly hazardous, I use Ian Bicking's wonderful workingenv.py script. To quote:

This tool creates an environment that is isolated from the rest of the Python installation, eliminating site-packages and any other source of modules, so that only the modules (and versions) you install into the environment will be available. This allows for isolated and controlled environments, as well as reproduceability. This is similar to virtual-python, but without the symlinks and with some additional features.

With that said, I'm going back to operation: "Wait for Baby".