jessenoller.com

So, I (and many oth­ers) have lamented pack­ag­ing issues in Python. Some peo­ple are focused on inte­grat­ing with ven­dor sys­tems (such as apt (.deb) and yum (rpm)) — while oth­ers are con­cerned with disutils/setuptools/etc.

Still oth­ers (like me, and maybe I’m alone) are trapped in a tween-state. We’re par­tially using ven­dor sys­tems, and par­tially using self-compiled ver­sions of python.

The car­di­nal “rule” has been not to “touch” the vendor-specific instal­la­tions of python (this includes you, Linux). For exam­ple, on OS/X — any time you run easy_install or pip you install into the global site-packages direc­tory. The same applies when you do the same on linux, and when you run apt-get install/yum-install. Things go into that global, shared directory.

This sucks. Here’s why:

• Ver­sions. Some appli­ca­tions depend on very spe­cific ver­sions of libraries. This is because the main­tain­ers of the libraries they depend on are bad, and break back­wards compatibility.
• site-packages becomes a toi­let. Before my near OCD lev­els of clean­li­ness, I checked my system’s site-packages direc­tory — I think all told I had about 250 dif­fer­ent .eggs/packages/modules/etc all lit­tered in there. And .pth files, and half-exploded things with meta­data direc­to­ries. And I think I found a squir­rel in there.
• “glob­ally” installing things like nose, pip and setup­tools put the binary scripts in /usr, /usr/local and so on. This again causes those direc­to­ries to become a toilet.
• In some cases, upgrad­ing some­thing out­side of your ven­dor pack­ages — say, some­thing pre-installed into RedHat’s python ver­sion can in fact, break and side-effect the sys­tem as a whole.

So, I guess you could say “system-level site-packages con­sid­ered harm­ful”. Once I real­ized the hor­ri­ble error of my ways, I switched to vir­tualenv/vir­tualen­vwrap­per. This works great for me. But at least on OS/X — some­thing was lacking.

That some­thing was depen­den­cies needed to com­pile some­thing like read­line into python. I could install the read­line egg from pypi and just “work around it”. Or I could install mac­ports (which is bro­ken in many ways) and install the read­line devel­op­ment libraries in there.

Unfor­tu­nately, mac­ports also side effects your sys­tem in unde­sir­able ways. Sud­denly you’re link­ing to things you don’t real­ize, you’ve got things com­piled in you don’t need/want, and so on.

So, what’s a guy sup­posed to do?

Well, since I’m not afraid of com­pil­ing things, I built a mini-macports for myself. I made a direc­tory (named “slash”) in my home direc­tory, and com­piled things like read­line into it. I then point the python com­piles to that direc­tory and move on with my life (I love you, –pre­fix). After compiling/installing PIL, Read­line, etc into this direc­tory as well as a pile of python ver­sions, and slap­ping vir­tualenv on top of it I was feel­ing pretty good. I get only what I need, and vir­tualenv keeps things out of the global directories.

Well. Minus the fact that it’s huge, non portable and it’s sort of a pain in the ass.

Then, I got an itch — I wanted to build a “python mega­pack” — I lov­ingly named it python-kitchensink. My goal was to repeat what I did above, and then offer it as a down­load for peo­ple who want to avoid this pain them­selves on OS/X.

Easy enough. Minus one nit.

You can’t tar the damned thing up. I don’t know if it’s a side effect of disutils/setuptools, but scripts being installed into this root, were hav­ing the #! lines hard coded to the exact path of the inter­preter. This means if you went through all this com­pi­la­tion, and then installed easy_install — and say you did this in “/Users/jesse/myslash” — easy_install would get “#!/Users/jesse/myslash/bin/python2.6″ hard coded into it.

Instead of kitchensink, I should have named it “jesse cusses a lot”.

So, back to square one. Or rather “think about this in the back of my mind, for­get about it and then change to a new job”.

For­get­ting about try­ing to do this for OS/X, I end up need­ing to do some­thing eerily sim­i­lar on Fedora Core. Now, com­pi­la­tion of python with all the bells and whis­tles on Fedora is sim­ple — “yum install xxx-devel” and then just run the compile.

The goal was to make a fully-featured python 2.6 install on FC10, and then boot­strap the user(s) into a vir­tualenv so that noth­ing got plopped into the global directories.

Well — minus the fact fedora core 10 ships with python 2.5. And tools like virtualenv/etc from the yum repos lag behind the ver­sions I want/need. Damnit. Do I stick to RPMs? Do I boot­strap it enough to “just work” and then pip install the rest? What about python2.6? Where are my pants?

There’s another catch: it has to work on *first boot* and there’s no net­work on that first boot.

So, for­get­ting my expe­ri­ences with com­pil­ing all this stuff myself on OS/X, that’s what I do at first. I install all the devel pack­ages, build an RPM which con­sumes a tar­ball I cre­ate, and add it to a local repo, and throw it in the kick­start file which spews out the images.

Oh but wait. The hard­coded #!‘s come back and bite me in the ass. The build server com­piles things in a tem­po­rary direc­tory, and then installs easy_install and all of the other tools into the –prefix’ed python install. That temp direc­tory is named some­thing like “–TMPxx1341234DFLKJ1341234.xxx.hahaha”. Soooo, I get “#!/–TMPxx1341234DFLKJ1341234.xxx.hahaha/bin/python”. That’s about as use­ful as a bee­hive in my toilet.

Easy fix though: just make sure the build­server doesn’t have any­thing in the even­tual loca­tion of the installed ver­sion from the rpm (/opt/lazercats (ok, not really)) and just com­pile every­thing there.

Suc­cess, and win. Heck, I even get it to boot­strap vir­tualenvs for the users. Then I find out I’ve increased the image size by 40 or so megabytes. This imme­di­ately wipes the grin off my face and makes me real­ize I have again, failed. You see, I can’t freely increase the image size like that.

I need python 2.6. So, step one is to swap to fc11. Ok, good. I also want to avoid using the lag-behind ven­dors pack­ages except for the bare min­i­mum foot­print I need to boot­strap the envi­ron­ment. This means mod­i­fy­ing the kick­start pack­ages list like this (note: I also can not install a com­piler — which is needed for a lot of packages):

# Python utilities
# python-lxml is == 2mb
python-lxml
python-setuptools
python-crypto
python-paramiko
python-pycurl
# Needed for virtualenv < 1.0 mb
python-devel
python-setuptools-devel

Why on earth is python-devel needed for vir­tualenv? Why python-setuptools-devel? Whyyyy??!
Ok, so I’m only going to be stuck with upstream ver­sions of lxml, setup­tools (which hasn’t revved since the earth cooled) and a few oth­ers. Fine.

I then jump into kick­start file and pop in:

%post --nochroot
cp python-dependencies.txt $INSTALL_ROOT/root/python-dependencies.txt
%post
%include post.txt
%end

In post.txt:

# Python environment setup

# Temporarily make DNS work
echo "nameserver 10.1.1.10" >/etc/resolv.conf

# Python environment setup
( cd /root
    /usr/bin/easy_install virtualenv
    /usr/bin/easy_install virtualenvwrapper
    /usr/bin/virtualenv /opt/thatthing
    /opt/foobar/bin/easy_install pip
    /opt/foobar/bin/pip -E /opt/thatthing install -r /root/python-dependencies.txt
    rm -rf build/ python-dependencies
    echo "export WORKON_HOME=/opt" >>/home/jnoller/.bash_profile
    echo "source /usr/bin/virtualenvwrapper_bashrc" >>/home/jnoller/.bash_profile
)
rm -f /etc/resolv.conf

# End Python setup

The python-dependencies.txt is a pip require­ments file and looks like this:

# use pip install -r


# http://code.google.com/p/boto/
boto

# http://docs.fabfile.org/0.9/
fabric

# http://ipython.scipy.org/moin/
ipython

# http://tools.assembla.com/yolk
yolk

# http://code.google.com/p/httplib2/
httplib2

# http://ipaddr-py.googlecode.com

http://ipaddr-py.googlecode.com/files/ipaddr-1.1.1.tar.gz

Note, I can’t also plop svn, hg, git, etc in here — so pack­ages not on the cheese­shop in or pack­aged right are a no-go.

The trick here is that the %post com­mands in the kick­start envi­ron­ment run in a chroot of the OS being cre­ated. This means, once the new image is loaded (say, in EC2) I can ssh in, and hit “workon thatthing”. In real­ity, the WORKON dir should be else­where, but I’m going to let users over­ride that. As it is, the “one true python” ver­sion is the one in /opt — no one (even me) gets to touch the sys­tem ver­sion of python.

I now have a python envi­ron­ment, avail­able on first boot, iso­lated from the OS-provided one. I can spawn infi­nitely more vir­tualenvs and play all day long. The few global things I have are easy_install and some libraries which I hope I don’t need to rev myself.

I still haven’t licked the OS/X part. I’m prob­a­bly just going to have to com­pile the barest pos­si­ble envi­ron­ment in some­thing like /opt/python-ks and go from there. Given I’d need to com­pile all of the depen­den­cies into it (such as read­line) I may just end up writ­ing a big script to grab all the bits and then com­pile it into a loca­tion the user pro­vides. The nice thing is that once I boot­strap python and vir­tualenv into the basic tree, I can use pip bundles/requirements files to pull in the rest.

All told, I sit here look­ing at the mess I’ve slogged through — and then I real­ize the entire python-packaging dis­cus­sion on python-dev just exposes a whole ‘nother can of worms. Ver­sion­ing in a sin­gle site-packages direc­tory, how app devel­op­ers con­flict with OS ven­dors, etc. It’s a mess. OS Ven­dors lag behind devel­oper released ver­sions, and come to depend on what’s installed there (have you ever bro­ken yum on a Fedora box? I have.).

I hope Tarek gets a chance to clean a lot of this up — and while I’m against “every­thing and the kitchen sink” in the stdlib — hav­ing some method/API of build­ing out “an official-like” vir­tualenv setup (maybe mak­ing virtualenv’s life eas­ier) would be nice.

Edit to add: I real­ize that hard­cod­ing the she­bang line is desir­able in many cases, the obvi­ous rea­son is that you need to be pointed at the inter­preter which has your dependencies/libraries in it. Not hav­ing a clear way of alter­ing that behav­ior (other than a “clever” sed script) is unfortunate.

See this fol­lowup as well

So, I (and many others) have lamented packaging issues in Python. Some people are focused on integrating with vendor systems (such as apt (.deb) and yum (rpm)) - while others are concerned with disutils/setuptools/etc. Still others (like me, and maybe I'm alone) are trapped in a tween-state. We're partially using vendor systems, and partially using ...