Comments on: Trapped in python package; send food.

By: kteague

kteague — Sun, 19 Jul 2009 11:00:19 +0000

Twiddling with the shebang, that’s a Distutils thing:

http://docs.python.org/distutils/setupscript.ht…

“The only clever feature is that if the first line of the script starts with #! and contains the word “python”, the Distutils will adjust the first line to refer to the current interpreter location.” Heh, one man’s “clever feature” is another man’s headache. The –executable option allows you to override this behaviour though.

But the other option is to declare a set of repeatable steps so that you can compile from source on fresh machines. Buildout is the uber-tool for this job! Here’s a decent starter for compiling a Python interpreter using Buildout (http://bluedynamics.com/articles/jens/build-pyt…). And generally anything is hairy or a PITA to compile on OS X, you can Google for “buildout hard-to-build-thing” and someone has usually put together a recipe for building it.

By: kteague

kteague — Sun, 19 Jul 2009 06:00:19 +0000

Twiddling with the shebang, that’s a Distutils thing:

http://docs.python.org/distutils/setupscript.ht…

By: Shaun Walbridge

Shaun Walbridge — Sat, 18 Jul 2009 17:54:02 +0000

Perhaps I'm missing something, but have you given buildout (http://www.buildout.org/) a shot? It's used to good effect in the Plone community, where it helps manage complex dependencies including compiled software, and works quite nicely once you've learned the basics of creating buildout.cfg files.

By: jnoller

jnoller — Sat, 18 Jul 2009 15:39:30 +0000

I missed that in 370, I thought it only handled the site-packages bit — I’m surprised it handles the bin scripts properly — they should not be installed in /bin. I’ll double check my stuff to check that, and the proper shebang line (cause so far, setuptools/distutils isn’t playing nice like that.

By: Damjan

Damjan — Sat, 18 Jul 2009 15:19:14 +0000

If you only need to use Python 2.6 (or better) you can use PEP370 instead of virtualenv:

all I do is:
PYTHONUSERBASE=$HOME/my-python
pip.py install --install-option="--user" PythonPackage

There's no copy of the interpreter, no setuptools required, the shebang is still "#!/usr/bin/python" everything is installed in $HOME/my-python/bin and $HOME/my-python/lib/python2.6/site-packages

If I switch to python3(.1) I don't have to do anything special since my custom packages for 3.1 will be installed in $HOME/my-python/lib/python3.1/site-packages (except that the /bin might be conflicting).

By: Giuseppe

Giuseppe — Sat, 18 Jul 2009 13:14:35 +0000

I forgot: boostrapping is also kind of easy using the API. These are the two file I use on a (toy) project I am developing:

http://pastebin.com/f71e884ba to boostrap a box and install python in it
http://pastebin.com/fe1f06fa to install non-python dependencies (python dependencies use a pip requirements file).

This is something I'd really like to automate, but I need to figure out some problems before starting to code...

By: Giuseppe

Giuseppe — Sat, 18 Jul 2009 13:07:39 +0000

Thanks :)

I completely agree about the symlink. Actually if there are better solutions, the code that does that is limited to few lines that are easily changed. However, I could not think of anything better.

I mean, on linux there would be FUSE, or some tricks with LD_PRELOAD like fakechroot does. But they would need FUSE/etc to be installed on the guest machine, while it is important for me to have as few dependencies as possible on that side. Besides, Mac compatibility is very important for me :) (I know, mac-fuse and everything, but it is a huge jump in complexity).

By: jnoller

jnoller — Sat, 18 Jul 2009 12:59:00 +0000

I like the way you think ;)

Ugh to the hoops you have to jump through to symlink things. That’s gross, I know why, but ugh.

By: Giuseppe

Giuseppe — Sat, 18 Jul 2009 12:49:48 +0000

Bash is without a doubt the easiest way to do it, but I'd like to try a scons/waf approach instead of a series of check_calls, i.e. abstracting some commands (configure, make, etc...) to python declarations. This could result in more platform independency (maybe it could even support windows.). The new system would not necessarily replace the bash one: bpt is made to support different ways of installing packages inside a box, it just provides a virtual filesystem based on symlinks. "build", "autobuild", etc... are just commands implemented on top of it.

The "virtual filesystem" /tmp/box_<...> trick works this way: when you run the env script, a symbolic link is created from your current box location to /tmp/box_<...> where <...> is an id unique to the box. So if the box is relocated the link is created to point to the current box directory (actually if you move the box you will have to remove the link by hand. Better safe than sorry).
This trick makes python work if it is installed in the box, because /tmp/box_<...>/bin/python will point to the correct binary regardless where you have put your actual box.

Thanks for the links about licenses, I'll have a look ASAP.

By: jnoller

jnoller — Sat, 18 Jul 2009 12:40:01 +0000

I went the bash script route too, given it's faster/easier then a series of check_calls from python. As for the boxes being portable - they're not, because of exactly what you mention - the hardcoded shebang line. If you move a box from /tmp/sandbox_<...>/bin/python to say, /home/jnoller/sandbox_<...>/bin/python - those hardcoded shebangs break. And I agree with compiling python *into* the sandbox, that's a trick a lot of us use to have many, not-system-wide installs of Python running.

I'd have to resurrect my notes (I don't know what I did with them) on the patching, but things like dependency resolution, virtualenv support and swapping to python scripts comes to mind.

As for licenses, that's really a matter of personal taste. I avoid it, and stick with Apache License 2.0 (compatible with the python software license) for most things. The more permissive licenses don't invoke additional clauses, meaning I can import bpt in my app, without making my app's license change (wherein the GPL would force the combined app to be GPL).

The licensing thing was part of a debate recently, without digging too much into my own reasons, see:

http://farmdev.com/thoughts/80/why-you-should-n...
http://zedshaw.com/blog/2009-07-13.html
http://www.b-list.org/weblog/2009/jul/14/licens...
http://jacobian.org/writing/gpl-questions/