I *do* like the recipe approach, although I admit that I’m not familiar enough with generating them to “really be keen on them” — also, some part of me wishes all of the recipes were centralized in some way to make them mentally mesh a bit more.

And I have headbanged on buildout a bit, and bounced off — it’s relatively alien to my workflow and personally I found the very simple pip/virtualenv workflow dead simple to use (and rapid to learn). But that’s just me.

This is only unfortunate in so much as setuptools hasn’t seen a lot of maintenance of late, and that setuptools does way too much stuff. Typically its behavoiur such as how scripts are generated by easy_install which irks people, or how packages are downloaded which bother others, or how package metadata is different between setuptools and distutils. But there is nothing in the python packaging ecosystem which says you need to consume or use the ugly bits of setuptools — none of the ugly bits have been immortalized in a PEP! And the ugly bits of Distutils that have been PEP-immortalized will hopefully be kicked to the curb (PEP 314’s Requires and Provides fields) without too much fuss.

Hopefully the ‘Distribute’ fork can further clarify what people want to keep and what to seperate out from setuptools. Heck, if someone came up today with yet-another setuptools fork, aside from an outcry of “code duplication”, I think this could be a good thing! It would further clarify for folks what parts of packaging are ‘standards’ and ‘interchangle formats’ and what parts are just instances of behaviour of a given tool and can easily be changed (by switching tools) and only affect user’s of that given tool.

I also think any truly satisfying solution to library management for a developer with reasonably complex requirements won’t involve any site-packages. Obviously site-packages isn’t going away any time soon, it caters really well for the “scripters” use-case where they just want to consume 3rd party libs in a more one-off scripting nature. But PEP 370 and VirtualEnv work within the assumption and the constraints of “we already have a Python installation with a shared ‘bin’ location and shared ‘library’ location” and then place installed files inside those locations. VirtualEnv is a total hack — but a beautiful one! It’s good how virtualenv is backwards compatible with all of the existing stuff — but by it’s nature it doesn’t try and re-think or re-work how the problem is solved. With any shared location, be it a root-only site-packages, a .local site-packages, or a virtualenv-cloned site-packages, any shared location is an all-or-nothing location. With any all-or-nothing shared location you will always have the potential for conflicts: installing or updating dependencies for one app may break another already installed app. When attempting to sort out version/dependency issues you can’t choose just some libs from a shared location, but easily ignore other libs of the wrong version in that location.

If you start without any notion of using a site-packages, then instead you can do something such as the Buildout approach:

~/projects
/app1
/buildout.cfg
/bin
/yolk
/app2
/buildout.cfg
/bin
/yolk
~/pythonlibs
/yolk-0.4.1-py2.6.egg
/yolk-0.4.1-py2.4.egg

This approach has the benefits of:

* Don’t need to have duplicate installations of the same version of the same library. Each version of each library only has to be installed once.

* Scripts are installed in a project-specific location, not in a shared space.

This does mean that if yolk-0.4.1 is used by two different projects, then Buildout will generate two ./[someproj]/bin/yolk script entries. But that’s a case where duplication is a good thing! While you may happen to be working on both projects today, and both projects are using the same libraries and python version, tomorrow could be a different story. You might want to update one project up to a cutting-edge dev version of a library but don’t want to do that right now for the other project. Or maybe one project is being migrated from Python 2.5 to 2.6, but you want to hold the other project back to 2.5. You might want to have a yolk-0.4.1 and yolk-0.5dev installed side-by-side for one project, where ./bin/yolk-stable and ./bin/yolk-dev are expressed. heck, you might even want two scripts that call into yolk-0.4.1, but one version of the script needs to contain an extra line or two of hard-coded python to handle a specific use case (./bin/yolk and ./bin/yolk-extra-easy). Scripts are always installed to support a specific application or project, so always putting the scripts into a project-specific location makes things a lot simpler and I think is the way to go.

Using a bootstrap in conjunction with VirtualEnv to bring a suite of dev tools into any new project is a good idea (e.g. pylint, nose, zest.releaser). But why state, “These are the dev tools I use and want to carry around with me”, when instead you can state, “This project uses these dev tools for support” and “That project uses those dev tools for support”. Each project states it’s own requirement’s and preferences for what tools (and possibly versions) it needs. In this way two developer’s who normally prefer different dev tools can easily collaborate on a project together. They don’t need to argue over what the “standard dev tool” suite should to be, instead they can say, “we should switch from Tool X to Tool Y for Project A because if gives use Benefit C”. Each project expresses it’s preferences of what scripts are powered by which version of which libs and which python interpreter (whew!). Then the only thing remaining is for installation tool to care of the ugly work of expressing those preferences by outputting the gory details of the hard-coded bits of a script into a project’s /bin/ directory.

Many language camps have toyed with the approach of writing out hard-coded scripts that declare library locations up-front, and typically they back away from it because this approach is deemed to be “clunky”. But it’s only clunky if you have to deal with the hard-coding manually! As soon as you check-in hard-coded scripts into version control, the other developers would rightfully berate you (and so instead you put “#!/usr/bin/env python” as a script header, but then you are only pushing the need to hard-code what python and what libs will be used into a hard-coded shell file …) But instead if you simply express the requirements of each script (which interpreter, which libraries, where to call into for the main function), then it becomes possible to allow a tool to automate expression of all of the clunky bits and from the developer’s perspective “clunky” becomes “effortless”. Furthermore if your definition of “clunky” is guided by actual performance benchmarks in a given deployment, then you can use different recipes to express script and package layout (flat install, egg-install, zipped-only install) until you reach a file layout that is optimized for a given deployment.

Another reason for taking the appraoach of having an install tool layout the gory details of setting up scripts and libs for a project is that you can accomodate more than one language. Sure, python is far and away my favorite language, but I appreciate python the language just fine without appreciate how the files to support a given implementation of python are layed out. Imagine if you had “VirtualRubyEnv” and “VirtualPerlEnv” and you working on some mongrelly project that wanted to combine Python, Ruby and Perl into one place (and in an acedemic/bioinformatics setting, mongrelly projects seem to be more common than pure one language only projects). Which “VirtualLang” would be the master one? What if one languages notion of where it puts stuff is incompatible with how another language lays out it’s files?

So we already have, in one incarnation via Buildout and zc.recipe.egg, ostensibly reached some form of Python packaging nirvana, where libraries are consumed from a multi-version only-installed-once-each repository and subtly different scripts aren’t attempting to overwrite each other. You still have the valid criticisms of Buildout being more difficult to approach and learn than it needs to be, and these practices need to be more standardized so that any install method is interchangeable with any other one. And people need to simply be aware where a practice that is causing them grief *is not* a set-in-stone standard (or any standard at all), but merely the behaviour of a given tool.

For the Buildout problem, this can be solved with either better docs, or simply using a different install tool but taking the same approach to installation (e.g. you could do all this with Paver). It would be be a sign of success if we could get to a place where a Ruby-centric developer who wanted to use a little Python in their project could easily manage Python script generation and library installation from Ruby code.

For standardizing things so that more and different tools can play and interoperate well (e.g. if Buildout could cherry-pick some, but not all, OS distro installed libraries). This is what the bulk of most of Tarek’s PEP writing has been pushing python packaging towards: adding install_requires and entry_points to the official metadata, writing out proper installation metadata so that you can query a location and it can tell you what package and version are installed there, and the mentioned but as-yet-un-pep’ed way to structure and consume a multi-version location.

http://pypi.python.org/pypi?:action=browse&show…

But as a general purpose project setup and installation tool, that’s kind of the point — different recipes to install different stuff. Although there are lots of recipes that are overly-specific and often a single more generic recipe is later developed which can be used, and other recipes that are near-duplicates and a “canonical” one needs to be chosen: templating of config files is a good example here.

But usually when people talk about Buildout in the context of managing Python libraries, they actually mean the zc.recipe.egg recipe, which is the most commonly used recipe used to install python packages and scripts.

As for reaching understanding with Buildout, well, I think there is still lots of room for improvement in making a more user friendly “getting starting with Buildout” document. Learning Buildout still does tend to require more headbanging than should be needed.

That being said, until Tarek can kickass and fix up distutils to be “on par”-ish with setuptools, and push out his fork of it, we play in a setuptools-based world, which is unfortunate in a lot of respects.

I *do* like the recipe approach, although I admit that I’m not familiar enough with generating them to “really be keen on them” — also, some part of me wishes all of the recipes were centralized in some way to make them mentally mesh a bit more.

And I have headbanged on buildout a bit, and bounced off — it’s relatively alien to my workflow and personally I found the very simple pip/virtualenv workflow dead simple to use (and rapid to learn). But that’s just me.

This is only unfortunate in so much as setuptools hasn’t seen a lot of maintenance of late, and that setuptools does way too much stuff. Typically its behavoiur such as how scripts are generated by easy_install which irks people, or how packages are downloaded which bother others, or how package metadata is different between setuptools and distutils. But there is nothing in the python packaging ecosystem which says you need to consume or use the ugly bits of setuptools — none of the ugly bits have been immortalized in a PEP! And the ugly bits of Distutils that have been PEP-immortalized will hopefully be kicked to the curb (PEP 314’s Requires and Provides fields) without too much fuss.

Hopefully the ‘Distribute’ fork can further clarify what people want to keep and what to seperate out from setuptools. Heck, if someone came up today with yet-another setuptools fork, aside from an outcry of “code duplication”, I think this could be a good thing! It would further clarify for folks what parts of packaging are ‘standards’ and ‘interchangle formats’ and what parts are just instances of behaviour of a given tool and can easily be changed (by switching tools) and only affect user’s of that given tool.

I also think any truly satisfying solution to library management for a developer with reasonably complex requirements won’t involve any site-packages. Obviously site-packages isn’t going away any time soon, it caters really well for the “scripters” use-case where they just want to consume 3rd party libs in a more one-off scripting nature. But PEP 370 and VirtualEnv work within the assumption and the constraints of “we already have a Python installation with a shared ‘bin’ location and shared ‘library’ location” and then place installed files inside those locations. VirtualEnv is a total hack — but a beautiful one! It’s good how virtualenv is backwards compatible with all of the existing stuff — but by it’s nature it doesn’t try and re-think or re-work how the problem is solved. With any shared location, be it a root-only site-packages, a .local site-packages, or a virtualenv-cloned site-packages, any shared location is an all-or-nothing location. With any all-or-nothing shared location you will always have the potential for conflicts: installing or updating dependencies for one app may break another already installed app. When attempting to sort out version/dependency issues you can’t choose just some libs from a shared location, but easily ignore other libs of the wrong version in that location.

If you start without any notion of using a site-packages, then instead you can do something such as the Buildout approach:

~/projects
/app1
/buildout.cfg
/bin
/yolk
/app2
/buildout.cfg
/bin
/yolk
~/pythonlibs
/yolk-0.4.1-py2.6.egg
/yolk-0.4.1-py2.4.egg

This approach has the benefits of:

* Don’t need to have duplicate installations of the same version of the same library. Each version of each library only has to be installed once.

* Scripts are installed in a project-specific location, not in a shared space.

This does mean that if yolk-0.4.1 is used by two different projects, then Buildout will generate two ./[someproj]/bin/yolk script entries. But that’s a case where duplication is a good thing! While you may happen to be working on both projects today, and both projects are using the same libraries and python version, tomorrow could be a different story. You might want to update one project up to a cutting-edge dev version of a library but don’t want to do that right now for the other project. Or maybe one project is being migrated from Python 2.5 to 2.6, but you want to hold the other project back to 2.5. You might want to have a yolk-0.4.1 and yolk-0.5dev installed side-by-side for one project, where ./bin/yolk-stable and ./bin/yolk-dev are expressed. heck, you might even want two scripts that call into yolk-0.4.1, but one version of the script needs to contain an extra line or two of hard-coded python to handle a specific use case (./bin/yolk and ./bin/yolk-extra-easy). Scripts are always installed to support a specific application or project, so always putting the scripts into a project-specific location makes things a lot simpler and I think is the way to go.

Using a bootstrap in conjunction with VirtualEnv to bring a suite of dev tools into any new project is a good idea (e.g. pylint, nose, zest.releaser). But why state, “These are the dev tools I use and want to carry around with me”, when instead you can state, “This project uses these dev tools for support” and “That project uses those dev tools for support”. Each project states it’s own requirement’s and preferences for what tools (and possibly versions) it needs. In this way two developer’s who normally prefer different dev tools can easily collaborate on a project together. They don’t need to argue over what the “standard dev tool” suite should to be, instead they can say, “we should switch from Tool X to Tool Y for Project A because if gives use Benefit C”. Each project expresses it’s preferences of what scripts are powered by which version of which libs and which python interpreter (whew!). Then the only thing remaining is for installation tool to care of the ugly work of expressing those preferences by outputting the gory details of the hard-coded bits of a script into a project’s /bin/ directory.

Many language camps have toyed with the approach of writing out hard-coded scripts that declare library locations up-front, and typically they back away from it because this approach is deemed to be “clunky”. But it’s only clunky if you have to deal with the hard-coding manually! As soon as you check-in hard-coded scripts into version control, the other developers would rightfully berate you (and so instead you put “#!/usr/bin/env python” as a script header, but then you are only pushing the need to hard-code what python and what libs will be used into a hard-coded shell file …) But instead if you simply express the requirements of each script (which interpreter, which libraries, where to call into for the main function), then it becomes possible to allow a tool to automate expression of all of the clunky bits and from the developer’s perspective “clunky” becomes “effortless”. Furthermore if your definition of “clunky” is guided by actual performance benchmarks in a given deployment, then you can use different recipes to express script and package layout (flat install, egg-install, zipped-only install) until you reach a file layout that is optimized for a given deployment.

Another reason for taking the appraoach of having an install tool layout the gory details of setting up scripts and libs for a project is that you can accomodate more than one language. Sure, python is far and away my favorite language, but I appreciate python the language just fine without appreciate how the files to support a given implementation of python are layed out. Imagine if you had “VirtualRubyEnv” and “VirtualPerlEnv” and you working on some mongrelly project that wanted to combine Python, Ruby and Perl into one place (and in an acedemic/bioinformatics setting, mongrelly projects seem to be more common than pure one language only projects). Which “VirtualLang” would be the master one? What if one languages notion of where it puts stuff is incompatible with how another language lays out it’s files?

So we already have, in one incarnation via Buildout and zc.recipe.egg, ostensibly reached some form of Python packaging nirvana, where libraries are consumed from a multi-version only-installed-once-each repository and subtly different scripts aren’t attempting to overwrite each other. You still have the valid criticisms of Buildout being more difficult to approach and learn than it needs to be, and these practices need to be more standardized so that any install method is interchangeable with any other one. And people need to simply be aware where a practice that is causing them grief *is not* a set-in-stone standard (or any standard at all), but merely the behaviour of a given tool.

For the Buildout problem, this can be solved with either better docs, or simply using a different install tool but taking the same approach to installation (e.g. you could do all this with Paver). It would be be a sign of success if we could get to a place where a Ruby-centric developer who wanted to use a little Python in their project could easily manage Python script generation and library installation from Ruby code.

For standardizing things so that more and different tools can play and interoperate well (e.g. if Buildout could cherry-pick some, but not all, OS distro installed libraries). This is what the bulk of most of Tarek’s PEP writing has been pushing python packaging towards: adding install_requires and entry_points to the official metadata, writing out proper installation metadata so that you can query a location and it can tell you what package and version are installed there, and the mentioned but as-yet-un-pep’ed way to structure and consume a multi-version location.

http://pypi.python.org/pypi?:action=browse&show…

But as a general purpose project setup and installation tool, that’s kind of the point — different recipes to install different stuff. Although there are lots of recipes that are overly-specific and often a single more generic recipe is later developed which can be used, and other recipes that are near-duplicates and a “canonical” one needs to be chosen: templating of config files is a good example here.

But usually when people talk about Buildout in the context of managing Python libraries, they actually mean the zc.recipe.egg recipe, which is the most commonly used recipe used to install python packages and scripts.

As for reaching understanding with Buildout, well, I think there is still lots of room for improvement in making a more user friendly “getting starting with Buildout” document. Learning Buildout still does tend to require more headbanging than should be needed.

That being said, until Tarek can kickass and fix up distutils to be “on par”-ish with setuptools, and push out his fork of it, we play in a setuptools-based world, which is unfortunate in a lot of respects.