| Subcribe via RSS

“Too many ’self’ in python.That’s a big flaw in this language.”

June 28th, 2007 | 6 Comments | Posted in Programming, Python

The title is of course not my creation - it is another thread on comp.lang.python (serious, you should stop by, I lurk a lot and read posts from that xiu fellow).

In this particular thread:

I'm currently using Python. I find that a instance variable must
confined with self,
for example: [snip] That's a big inconvenience in coding ,especially when you have lot of variable If you method need 10 variables ,you have to type "self" for 10 times and that also makes your variable longer. From My point,I think this only help python interpreter to deside where to look for. Is there anyone know's how to make the interpreter find instance name space first?
Or any way to make programmer's life easier?

That's the original post. Without delving too deep into the nuances of his critique - I like explicit self. As Bjorn wisely points out "Explicit is better than implicit.

I really don't understand the near visceral dislike of "self." some people have. I mean, "self" you're passing in, calling, reference your local, "self" scope/object/underpants.

I mean, I really like self, and I find myself missing it learning Java-Fu.

Chroot and Python discussion and random pyc thoughts

June 28th, 2007 | 7 Comments | Posted in Programming, Python

Since I'm finally back from another exciting edition of almost-labor at the hospital and catching up, I thought I would point out a discussion on python-dev about chroot jails and python. Interesting information and tangentially related to some of my thoughts on the .pyc location stuff.

The conversation is going on here and you can view some other information.

If course, there's a shout out to Brett's security work too.

Here's is the wiki page referenced in the thread: How can I run an untrusted Python script safely (i.e. Sandbox)

A lot of the utilities are interesting, but I'm still interested in the byte-code location of things.

Some thoughts on my pyc thing:
- One thing to note, is that if the user running the python interpreter does not have write (+w) access to the directory the imported .py is located in, the .pyc/.pyo file is not written. A .pyc file is an optimization for module loading only.

Since this is the case, is worrying about layered filesystems/storing .pyc files in "other" directories really that hot of an issue for me? Maybe. I'd still like to see if I can get the wherewithal to drive pep 304 forward - I'd still like to be able to control where to put things, but if you use compileall() and ship the .pyo/.pyc stuff *or* you just make sure the daemon that's invoking the interpreter does not have +w on it's script/binary directory (which it shouldn't) you could be ok.

Pep 304 followup

June 27th, 2007 | | Posted in Personal, Programming, Python

I'm going to be following up on the comments from Brett and others about python bytecode location for this post as soon as I find sanity again - "Operation Pending Baby 1" has caused a mild for of insanity to set in.

10 Great Tips on How Not to Be the Obnoxious Newcomer in Your Workplace

June 25th, 2007 | | Posted in Comedy, Personal

10 Great Tips on How Not to Be the Obnoxious Newcomer in Your New Workplace: "You've just joined a new company and are all fired up to change things—stat. While your enthusiasm is commendable, you might want to proceed slowly at first. It's too easy for your helpful (and very possibly dead-on) observations and suggestions to be seen as criticisms and demands."

My question: Can they do a followup, "how not to be the embittered cynical old coot" or "How not to eyeball new technology with dismay, distrust and a long stories about how "you used to have declare variables in the constructor back in the days and you loved it""?

(Via digg.)

Cleaning out the inbox: Updated Import pseudo code, filesystems and workingenv

June 25th, 2007 | 7 Comments | Posted in Programming, Python

In between bugs(bees) and pre-natal exams and other "bodies of agitated fun" I figured I'd continue the cleaning out of the inbox.

A little while ago, Brett Cannon wrote up the pseudo code version of python's import system for his importlib project. He updated it on 2007-06-12 to include how modules are searched for, which, for me, is fantastic given the various ways I have to *cough* abuse the import system sometimes1.

Of special note (for me) is this:

# If the module is already cached in sys.modules then move along.
if name in sys.modules:
continue

I sometimes wish that A> I had robot eyes, and B> Python's import system would take the last module in a path as the module of that name to use. Mind you, this is a small wish and easily forgotten. Also, reading the rest of it I also (as others do) wish that Python did not have to compile .pyc files in the same directory as the .py files. This is all sorts of bad for security (as a coworker recently admonished me about). I really wish I could do something like:

export PYTHON_PYC_CACHE=/tmp/._pythonCache
... dance all night

Or something along those lines. I see a future coming closer where layered filesystems2 (users/runtime space hard separated from binary/os space) and some level of virtualization will become the status quo rather than the exception.

(Note: Know what would be crazy? Using a no-bullshit object-store style filesystem with layering and having the python byte code put in the right place. But that's just because I think it would be cool, the object-store part is optional, and just because combining things is cool)

Now, something people might forget, your current working directory (i.e: the directory of the script) is always position 0 on sys.path:

woot:~/subversion jesse$ python
Python 2.5 (r25:51918, Sep 19 2006, 08:49:13)
[GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', ...wall of text

This of course means that if you have an OS module in your $CWD and one in the stdlib, due to the import system's method of first-in-is-cached/used, this means your local os.py module overrides the builtin os module. So dutifully call it something other than that, or invest in a namespace. For example, a demo of the sitecustomize.py I use when working on a side project:

 
#!/usr/bin/python
import os
try:
    import site
except: pass
 
if 'MYPACKAGE' in os.environ:
    site.addsitedir(os.environ['MYPACKAGE'])

Generally speaking, MYPACKAGE resolves to /Users/jesse/projects/python/some_package/ which has a magical __init__.py sitting in it. Or, if I am feeling particularly hazardous, I use Ian Bicking's wonderful workingenv.py script. To quote:

This tool creates an environment that is isolated from the rest of the Python installation, eliminating site-packages and any other source of modules, so that only the modules (and versions) you install into the environment will be available. This allows for isolated and controlled environments, as well as reproduceability. This is similar to virtual-python, but without the symlinks and with some additional features.

With that said, I'm going back to operation: "Wait for Baby".

  1. see PyMOTW: os (Part 2) - and popen2 isn’t thread safe []
  2. see: Mike Fletcher's "I want an auto-journalling overlay file-system", "Bash in Bitfrost" and "Unioning File Systems for Fun and Profit" []

Cleaning out the inbox: Contracts, Typecheck and Lockfiles.

June 24th, 2007 | 2 Comments | Posted in Programming, Python

Time to scrub some of the fifty+ things I've had flagged for followup from the past few weeks now that I can finally crawl out from under a rock. I watch the cheese shop RSS feed and the ANN: emails on the python-list quite a bit. Here's are some of the more interesting highlights I've found.

First up: pycontract.

Programming by Contract allows a programmer to document a function/class with statements describing behavior.

This is interesting in that it uses the same-style trick epydoc does to an extend and adds syntax into the doc string - in the case of epydoc, it's all for doc-generation.

In the case of pycontract, it's actually parsed code. To quote the code:

 
# Implementation
# --------------
# This module is divided into four parts:
#
# 1 -- Find docstrings and parse them for contract expressions...
#
#      This is accomplished by:
#      a. recursive enumerating elements in modules, classes using code
#         lifted from the 'inspect' Python standard module
#         see: checkmod, _check
#      b. scanning the docstrings of public functions with a regular
#         expression looking for lines that matches 'pre:' 'post:' or
#         'inv:' at the start
#         see: parse_docstring
#      c. Using the 'tokenize' Python tokenizer to build expressions
#         see: _read_block, _read_decl, _read_token
#
# 2 -- Construct functions that do contract checking...
#      This is done by just constructing big strings that are function
#      definitions.  Each function or method gets associated with four
#      'helper' functions... to check pre-conditions, to save old
#      values, to check post-conditions, and of course the saved
#      original code.  These are stored as function attributes.
#         see: _install_wrapper, _define_checker
#
# 3 -- Run-time support for call checking
#         see: call_public_function_*, call_private_function_*,
#              call_public_method_*, call_private_method_*,
#              call_constructor_*, call_destructor_*
#
# 4 -- Declarations for use within contract expressions
#      includes 'forall' 'exists' 'implies' and exception classes

Note that if you look at the example sort.py script it does not exclude the possibility of doctests, in fact it uses them (I'm still not a huge face of doctest - but that's a different thing entirely).

Obviously, the first thing that comes to mind when looking at this (at least for me) is the ability for enforce argument types and other return values/etc which is nice - I just can't help but think that this could be done with decorator syntax instead ala the typecheck's module syntax:

 
@typecheck_args( (int, str) )
 
	Takes a two-tuple composed of an integer and a string
 
@typecheck_args( [int, int, str] )
 
	Takes a list, with types in a repeating pattern of integer, integer,
	string.
	The list must complete the pattern; for example, two integers would be
	invalid. For example, [3, 4, 's', 5, 6, 'f'] matches, but [3, 4] and
	[3, 4, 'f', 5, 6] do not because they do not complete the pattern.

For more on decorator syntax, you should read the PythonDecoratorLibrary wiki page!

Next up, is lock_file. Now, I think this is one of those things that everyone has had to do sooner or later. I know that of at least 4 projects I've worked on I've had to do something like this. This python module is one simple lock_file.py module and implements a nice, clean interface to this common problem.

I am a huge fan of this:

 
    from lock_file import LockFile
 
    lock_f = LockFile('/var/run/app.lock', wait = True)
    try:
        do_something_useful()
    finally:
        lock_f.release()

Obviously, if you wanted to get cute, you could also implement this in a decorator ala:

 
@acquire_lock
def main():
    ... stuff ...

And you could collapse it all nicely. I'm just a fan of this module based on how simple and clean it is. I've had to spend an inordinate amount of time in the /var/run/... tree and coordinating /var/lock/subsys/... lock files for shell scripts/init scripts, and I just want to hug the lock_file module. I'd love the inclusion of something exactly like this in the stdlib, but I know why that would/should not be.

Python 3000 Resource page

June 24th, 2007 | 2 Comments | Posted in Programming, Python

I've just put together a page of resources together for Python 3000 based on information I had bookmarked + links from the last post.

http://jessenoller.com/python-3000-resources/

Check it out, let me know if I need to add something.

ohloh: Here is the beer list.

June 23rd, 2007 | 4 Comments | Posted in Programming, Python

Via a link in Brett's weblog, I found ohloh. Better yet, I found the list of "people to whom I wish to owe a beer"

Right now, there are 92 UIDs on this site. I can't count the ones in the cheese shop, because if I did I'd need VC-level money to say thanks.

The more you know - the discussion around Python 3000

June 23rd, 2007 | 5 Comments | Posted in Personal, Programming

I had a nice long post to talk about all the rounded up posts about python 3000 - all of the ones worried, with admittedly good cause, and all those defending the process.

Rather than post the long winded version, here's the abbreviated summary and a few side notes - Doug has already summarized some of my thoughts and with Brett's latest "Python-Dev does care about the 2.x -> 3.0 transition" post really taking the cake with regards to how I feel about this.

Per Brett:

In the end it all doesn't matter. Python 2.x is not going anywhere, so even if Py3K turns out to be a flop Python will live on. But if Py3K does do well (and I expect it will in the end), Python will be better for it. I know that I will end up developing for Py3K exclusively whenever I can and be happier for it. Hopefully most of you will end up being happier as well.

To date there has been some arm waving and concern raised by various people around the fracturing of the Python community (not just Martijn mind you (so I am not picking on you)) around the python2x to 3x conversion and changes coming in the language. There have been many discussions around this across the internet(s), mailing lists and blogs.

But a warning - before you take any single blog post at face value, and before you jump to any conclusions about how hard, or what has already been discussed - you really need to look up the history (via the peps and mailing list) of Python 3000's development, that's where a lot of the real work has occurred.

Remember many people in the community are not exposed to the py3k mailing list. Heck, most people only hear second-hand information from a fellow Pythonista about py3k unless they attended pycon, or - like me - are obsessed with reading things on the internet and constantly troll mailing list archives. So when something like Guido's original post gets around - people who were not directly exposed to the discussion around why changes where made, who have not read the PEPs, or are worried about the fracturing of the community make posts on their weblogs, and the echo chamber effect takes hold - someone sees one posts, responds, someone sees the response to that, and responds but lacks the context of the original, and so on. The discussion gets so far from the original decision that it gets difficult to figure out the original point. All of this takes place in public.

We should all get closer to the source - I actually created a Python 3000 search engine: Python 3000 Google Search to help look up python 3000 information (I'll add more sites as I find them).

Through all of the discussions, here's one of the ones I clapped for:

Python is not trying to be Haskell. Besides, if it is Haskell we want, we know where to find it. Only that I had to write Python code, after something like one-and-one-half years of straight Ruby (no, not Rails, just scripting) and Haskell + OCaml, and I found that my expectations had changed along with my style. More recursion, an expectation of implicit returns, et al. So, I'm generally rusting at Python. But ... Python is becoming a better Python, and I really would have loved to be in the community yet. Python is great, even without being of the paradigm I favour these days.

Yes. Exactly. Precisely.

Doug already listed out a bunch of the posts - here's the list again, with additions as I find them:

There have been a lot of historical posts too: By far the best place to look, or get a sense of look, feel or other things?


woot:~ jesse$ cd subversion/
woot:~/subversion jesse$ ls
Pyro python3k-svn update-py3k update-python
pypy-dist scriptutil update-pypy update-scriptutil
python-svn tahoe update-pyro update-tahoe
woot:~/subversion jesse$ sh update-py3k
U python3k-svn
Checked out revision 56083.
woot:~/subversion jesse$ cd python3k-svn
woot:~/subversion/python3k-svn jesse$ ./configure
checking MACHDEP... darwin
woot:~/subversion/python3k-svn jesse$ ./python.exe
Python 3.0x (p3yk:56083, Jun 23 2007, 21:57:25)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> print 'hey dude'
File "", line 1
print 'hey dude'
^
SyntaxError: invalid syntax
>>> raise Exception, 'Oh noes'
Traceback (most recent call last):
File "", line 1, in
Exception: Oh noes
>>>

In all seriousness, with regards to Python 3000 - I don't think there are any losers, at all. Yes, the transition will be gradual, the size of the individual code base being converted will obviously dictate the length by which that team has to stick with the 2.x series. Yes, the "core python team" will with everyone they can to help drive this effort, as will all python 3000 adopters.

And yes, it will be a long road - as anyone who has worked on an appliance/embedded device will tell you, change to core internals (JVMs, interpreters, Kernels) comes slow. Sometimes, you have to have 2-3 or more versions actively being tested or installed as a fallback position. Heck! Some of the machines I work on are still based on python 2.3, which is why on my laptop I have 2.3, 2.4 and 2.5 - the project I get paid to work on is largely 2.4 which means I can not adopt much of the "rest of the communities" nice tools that came in 2.5. 2.6 is a year or more from my grasp.

Many people will first experiment with whatever their OS ships. OS/X Leopard will ship with Python 2.5 - Fedora Core, Ubuntu and other Linuxes will see 2.5 rolled out as well. The world will continue for us on 2.x for the next few years as we work on conversion kinks. Most OSes should ship both a 2.x and 3.x interpreter as soon as 3.x hits final - the larger frameworks will probably start their initial courtship with 3k around alpha/beta.

In 3 years? 2.5 will be a punch line to a joke in a doc string. This is not revolution - nothing is going to die here. But as in Evolution, some people get wings sooner than other, that's all - and we might have to glue wings onto some of the lowly earth-bound. As Martijn points out, there will be confusion - yes! There shall be such confusion - and all of us who are debating this now amongst ourselves will fly from our mighty ivory towers to help and guide newbies to the Python3k FAQ, to tell them to read the news group archives, and maybe we'll even help them convert their code. Change is scary, but it's also fun.

Brett, Guido and the rest of the core devs are people just like the rest of us - the process is open, transparent and discussed (in the open). Get involved, sign up to the mailing list and air your concerns, sync subversion - get syntax errors! Go forth and be Pythonic but remember all of this is still being developed, and we can all enact change.

update: Shilling my brand new Python 3000 Resource page, I am!

Guido van Rossum’s Weblog: Python 3000 Status Update (Long!)

June 19th, 2007 | | Posted in Programming, Python

The BDFL has spoken:

Guido van Rossum's Weblog: Python 3000 Status Update (Long!): "Here's a long-awaited update on where the Python 3000 project stands. We're looking at a modest two months of schedule slip, and many exciting new features. I'll be presenting this in person several times over the next two months."

Of particular interest to me is the renewed focus on python2.6 as the warning gap between 2.5 and 3k, to quote:

Python 3.0 will break backwards compatibility. Totally. We're not even aiming for a specific common subset. (Of course there will be a common subset, probably quite large, but we're not aiming to make it convenient or even possible to write significant programs in this subset. It is merely the set of features that happen to be unchanged from 2.6 to 3.0.)

Good. Now don't be scared, because he goes on to explain:

  • Python 2.6 will support a "Py3k warnings mode" which will warn
    dynamically (i.e. at runtime) about features that will stop working
    in Python 3.0, e.g. assuming that range() returns a list.
  • Python 2.6 will contain backported versions of many Py3k features,
    either enabled through __future__ statements or simply by allowing
    old and new syntax to be used side-by-side (if the new syntax would
    be a syntax error in 2.x).
  • Complementary to the forward compatibility features in 2.6, there
    will be a separate source code conversion tool. This tool can do
    a context-free source-to-source translation. As a (very simply)
    example, it can translate apply(f, args) into f(*args).
    However, the tool cannot do data flow analysis or type inferencing,
    so it simply assumes that apply in this example refers to the
    old built-in function.

You should really go read the full write up - here's a thing I am happy about (other than Abstract Base Classes):

You might have guessed that "classic classes" finally bite the dust. The built-in class object is the default base class for new classes. This makes room for a variety of new features.

Yay! I've been grinding on really learning meta-classes and descriptor syntax the last week or two for some side projects and this makes me happy.

ps: Voluntary Abstract Base Classes

(Via Planet Python.)