jessenoller.com

Py3k: passing tests like it ain’t nothin.


woot:~/subversion/python3k-svn jesse$ make test
running build
running build_ext

running build_scripts
./python.exe -E -c 'import sys ; from distutils.util import get_platform ; print(get_platform()+"-"+sys.version[0:3])' >platform
find ./Lib -name '*.py[co]' -print | xargs rm -f
./python.exe -E -tt ./Lib/test/regrtest.py -l
test_grammar
test_opcodes
test_dict
...snip...
test_xmlrpc_net
test_xmlrpc_net skipped -- Use of the `network' resource not enabled
test_zipfile
test_zipfile64
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_zipimport
test_zlib
297 tests OK.
25 tests skipped:
test_bsddb test_bsddb3 test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_gdbm test_largefile test_locale test_normalization
test_ossaudiodev test_pep277 test_socket_ssl test_socketserver
test_ssl test_startfile test_timeout test_urllib2net
test_urllibnet test_winreg test_winsound test_xmlrpc_net
test_zipfile64
1 skip unexpected on darwin:
test_ssl
woot:~/subversion/python3k-svn jesse$


Python 3.0 Alpha 1 released: You want it because it’s cool.

It doth seem that per the main python.org site, Python 3.0 (aka Python 3000, aka "WHERE IS REDUCE?", aka "There's a GIL?") is out and available! Hoorah!

If you want to know what's going on hit up the newly formatted "What's New?" docs (they use the new-style docs) and read up.

For code conversion - you're going to need the 2to3 tool from subversion linked off the py3k page.

Test, test and re-test! File bugs! This is an alpha - the final is slated for August 2008 (per the page) so remember, no warranty is implied here.

SecMalloc: c-lib to prevent malloc’ed memory from swapping.

An acquaintance of mine has released a new C library called secmalloc to quote:

Most modern systems have some notion of swap, where the contents of memory can be written to disk, freeing up the memory for other purposes. This allows the system a lot of flexibility in managing its memory. Infrequently used data is a prime candidate for swapping to disk, thus freeing up the real memory for more useful purposes. This can be a problem when using cryptography as there is a danger of keys or other sensitive data ending up in swap where (eventually) it may fall into the wrong hands. Secmalloc provides a secure version of the common 'malloc' interface for managing memory. All memory allocated by secmalloc is locked, so that it cannot be swapped out.

Him and I have been conspiring about a way of being able to leverage this in cPython, there's been talk of swig and other things - I pinged Brett last night to get his thoughts (based on his other security work). Take a look at it - it's fairly specialized, but if you've been looking for something more surgical than calling:

[root@lolz~]# /sbin/swapoff

For preventing protected things from swapping off, then this is for you.

Good Reading: “Why I am a Python Advocate”

It's crossed reddit a few times - a blog post by $ entitled "Why I am a Python Advocate". I thought it was a good read, and I obviously concur with many of the points made (but I'm dangerously close to being a DLW (dynamic language weenie)).

Python Sprint at google.

Just sharing the post from google-code-updates about the Python sprint at Goooogle. Last week. Between this, and the GTAC (google test automation conference) - lots of good stuff seeping from mountain view.

Also, more pics from GvR

Python: Does it scale?

And no, I'm not talking about threads, concurrency or inter-process messaging. I'm talking about teams, products and organizations.

For example, a friend and coworker of mine looked at me a few days (weeks? I have no sense of time anymore) and asked me the simple question "Do you think you could build the product we have today, in python?".

This stems from a conversation (as usual) about threads/types/static typing and social enforcement. It's interesting for a few reasons - one, we're not talking about a web app. We're also not talking about a GUI or direct-to-end-user application per-se. We're talking about a highly scalable, distributed archiving system.

A large scale, highly distributed storage system with one goal: it can never, ever loose data. So the better question is: Is Python, as a language, appropriate for distributed/fault tolerant mission-critical-zero-data-loss systems?

I like to think that as a language (and platform) it can be used for bigger and better things. Especially with things like stackless, and jython becoming more and more mature (never just focus on cPython).

I know of (what I think) are the "top three" projects:¹

• EVE Online (video game, uses stackless)
• YouTube (web application, potentially large and complex backend)
• ITA (flights/ticketing system)
• Tabblo

What other "large scale" systems do you know of? If facing down the possibility of someone handing you a few million dollars for you're next big idea - the next big disruptive technology - would you "bet the farm" on Python?

In the end: it's a question of technical merit. Does Python, as a language lend itself to the problem domain? What, if any, drawbacks would it have in that domain? Are there third-party add ons and projects which could help you?

I think - if I get the choice - the next big project I want to work on will be done in Python (should it fit the problem domain). I'd like to see my (language of choice) scale with my own eyes.

Maybe one day we'll see a python implementation of the mighty Google Filesystem, or BigMap - although in the latter case, python bindings to a c-library might be appropriate.

1. More here Python::Success [↩]

More on Typing

Another interesting article on Types and Typing: "Stop Sitting On The Type Fence" - it makes several good points about language evolution, and about both the static and dynamic camps. I did love this quote, as it pointed out the link between mathematical thinking and static types.

So static typing isn't just about making sure that you don't add two credit card numbers together, it's about expressing code as close as possibly to mathematics. Mathematicians tend to be very good at reducing problems to their smallest representations, and coming up with very general solutions, so there can be no doubt that this approach will (and already has) produced amazing results.

p.s.: I still don't like lambdas: write a function!

Miscellaneous bits of Python (types, bugs, etc)

Miscellaneous things from around the python world -

• "Paddy3118" replied to my wikipedia duck-typing post with a post of his own - Duck Typing Explained, attempting a further generic explanation of duck-typing. Check it out.
• Python has moved away from the sourceforge bug tracker to roundup (yay!) per Brett Cannon - the new tracker is so much easier to use, and is located bugs.python.org.
• Brett Cannon's importlib (import in python) work is proceeding and looking like it's on track for py3k - you should check out his posts on it, and the SVN tree.
• I found this (dated) page and marked it as followup, Python anti-pitfalls - it's from 2003, but points out some of things I love about the language.

Java needs an overhaul.

This just crossed reddit - Java needs an overhaul - having spent the last few weeks "drinking the kool-aide" so to speak, I can see (and comisserate) with some of the things he's saying.

The standard lib is not, in fact, standard. There's an amazing amount of schizophrenia with the libraries in the stdlib, from style, to interface, documentation and general usefulness. Remember, I am not a Java expert by any stretch - I'm just someone coming into the playground and wondering why everything smells funny.

There are certain things in the language which I really do like (say, interfaces) - but then you have pieces that really feel odd, like the new generic syntax. The overall verbosity of the language (say, versus Python) is, well, frustrating¹.

This quote from the article rang a bell for me:

People have been tinkering with Java for years now, and there's still no hope in sight. There's something about the Java culture which just seems to encourage obtuse solutions over simplicity. As a Java developer, I was always so amazed at how difficult it was to use the standard Java Class Libraries for day-to-day tasks. Every app out there ends up having to include 20MB of .jars in order to get even the simplest functionality working because Java libraries are so low-level and incomplete. Having to write a loop to pull in bytes and assemble them into a StringBuffer myself if I want to get a web page is not only a pain, it's also incredibly short of implementing the HTTP standard compared to something like libcurl. Why is this sort of basic functionality missing from Java's standard install?

Java, like all languages is just another tool for your toolbox, and a useful one to boot - not to mention (as pointed out in the article) Java is everywhere. I'm enjoying learning it, even if I see problems with it (hey, I see places python could improve too!).

1. Also, the seeming religious fervor that layers and layers of abstraction are used/adhered to is alien to me [↩]

Twisted, Pythonpath and Plugins

Glyph Lefkowitz of Twisted/Divmod fame has posted a fairly interesting article on the twisted plugin system as well as some interesting insight on the PYTHONPATH system the interpreter uses.

One of the links in his post points to the docs on the Twisted plugin system - another enlightening read (given I have partially been writing a plugin-based automation framework).