jessenoller.com

Secur(e/ing) Python?

I was reading over some of the new(er) comments on the Bugzilla Language comments page and one of them caught my eye:

Zope 3 indeed has zope.security (soon coming as an egg near you :). README.txt, code, svn co svn://svn.zope.org/repos/main/zope.security/trunk zope.security) - faassen

Reading that reminded me of Brett Cannon's work around securing the interpreter.

Restricted access / sand boxing is something I'd love to poke around in (one day, the amount of things I want to work on is growing) - the OLPC uses the very nice Bitfrost model for it's security "platform" and having something easily access/leveraged within the Python interpreter landscape for secure python application implementation would be great.

You can see some of Brett's additional work here:Python security paper online and Rethinking Imports

Here's a cool ASPN recipe too: Restricted "safe" eval.

I highly recommend people look at the new Zope stuff and Brett's work, I'll be freshening my python src tree dump and try compiling it one of these days.

type(Duck)’ing: On Duck vs. Static Typing

Last night before bed - I sat down and read two articles that seemed to have grown some legs - the first being "Duck Typing Done Right" and the followup, "Answers to Duck Typing Done Right" (see the reddit comment thread). Ignoring the XML/DTD/Semantic web "solution" (see: Eby: XML is not the answer) offered for "doing it right" I got spun up on what "duck typing" really is, and what it really means for people (me) as (a) programmer(s). Invariably I didn't sleep so well.

What is duck typing? I hope most people reading this know what it is - but here is what I think it is, and why I think that a lot of people miss the point. Admittedly, this is slightly rant-ish.

To quote wikipedia on Duck Typing:

In computer science, duck typing is a principle of dynamic typing in which an object's interface and attributes determine valid semantics, rather than having the language enforce rules regarding inheritance or require type casting.This allows an object to be interchangeable with any other object so long as they both implement sufficiently compatible interfaces, regardless of whether the objects have a related inheritance hierarchy. Duck typing is a feature of programming languages such as Smalltalk, Python, Ruby, JavaScript, and ColdFusion.

Duck typing is the notion that if I create an object, and I give it the attributes of a Duck - let's say, a bill, a quack, a scent and feathers - then what I am giving you when I pass this object to you is a Duck (or at least supports the Duck interface(s)). This is also known as a "gentleman's agreement" - if I tell you I am giving you something that quacks, has feathers and a bill, then you can trust I'm giving you a damned Duck (or something equally compatible).

And there's the rub. People lie. Adults lie. Java/C/static typing people would lead you to believe that this type of flexibility is "bad" and that the "social contract" of typing should be enforced by an independent third party w.r.t: the Compiler/Interpreter.

My point is if it walks like a duck, smells like a duck, looks like a duck - but it's really a dog with no Duck interface, someone needs to acknowledge reciept of a PinkSlip(object) because they broke the contract - they knew that a duck was expected and they didn't have the politeness to wrap up their dog in a duck! who cares if it's a Dog in Duck's clothing if it can Dog.quack() when it needs to?

(note, I like this definition - "Duck typing is the system wherein you wish to pass something that requires a Duck object, and you only have a Dog, you have the decency to wrap it in a Duck suit.")

This is the beauty of Duck/Dynamic typing - I don't care if an "int" is really an integer as long as it acts like one. If you can fulfill the contract (the interface) that the method/function/etc you are about to call requires, i.e: the Dog() with Duck() interface, then by god, everything should be fine!

I see this kinda like a conversation:

Me: I want a Duck man.

You: Well, I have something that's got wings, a bill and that quacks

Me: I don't know... Does it have feathers?

You: Why yes, yes it does (def feathers(self): pass)

Me: Is it really a duck?

You: It's a Dog(Duck)

Me: Close enough for government work.

Note that his just so happens to align with my very libertarian world-view: What you do in your own free time is cool, as long as when you come over to my place you're civil according to my rules in my house.

The other side of the coin is the criticism that due to the lack of the interpreter nanny-state to protect against evil lying people from slinging things around, you can not "find bugs" until runtime. (by bugs, they mean syntax, type and other runtime errors).

The irony of this is that the same individual that is stating how Java/C++ is better in this regard is saying it while they're running xUnit Tests prior to a full build. And that bring me to my next point: A sufficiently large enough code base will enforce the contractual obligations of the objects labeled Duck() through unit tests. Testing is always the contractual enforcer of Code. If you say X does Y and it doesn't really do Y - then all the type checking-compiler-time magic in the world wouldn't save you.

I come from a QA background. I love testing. With python - I don't view a lack of a compile as a problem - I view unit testing and a dose of pylint/pychecker as the compiler. Not to mention - I can prove hundred of more assumptions in a unit test framework than I can in a compiler: I can't walk AST in my sleep, it's much easier for me to make a good, solid suite of unit-level tests (or hell - do it by hand!) to prove that my function/object/etc works/impersonates/runs as desired.

Not even compiler-jockies would suggest shipping something without comprehensive tests! So far, the only person I know who would suggest it also talks to cheese sandwiches.

In the age of the Agile Methodology - i.e. "Writing a Test before Code" - why do you need need someone/something else to contractually obligate you to a contract of type()? Why can't you - as a consenting adult, working with other consenting adults be responsible enough to agree to the contract and at least test it out.

Duck/Latent Typing buys you a lot of things - eval(), more interesting (better) polymorphism, among others, and of course static typing has it's benefits! In fact, I like static interfaces and types - see my previous horn tooting about Abstract Base Classes. I like/want the ability to force the contract/type when it's really important.

What I take umbrage too - and of course, this is outside of the scope of the original article(s) I linked to is the idea that a third party (the interpreter, a new language, my mom/wife/government) needs to tell me how to infer types, manage inputs, or build interfaces.

The best type of implementation is the light one. If people want more rules to enforce the contract - then give it to them (ergo: ABCs in Py3k). Otherwise, leave those of us willing to sit down and learn to write comprehensive tests to prove the contract is fulfilled alone. Duck typing is not a lesser creature to your fascist compiler-state. It's just different, and puts more responsibility on both the user and creator of an object and an application.

(maybe we should call it "contract typing", and if you really get hung up, get over it with isinstance().)

Edit to add for the Haskell people: Haskell is on the list of learning, but learning it is of a lesser impact on the paycheck than Java/Python hackery. But, people should also look into it (if you really like functional programming).

Hello, Reddit.

Also see my followup - Schrödinger’s Type (is a namespace a box?)

For more delicious discussion/links, check these out (not everyone agrees with me!):

• PEP 3119: Introducing Abstract Base Classes
• PEP 3133: Introducing Roles
• Strong versus Weak Typing, A Conversation with Guido van Rossum part V
• Contracts in Python, A Conversation with Guido van Rossum, Part IV
• Static Typing Where Possible, Dynamic Typing When Needed: Lambda the Ultimate
• The Perils of Duck Typing, Otaku, Cedric's weblog
• What Type is my Data?, Fuzzyman
• I'm over it, Bruce Eckel
• MonkeyTyping Proposal, PEAK

Bugzilla in Python? Yes, Please.

Via various places, I've noted that there has been discussion about the possibility of implementing Bugzilla in alternative languages (:cough:python:cough:). I doubt that the core bugzilla team would pull up their roots and really switch but a python-version of bugzilla (trac doesn't count) in Django would be really, really hot.

I linked to the main wiki page up top - but here's a link to the discussion page.

The primary page lists come common (albeit silly for the most part) "cons" that python is viewed as having. I think the discussion page is more interesting personally, but in either case - who is up for doing an implementation anyway?

I would shave every hair off my body and paint myself blue for a Bugzilla/Testopia implementation in Django it's been on my "list of things to do one day" ever since I started poking at Django a year or so ago.

Lawrence Oluyede: Producer/Consumer with multitask library

Lawrence Oluyede beat me to it! As part of my "cheese shopping" I had the new multitask library as one of the ones to talk about. Lawrence beat me to it by actually implementing Producer/Consumer with it!

I've only solved that problem using Queue and Thread. I feel so behind the times.

As Lawrence points out - Generators (and the improvements to them in 2.5) are really powerful - and I think we're going to see a lot more attention paid to them as people start switching off of python2.3/2.4 and heading into 2.5.

See more on Python Generators in pep 342.

(Via unofficial planet python.)

Ten Small Python Programs

A comp.lang.python thread caught my eye (after I was done gouging them out after reading the PEP8 discussion - we lovesOurBouncyName) entitled "Ten Small Python Programs". To quote Steve Howell:

I've always thought that the best way to introduce new
programmers to Python is to show them small code
examples.When you go to the tutorial, though, you have to wade
through quite a bit of English before seeing any
Python examples.

Below is my attempt at generating ten fairly simple,
representative Python programs that expose new users
to most basic concepts, as well as the overall syntax.

It was an interesting exercise. I constrained myself
to ten lines or less, and it was pretty easy to
incorporate loops, conditionals, print, open(), lists,
tuples, dictionaries, and imported modules.

It was harder to show classes, and my ShoppingCart
class is nothing more than an encapsulation of a list,
which has dubious value (although it's the start of
something more useful).

This is a fantastic idea. This follows the same type of teaching style a fellow at work and I used to explain Python concepts to Java guys new to Python. Steve started a python wiki page here.

Cheese shopping: Modules to follow up on: Hive.

My RSS reader betrays me each week. Every day, I get notifications on new items on the python cheeseshop and each day, I "mark as flagged" (inside of NetNewsWire) for future followup.

Each week, my "flagged items" queue (and bookmark folder labeled "TO READ") gets larger and larger. Maybe 'logging about some of the ones that have caught my eye would help.

So here is an interesting pieces of cheese:

Hive:

This is a basic concurrency module that uses only dependencies available in the Python 2.5 standard library. It allows the creation of a jobfile for uses to queue work that any number of worker processes with access to the jobfile can pull from the queue and run.

Hive looks interesting (except it's 2.5 only - I can't touch it, have to keep my head in 2.4 compat mode only). Calvin's created a little module that accepts in a text file of jobs which then loops back on itself spawning the subprocesses:

 
    def worker(self, globalcallable=None):
        """Create a worker process. Optionally associate it with a global
        callable, which it will process and ignore others.
        """
        worker = subprocess.Popen(['python', __file__, '-j', self.filename], shell=False,
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)
        self._workermap.setdefault(globalcallable, []).append(worker)
        return worker.pid
 

Calvin also made Hive aware of the platform (unix vs. win32) which is nice - the one thing I do need to figure out is the format of the job file. (I think it's "functionName *args **kwargs"). Calvin parses the job and inserts it into a sqlite database, which is actually pretty smart given you can then pass the DB around to the various subprocesses.

On the Hobo Visitor Pattern.

Speaking of the Hobo:Visitor Design Pattern.

PyMOTW: os

The OS module is the target of Doug Hellmann's PMOTW. I live, eat and breath this module day to day (it's key when working on Linux stuff) - take a peek! Doug Hellmann: PyMOTW: os

Design Patterns and Python

Daniel Nouri made a post this weekend entitled "Patterns, simplicity and simple patterns" - it came at a good time. One of my current side projects (besides infant project planning, shipping stuff at work and other stuff). Is to sit down with the Design Patterns and Head First Design Patterns and implement each of the patterns in Python.

Then, I plan on back-tracking on myself and doing it again in Java.

Daniel's post however, makes some good points (brought up by Alex Martelli in "Five Easy Pieces: Simple Python Non-Patterns") - some of the Software Design patterns people have really come know "know and love" are baked into the lowest parts of the Python language itself.

From Daniel:

The most important point that Martelli makes in this paper is that in Python, some of the traditional software design patterns don't apply, because the Python programmer has facilities that other OO languages lack, the lack of which makes certain patterns necessary in other languages that seem superfluous in Python.

I've already run into this - not in design patterens per se - but in the normal "Python is not Java/Perl/C*/etc" way. For instance, the age old struggle around static interfaces. (See: Phillip J. Eby's work "Python Is Not Java" and "Java is not Python, either...")

I think implementing each of the patterns (Why can't we have Hobo Visitor as a pattern? You know, you add new methods onto the object on the fly, and then they go away and take something else with them?)

Fundamentally, some people hear "design patterns" and they run screaming, or they make a look like you just put something in their coffee. I'm personally of the opinion that they're something you just need to know in programming - even if you don't use them (or like me, use them without knowing) every day.

Oh! And I almost forgot, Ryan Tomayko's articles "Getters/Setters/Fuxors" and "The Static Method Thing" are good reads as well.

ps: Also see Allen Holub - "Why getter and setter methods are evil" and Eby's "Python Interfaces are not Java Interfaces" and Ned Batchelder "Interfaces". All in all both Python and Java have pros, cons and things that make you go ewwww... But you should always use the right tool for the right job.

Brett Cannon: Abstract Base Classes PEP accepted

Brett Cannon: Abstract Base Classes PEP accepted

ABCs are one of the cooler things (in my opinion) that are coming in Python 3k - however, I know that there's many people who have problems with the proposed implementation. (Brett points out PEP 3133 which is the Roles proposal).

Right now - I personally like the ABC approach - it's flexible (minimalist) and will not get in your way if you don't want/need it (one of Pythons biggest draws for me). Obviously we'll see how it stacks up in implementation - it could simply be that I don't know/understand enough about the alternatives to "know any better".

I'm about to seriously start digging my teeth into Java programming (while I continue on my Python path, nooch) and one of the key things the Java guys and I talk about with some regularity is the notion of static interfaces and types.

There is something nice about static types and static interfaces - you always know what you're getting and expecting. The problem with that of course, is that the guy who wrote the object to begin with has to account for every use of the object.

Of course: that's a plus in a large code base with a large team. I instinctively pull back from the concept of those handcuffs - how do you know what I need to do with that object? Why not let me change it/alter it dynamically?

I'm sure we're going to see more about ABCs in the coming months. Meanwhile - I'll continue to learn. I like doing that.

Random thought of the day: How early can you start teaching a kid Python?

(Via Brett Cannon.)