jessenoller.com

Since the news of PEP 3003 being released crossed the ‘web — I’ve seen more than a hand­ful of asser­tions and state­ments made that seemed to grossly miss the point.

I fig­ured I’d expound on my last post to fur­ther explain the why of the mora­to­rium and some of the other per­sonal rea­sons I stepped for­ward to help author it.

The one that really gets me is the asser­tion that the mora­to­rium was some ego-driven “python is per­fect” state­ment. That Guido/core dev think the lan­guage is “done” and that no new syn­tax or con­structs are needed.

False. Cat­e­gor­i­cally false. Quite the oppo­site is true. Most of us who are involved on python-dev and python-ideas and the var­i­ous –sig lists are actu­ally quite sure that some changes, whether they be syn­tac­tic sugar for exist­ing things, or new fun­da­men­tal con­structs con­tinue to be “needed” (“nice to have”) for Python The Language.

Still yet; many of us do think of new things to add — there is always a small wish list of “nice to have ponies” wait­ing in the wings to add to the lan­guage by a vari­ety of peo­ple. So, no, no one asserts that “the lan­guage is done”. We all know, accept and look for­ward to the con­tin­u­ing evo­lu­tion of the lan­guage over time.

How­ever, lan­guage changes have side-effects. For exam­ple, con­text man­agers intro­duced two new key­words — “with” and “as” — you should take a moment to go back through the archives to see how many peo­ple were up at arms at those two new key­words being added. Appar­ently, a lot of devel­op­ers use them as vari­able names.

To this day, I still down­load pack­ages off of PyPI which throw warn­ings about those two key­words. Which is amus­ing, because they just crash under python 2.6. It was added to __future__ in 2.5 — but became syn­tax in 2.6.

I only men­tion this to illus­trate the con­se­quence of adding new things — does this mean adding new things is bad? No. Does it mean we have to be wary/aware of the ram­i­fi­ca­tions of doing these things? Yes.

In the past 5 years, Python has gone through an accel­er­ated evo­lu­tion — dec­o­ra­tors, con­text man­agers, etc. More new syn­tax, key­words and cool things have been added to the lan­guage than nor­mally occurs in other lan­guages. This has consequences.

The first con­se­quence: Users. Users (devel­op­ers) who want to sup­port many ver­sion of python walk a tight rope. Ulti­mately, they have to write code for the low­est ver­sion (say, 2.3) and make sure their code doesn’t use any­thing (or any key­words) that will con­flict with a newer ver­sion. That’s life — I think we, as devel­op­ers, all accept that.

But the sec­ond one is more inter­est­ing. The sec­ond con­se­quence is more of a recent devel­op­ment. For many years, CPython has been the imple­men­ta­tion of Python. When you asked a per­son “hey, what python inter­preter are you run­ning” they typ­i­cally say “the one from python.org” or “the one that came with my OS” — this is always CPython.

That’s chang­ing. Take for exam­ple my sur­prise when I asked a friend “so what are you guys run­ning” and his reply was Jython. He then lamented the fact that Jython was behind the curve on syn­tax changes in the lat­est ver­sion of Python, etc, etc.

More and more we see the fact that Python The Lan­guage is more than Python The C Inter­preter. We have PyPy, Iron­Python, Jython, Unladen-Swallow (really, a branch of Cpython) and oth­ers. Sure, most of the com­mu­nity is still run­ning good-old CPython (“old reli­able”) but we’re at a point where more and more peo­ple are run­ning “those other guys”.

We’re also at a point where some of the “cool” inter­preter changes, such as JIT, free thread­ing, etc are com­ing from those other inter­preters. We face a pos­si­ble future wherein CPython is just the ref­er­ence imple­men­ta­tion of the lan­guage, but most of the world runs “one of the oth­ers”. This is not a bad thing! It’s not that the CPython inter­preter is dead — it’s sim­ply that the other inter­preters have a greater focus on “cool inter­preter stuff” which CPython has a focus on “stable”.

Adding syn­tax, key­words, and new con­structs to the CPython imple­men­ta­tion neg­a­tively impacts those imple­men­ta­tions. For the longest time, Jython was sim­ply aim­ing at Python 2.5 com­pat­i­bil­ity — I’m sure us adding a pile of new things in every ver­sion doesn’t make their lives any eas­ier. Sure, we can feel free to keep adding things in there, but with­out all of the imple­men­ta­tions being at some par­ity with one another, every time we do so, and ship it, we set them back and pos­si­bly con­fuse users.

Now con­sider the fact that in the past 5 years, we’ve done a pile of releases, adding a bevy of fea­tures to the lan­guage, the lat­est and great­est being Python 2.6. Stop and real­ize for the moment most of the world is prob­a­bly still run­ning Python 2.4 or 2.5. Rumor has it, some peo­ple are still run­ning Python 2.3. The fact is many oper­at­ing sys­tem ven­dors, and nor­mal peo­ple using python day to day, aren’t get­ting expo­sure to the lat­est and great­est things we’ve added. This means all the cool stuff we’ve added hasn’t seen the light of day — sure, some of it is “pretty minor” — but when you’re adding things to a lan­guage, you gen­er­ally want to see them get used heav­ily so you know you “hit the mark” and don’t need to go and fix the thing you just added.

It’s impor­tant that we don’t (and shouldn’t) just chuck “really cool ideas” in there and then never look back — sure, some of them might be con­structs from other lan­guages which have been proven in that lan­guage but Python isn’t that lan­guage — Python is Python, and we have to be very self-aware to the con­se­quence to Python a change might incur. It takes time, and adop­tion (see above) of those fea­tures to see how they stand up to real-world usage. Some­times, we sim­ply get an interface/API wrong, or the behav­ior isn’t as clear as we would like. It hap­pens, but just adding thing on top of thing with­out tak­ing a moment to con­sider how it worked out is bad.

Even with an exten­sive review/PEP process, even with some of the smartest peo­ple I have ever had the plea­sure of talk­ing to, mis­takes hap­pen. The only thing which allows us to see if a mis­take has been made is time.

And now we get to Python 3.

Python 3 is an inter­est­ing ani­mal — it breaks back­wards com­pat­i­bil­ity in the name of for­ward progress — some of the changes made to the lan­guage required that break to occur. Now, throw a new, back­wards incom­pat­i­ble ver­sion of Python onto the pile of things users, pack­agers, devel­op­ers, OS Ven­dors, etc all have to deal with. Remem­ber of course the point above — many peo­ple are still running/shipping python2.4/2.5. How are we going to get any run­time on the syn­tax we added in the 3.x line?

Python 3 is the ulti­mate syn­tax change — not to men­tion, to most users it’s sev­eral years off for them, given they’re reliant on third-party pack­ages that haven’t ported, their OS doesn’t ship it, etc. For all the prob­lems a “nor­mal” python release suf­fers with adop­tion — Python 3 suf­fers from them in spades.

So now, we have Python 2.6, and Python 3 — both in the adop­tion pipeline. Python 2.6 is an eas­ier pill to swal­low, Python 3 is a bit more painful as it requires much more inter­ven­tion to port to it, and main­tain a com­pat­i­ble code base.

Let’s also account for the fact that, as of this writ­ing, Python 2.7 (sched­uled for next year) is intended to be the End of Life release of the Python 2.x syn­tax — Python 3 being the next evo­lu­tion­ary step.

Python 2.7 is going to be a good release — there’s going to prob­a­bly be some Python 3 stuff pulled back into it to help port­ing to Python 3, there will be some “new” things — minor fea­tures back ported from Python 3 as well as more migra­tion helpers to assist in the 2 to 3 jump.

Notice in that last part — almost every­thing going into 2.7 is a feature/change of Python 3 we’re pulling back to help devel­op­ers and users alike. Big time fea­tures and changes are not typ­i­cally going into 2.7. In fact, there’s a grow­ing num­ber of changes which are not being pulled back into 2.7, Antoine’s recent Global Inter­preter Lock changes (also here) being amongst them.

I haven’t even men­tioned the fact that every change to the code base has to go into at least 2 branches, and in many cases 4 indi­vid­ual branches (py26-main, trunk, py3k, py31-maint). Many changes don’t have a clean-merge, given the change of syn­tax each time you move some­thing between trunk and 3k, you have a non zero chance of hav­ing to make decent changes to the patch you’re merg­ing. This causes non-zero pain for core maintainers/developers.

So, here we are — a 2.6 release still being adopted by users, OS ven­dors, etc. A new 2.x release com­ing down the pike within the next year and a back­wards incom­pat­i­ble release (Python 3) also in the pipeline.

At this point, it only makes sense to put a mora­to­rium on the lan­guage — we have to arti­fi­cially put on the brakes for the syn­tax and core lan­guage to make sure the next few releases (2.7, 3.2) don’t increase the bur­den for adopters. We have to let the other imple­men­ta­tions catch up — we have to focus on things like stdlib improvements/fixes/cleanup, we can focus on inter­preter improve­ments, tests, clean­ing out the bug list, etc.

The next few releases of Python won’t intro­duce new lan­guage and syn­tax — they will be releases focused on bug fixes, cleanup — things that make everyone’s life bet­ter. This time will serve as a reflec­tion period for the syn­tax we have, and allow us to help push along Python 3 adop­tion, maybe help­ing big­ger projects port, improve port­ing tools and guides, etc.

I’d also like to add — this mora­to­rium doesn’t apply to the stan­dard library! We can still fix things like the mime­types mod­ule, we can improve warts and fix things which are bro­ken. We can keep clean­ing the stan­dard library up — we can pol­ish all of those bat­ter­ies which help make Python, well, Python. Ulti­mately Python isn’t just it’s syn­tax — many peo­ple adopt it because of the stan­dard library (over 216 mod­ules, at last count) — and that deserve as much, maybe more, atten­tion than adding cool syn­tax to the core of the language.

So, don’t be afraid of it — don’t think that Python is evo­lu­tion­ar­ily dead — it’s not. We’re tak­ing a sta­bil­ity and adop­tion break, a breather. We’re doing this to help users and devel­op­ers, not to just be able to say “no” to every ran­dom idea sent to python-ideas, and not because we’re done.

We’re doing it for you — the devel­oper who needs sta­bil­ity, the OS ven­dor still work­ing on a new release, the language/interpreter imple­men­ta­tion group still catch­ing up to 2.6/2.7 and to all the peo­ple who would like to adopt Python 3 in the next decade.

Python’s not dead peo­ple. It’s just out­side sit­ting in the grass think­ing about what it’s going to do next.

Since the news of PEP 3003 being released crossed the 'web - I've seen more than a handful of assertions and statements made that seemed to grossly miss the point. I figured I'd expound on my last post to further explain the why of the moratorium and some of the other personal reasons I stepped ...