Python: Does it scale?

by jesse in ,


And no, I'm not talking about threads, concurrency or inter-process messaging. I'm talking about teams, products and organizations. For example, a friend and coworker of mine looked at me a few days (weeks? I have no sense of time anymore) and asked me the simple question "Do you think you could build the product we have today, in python?".

This stems from a conversation (as usual) about threads/types/static typing and social enforcement. It's interesting for a few reasons - one, we're not talking about a web app. We're also not talking about a GUI or direct-to-end-user application per-se. We're talking about a highly scalable, distributed archiving system.

A large scale, highly distributed storage system with one goal: it can never, ever loose data. So the better question is: Is Python, as a language, appropriate for distributed/fault tolerant mission-critical-zero-data-loss systems?

I like to think that as a language (and platform) it can be used for bigger and better things. Especially with things like stackless, and jython becoming more and more mature (never just focus on cPython).

I know of (what I think) are the "top three" projects: ((More here Python::Success))

  • EVE Online (video game, uses stackless)
  • YouTube (web application, potentially large and complex backend)
  • ITA (flights/ticketing system)
  • Tabblo

What other "large scale" systems do you know of? If facing down the possibility of someone handing you a few million dollars for you're next big idea - the next big disruptive technology - would you "bet the farm" on Python?

In the end: it's a question of technical merit. Does Python, as a language lend itself to the problem domain? What, if any, drawbacks would it have in that domain? Are there third-party add ons and projects which could help you?

I think - if I get the choice - the next big project I want to work on will be done in Python (should it fit the problem domain). I'd like to see my (language of choice) scale with my own eyes.

Maybe one day we'll see a python implementation of the mighty Google Filesystem, or BigMap - although in the latter case, python bindings to a c-library might be appropriate.