“A large scale, highly distributed storage system with one goal: it can never, ever loose data. So the better question is: Is Python, as a language, appropriate for distributed/fault tolerant mission-critical-zero-data-loss systems?”

Aching for Shane Hathaway to unveil Bit Mountain:

“The Bit Mountain Research Project”

- Paddy.

Not having to write code is an advantage of scaling up teams.

• I was going to mention Allmydata-tahoe, and the other like-kin, but I haven’t had a chance to dig into tahoe yet, it’s probably the closest kin to the system I’ve dealt with.
• The quote “Python lets you easily take your first effort, break it down, and re-purpose existing code for a new set of constraints.” is an excellent point.
• As for re-writing, I did not mean to insinuate “you would always have to rewrite” (in fact, I would always like the shortest path from prototype->production). I do know about the LAN vs. WAN latency issues in distributed systems though. Depending on the system you’re dealing with, hitting a disk/network bottleneck is agreeably difficult to reach though. Frequently you spend too much time elsewhere in the system-layer.
• I have had multiple experiences with “You will re-write most of the code eventually, probably several times.” and you’re dead-right.
• Is there a 6 degrees of Twisted game out there? :)

Overall, your points are very well made - also note, I am not talking about a planned or current project. I already work (day to day) on a distributed filesystem/archiving system, and have for some time. The question (rather rhetorically) I was trying to answer is the one of “could Python “scale” up to these requirements”.

Also, one of these days I am going to be able to actually sit down and write something in twisted, damnit.

Python has several advantages that you touch upon briefly but need to be repeated. It makes prototyping easy, which is a big win. You may think that it burns cash/time before you re-write it in a “proper” language, but there is nothing about such a project like this that necessitates it being written in c/c++ or Java — once you are dealing with distributed storage across a WAN boundary you will discover that managing network latency is the big bottleneck (unlike a LAN filesystem where disk latency and component optimization can become an issue.) Therefore you will be writing your prototypes in your “shipping” language, but will have more flexibility while you are building your product. The advantage of this cannot be understated for a complex problem like a large-scale distributed system. You will re-write most of the code eventually, probably several times.

If you really think hard about your problem domain, there optimization that you will need to do will be more about process and algorithm optimization than about whether or not a particular loop is running as fast as possible. For this sort of a problem you will discover that no particular language is going to offer you anything more than a 5-10% speedup in actual execution time of any particular component, so what you need to do is optimize programmer time. This is a _huge_ task.

One other point I really hate to make here is that you are going to want to look at Twisted instead of Stackless. I really love Stackless and prefer it over Twisted whenever given the choice, but there is a sizeable amount of existing code in this particular area that is already written in Twisted and you will save yourself some time by choosing that framework over Stackless.

Maybe deep down inside that’s the question/point that counts the most: what will let me get this done the fastest to “prove” out an idea/concept.

But when you’re aiming at “something big” from position 0 (let’s say, oh, a distributed filesystem) - often the time you spend prototyping can chew into your cash and limited time, and then you run the risk of find yourself having to re-implement in the “proper” domain language later (let’s say, a 60% or more “optimization” re-write in Java or C++).

The second half of that is: Does Python scale in teams (i.e: can duck typing scale)?

Now the landscape is so different. I would have leaped at Python back then if I had access to it. One of Python’s strengths I have used time and again is its ability to glue pieces together. If profiling a python app indicates that there are delays, then most likely I can get someone on staff to write the code to use the native C or ASM code to drive it more efficiently. At the scale of something like a YouTube I don’t think languages per se are the issue.