Python Debugging; Embarrassment, Contracts and Nothing is private

by jesse in , , ,

Random links

(photo courtesy of Sean MacEntee via flickr)

Some interesting bits and pieces (leftovers) from the weekend - I have a tendency to queue up a pile of "read later" stuff or emailing myself a pile of things to talk about/link to/etc. Sometimes, I actually get to go through it all. Today - I get to share it to!

First up - Michael Foord (fuzzyman) did two excellent posts - the first is on privacy in Python - not big-P privacy, but rather the programming/object privacy. It's a good read because it reenforces the point that even if you think you're being clever and hiding something in a closure to make it private, you can still get access to it. Remember too - dunders (__foo) are just name-mangling. In Python, nothing is private (insert bad tasting joke about Python being facebook here).

Michael's second post is on the NamedTuple kerfluffle that was stirred up Kristjan Valur's post on the use of exec() and namedtuple (short version: namedtuple creates a string defining the class and then calls exec to create the object. I think Michael is spot on - I think exec() gets a bad rap frankly, sure - it's something of a hand-grenade, if you use it wrong, you're going to get hurt, but in this case I have to agree with Raymond in his comment on bug 3974 - the current implementation is clear and maintainable. I don't like the sternness of the reply - but he is right. Kristjan's use-case is an interesting one, but I don't think it's one that warrants the removal of the existing implementation. I'm wondering if a "fallback if exec doesn't exist" is worth inclusion.

Then again, I've spoken to people who refuse to use namedtuples because they now know how the sausage is made. I still think the sausage is delicious.

Next is a pretty interesting discussion on a presentation that came out of Pixar on getting over embarrassment in order to get things done - I don't have much to add above the comments on hacker news, except to say I think the same mental blocks they talk about for animators apply to programmers, writers, etc. We hide from code reviews, we hide our writing until we think it's "Pitch Perfect" - when in reality, we shouldn't. We should be sharing, discussing and collaborating earlier, more frequently and more often.

Share and Ship early and often - see also "Don't Go Dark".

Finally, I was happy to see the "Quick and Easy Debugging in Python" post from Jeet Sukumaran - it's always nice to evolve past sprinkling print fairy dust all over your code for debugging - even if we all still do it despite knowing or using PDB. Just to add to his post, if you want to increase your PDB-Fu, I recommend this "Debugging in Python" post by Steve Ferg, and the official documentation for PDB. We should really add a HOWTO for this.

p.s. For additional good-reading, check out "Priorities Don't Exist in a Vacuum" and "First Care"- while not germane to what's I've already written about, they're a good essays on priorities. Thanks to the latest "Back to Work" podcast.

Getting to do what you love, with people that are awesome.

by jesse in , ,

On Friday of last week, a new post I wrote for my employer (Nasuni) went up - "Encryption Keys, User Data and Subpoenas". In that post, I got to outline, in clear "non slippery" language how my employer manages encryption keys, what data they have access to, etc. One of my favorite quotes:

If a customer has provided their own encryption key(s) - Nasuni, or the cloud provider, do not have those keys, and can not provide them as part of a subpoena or other legal process. We can not decrypt or access your data. We can not supply a key which we do not have. This is not policy or trust level protection: It’s impossible.

We offer auto-generated and escrowed keys as a convenience to the user - the benefits of having this feature outweigh the cost. A user or company who knows nothing about encryption keys and key escrow can still have strong data security and instantaneous disaster recovery, they can install a Filer in minutes and immediately be up and running.

Not only was I able to work with the team to get this post up - for those of you wondering how many layers of approval, legal cleansing or other "typical filtering" it went through - the answer is "none" (or close to it). We value honesty and being forthright above just about everything else, and so posts like this - or ones in the past on "5 Weaknesses of Cloud Storage Gateways", "Cloud Storage Isn’t Cheap: How the Price of Cloud Storage Compares to Traditional Storage", "What’s the Cost of a GB in the Cloud?" which are brutally honest and frankly, would probably never see the light of day in larger companies are actually encouraged, not discouraged. Does that mean that every little thing that could be controversial goes up? Obviously not, but we, as a team, have no problems  being honest with ourselves, our users and doing what we feel is the Right Thing.

It's one of those posts that makes me exceedingly proud to be part of a team that continually surprises me. Hence the actual subject of this post. I'm one of those lucky people who get to do something I really do love - actually, multiple things - with a team that surprises me on a daily basis. Of course we're a VC backed startup which means we're smaller and more intimate then most companies many people work for, and in turn means our interpersonal relationships are that much more amplified and intensified. You don't get into bed with a startup unless you know and trust the people (or at least a few of them people) involved - trust is an fundamentally critical component - it's akin to getting married.

I know - you've probably heard stuff like this before echoing from the west coast / silicon valley area - about how startups are magical lands of unicorns and fairies and how you're going to get a bajillion dollars data mining the social relationships of some poor floor waxing guy in south dakota - but these "families" - these startups where you can have these solid relationships with those you work with, and do multiple things exist well outside the silicon valley echo chamber (for example, we're Massachusetts based). You just have to look.

Sure, there's a lot of hype in tech about startups and getting rich quick - but having worked through the dot-com bubble, I've found that I value the team (the family) and the project much more than chasing down the latest IPO. The fact is, despite being at a startup that could fail at any moment (and many do) I feel more secure in my role, and with my paycheck than I would filed away in a cube at MegaCorp. I feel more secure in that I am not just another cog in a machine, and that it is within my power to change the face and direction of the company every single day.

Considering I've worked mainly for startups since I was 18 (some of which got acquired into much bigger companies) I have a bit of confirmation bias, admittedly. Sure, I could also find potentially even more freedom and go freelance, and build relationships that way, and sure - there's tons of people very happy working for massive companies, but right now, despite having one kid, and another on the way - I'm just not that type of person. I thrive on being able to go into work and find a new challenge every day, I thrive on trying on different roles, branching out into new things. Startups - especially ones where you trust those you work with, and those people support you and work with you are an excellent place to do these things.

For example - for some time now I've been leading the User Interface project for the product - we have an embedded web-based UI that allows for the management and administration of the device we sell (a data protection/storage device). Over two years ago, I was primarily focused on concurrency and parallel programming (fast is better) on a different team (this is when I was most active on multiprocessing) - now I'm writing a web application and a lot of middleware/management stuff on a device that has a much smaller amount of RAM and cores in most deployments. Before that, I was working on what is now called the HCP (Hitachi Content Platform) which is a ultra-large, highly scalable and distributed storage system (object store) which could be construed as a parent of the current cloud storage systems (it was originally built by Archivas).

Before that? I was doing work for Allaire/Macromedia on the ColdFusion and Flex teams.

You see, these startups, each in turn, has allowed me an amount of freedom to go places and learn things I don't think I would have otherwise had the opportunity to do. Sure, it's not all roses - there are hard times, layoffs, and sometimes outright failures. You know this (or you better know this) going into it.

I mention all this because as I was sitting here sipping my morning cup of coffee (only allowed one a day) - I was reading yet another visual design site and updating my list of books on my Amazon wish list and in looking at it, it struck me how much I, as a person and I, as an employee have changed due to the opportunities and challenges given to me. I get, on one hand, to learn UI/UX (and drool at the skills of someone like Idan Gazit). I get to think about distributed storage systems, CAP theorem, blocks versus files/objects, and the "cloud" every day - but then I get to sit down and work on images, CSS, user interface, experience and think about what analogies and abstractions work best for our users.

In less than two years, I went from "oh hai concurrency" to debating shades of blue and typography with Greg Newman. How awesome is that? That's not to say that the UI I'm leading/building/hacking on is a solo effort - it's not, we're a team, nor is to to say it's perfect (I'm still learning). It's just to show that a lot can change, and when you have a team you trust, management you trust and you work for a small, agile company you really can do just about anything. Our UI has clunky elements, and things we should clean up - but that is part of what's so great - I have the opportunity to do that. I can walk into my boss' office tomorrow and simply ask him "do you mind if I change X to Y, and here's why", and if it's a sound technical argument, about two minutes later I'll be doing it.

And that change will show up on every one of our customer and trial users boxes in pretty rapid succession. Did I mention I've also gotten to learn I enjoy writing a lot more than I thought I did before, and also - marketing is actually pretty damned interesting, and not such a filthy word as most hackers and programmers think (The joke is that I'm some sort of weird programming/marketing hybrid - not sure if it's an insult or not)? Yeah, I've gotten to do writing, UI/UX, learn web programming, and help out in marketing (and even sales and market strategy to a much lesser extent). It's a rocket sled flying through a forest of cool things to do and be a part of.

Interestingly, as an aside - this change of roles/changing of focus has also changed my focus and contributions in the Python Community - being part of the PSF Board, spinning up sprints/outreach, working on explaining what the PSF is, mentorship - taking on PyCon management - all of these things reflect a change in what my life is focused on, and it's pretty awesome.

Choose a job you love, and you will never have to work a day in your life. - Confucius

Find the right team - the team you trust, that knows what they're doing and where they're going (and that trusts you). Accept that you will probably fail - meaning, you will run out of money, or not get traction (and run out of money) but build that family, that team and have blast trying your best and just accepting all the new challenges and opportunities that come up. Find people who won't not challenge you - who won't ask you to adapt to new times and technology. Accept that you're no safer at MegaCorp because they literally only view you as a human resource and not as an individual.

Most of all - Find something you love doing, and do it with awesome people.

FWIW, here are some good design reads since I'm on the topic (feel free to mention ones you love):

How can you compete with Google?

by jesse in , , ,

The official announcement (well, the addition of a website for it) of Google's URL shortening service’s new website and features on the 11th got me thinking really hard about competition, and Google. Specifically - how do you compete against the biggest technological behemoth ever seen by man? Something I’m sure is on a lot of peoples’ minds at an increasing rate.

To be honest, many of these thoughts can probably be applied to many incumbents in the tech industry (including "enterprise" software/hardware giants), Google is an easy target for these thoughts though, because they are simply so bad at some of this. This is part rant, part thought experiment – it’s entirely possible I am entirely wrong.

What drove me to thinking about this (for well over a week) is a base terror I felt about the vague possibility of being in a market Google might whimsically enter at one point. Like, say I was - and happily the most popular URL shortening and analytics firm with thousands of customers, millions of shortened links, etc, etc. How would I feel if Google coughed and suddenly entered an already tight (some would say artificial) market with all salvos aimed right at my business ( seems game)? Can an ecosystem of startups survive if Google pops into the room – can they still get VCs or Angel investors to listen and invest in them? (See also:

It's not that Google suddenly came out with a "better" thing then - Google simply came out with something which "does the job" to the technical specifications they think are superior, sitting on Google's nearly unbeatable infrastructure and then threw the weight of their brand behind it.

Does it have all the pretty analytics has? No. Does it have custom URLs? No. Does it need all of that? No, because it's made by Google. The UI is perfectly functional, but nothing to write home to mom about. Millions of people will flock to the new service and happily use it because it is Google. could very well now be on life support, and will quickly run out of oxygen when/if Google ever decided to give preference to within their sites and applications (see the security argument in the announcements – how long until the other shorteners are deemed “too insecure”?).

The very thought of this possibility happening on something I work on terrifies me. I'm pretty confident on the technical prowess of the teams I work with and of the products we make, but I'll be damned if Google couldn't wipe us out with a "product" with 25% of the features we have, simply because of who they are. Maybe we could scratch by – maybe we already have an established user base. Maybe Google would kill their implementation in a few months – who knows.

But Google has a flaw, several, in fact.

Competing with Google on a technological level is incredibly hard - it's not impossible, just hard. They have more PHDs and engineers per square foot than just about anyone. I think that breathing the Googleplex air alone probably increases your IQ. I don’t know – I have some of the air on order. It's easy for Google to build something fifty percent of the way and release it, therefore sucking the air out of the room. They don't even need to "finish" it - the very fact they've made it and put it everywhere is enough to make a market dry up and users to flock to it. It will have enough functionality - and just enough - to get the job done (“perfectly functional, albeit Spartan”).

Google is good at raw functionality and utility. They solve a problem in normally the most efficient way possible, and Google is going to probably go down as the most successful technology company in history.

Where Google fails - time and again - is being human.

No one invites Vulcans from Star Trek to come and decorate cakes or entertain them at a party. No one accuses Vulcans of having "really good empathy and customer service skills". No, people call Vulcans when they need to figure out a hard problem, or need some objective analysis. They don’t expect balloon animals and a Dora cake from them.

Google is a utility/commodity technology company (an exceedingly shrewd and powerful one) - but Android wins market share because it's on more phones, not because the experience is better but simply because it's everywhere – it’s on more and more phones every day. Plenty of the manufacturers who have adopted it spent millions designing UIs that sit on top of the default Android UI and make it "more friendly". Every market they touch they fundamentally change the economics and expectations of.

Google has become top dog for a reason - their technology. It is really top notch and their search engine and adwords system changed the market (for the better), but it all shares the cold robotic embrace of the other Google products. Their technical skills are beyond reproach, but they still lose in many cases against smaller, "richer" applications and sites because they fail at being human.

Experience Matters.

To Google; you are a statistical note - something to be tracked, categorized and profiled. Why? Not though malice or ill intent - not even slightly - rather, it is how they aim their real business at you: Advertising. Google is not malicious, nor is it evil. Google is the logical robot who will tell you you've got cancer while asking for the time and not even blink. They continue interesting projects which could change humanity – but with the bedside manner of a toaster (note though - the cold, calculating nature of the projects doesn’t diminish the value).

When a competing company's users are statistics: show those statistics love and a human face and they will follow you to the ends of the earth. Incite passion – give them a relationship. A wise man once told me "the only way you can succeed against an entrenched player is by loving your users to death".

Love your customers - say you make a code hosting service - it's hard to beat free (as is Google Code) and it's hard to beat the fact that, yeah, they have all the basic features a code host should have - but you compete where Google can't. You beat them in the User Interface department - you beat them with warm, inviting documentation and a well designed, inviting website. You beat them by hiring a support staff that actually answers emails and picks up the phone. (See also: “Google Gets a C- from the Better Business Bureau”)

You compete against them by not being a cold, Spartan feature robot. You make your thing usable, you make it pleasant. You make it so that users want to come back to you again and again because each time they do they don't feel like they just got a hug from a Craftsman workbench. You make them feel like Mom just gave them a warm hug on a Christmas day every time they use your product. Not like making out with a socket set.

But, you say, Google can make a UI, right? Not quite - Google Wave may have been the best thing in the history of earth: but no one except a few people could figure out how to really use it. Technologically, it was awesome, usability? Not so much. It was a bag of technically accurate features – but not a human interface. It was a “social network” put together by Vulcans.

The biggest thing, in my opinion, that Google has brought to the human side of the technological table is that it has helped in recent years by bringing back a wave of minimalism, simplicity of interface and speed to web application design as a whole. In the right hands, minimalism and simplicity are powerful tools. When they're not in the right hands - well, hugs from a Craftsman bench.

So - in order to compete with Google, you attack them on design – on engagement. You make your social features and good customer service into the barbs of loyalty. You pick up that phone and let them know there's someone else at the end of the line willing to hear them out at 3am when everything goes to hell and they're all alone. Even if that customer is crazy, you show them the respect they deserve as people.

Get vocal, passionate users and build a loyal community - that alone will help you succeed against Google. Make sure your customers know you love them, know that you support them and want them to succeed. Don't just enable them to do something, enable them to connect.

Build a brand against Google. Don't be content with doing something - make sure you're not just "the guys that did that thing" - or "those guys who came out with that thing". Make your name synonymous with that thing. Make it so that the first thing people think of when considering that thing is you.

Make it so that Google could come out with - say a video sharing site - tomorrow, and while it could be the best, most distributed video thing ever (the better technological choice) make it so that your users are so fiercely loyal that Google has to buy you and extinguish the flames of the passion you've incited just to get the announcement for their new thing two minutes of air time.

You can only do these things - building a brand - and building a "cult" by doing the things Google - given its robotic failings - cannot do. Love your users, infect them with your passion - not just your technical prowess or ability to scale or release new web codecs, or give them the right search results, or giving away source – infect them with your passion for what you do. Support them, respond to them - even if you're giving it away for free - after all, nothing is free.

Passion, compassion - connecting with other humans, people are always looking for a place that accepts them and makes them feel welcome. They want to get real support instead of emails that get sent to unknown voids and are never answered. Making things warm, inviting both in language and in the feel.

Just remember - Google is a fantastic, nearly unbeatable technical powerhouse. You’ve got to be fast, high quality and better where it counts the most.

“What about Don’t Be Evil?”, you say. Again, this is not an accusation of Google being evil – they’re not. They’re being coldly logical in the way humans dealing with other humans aren’t. When Eric Schmidt, the CEO, stands up on stage and talks about privacy being dead in the age we live in (the age of Facebook, and Twitter), or the refuge for criminals – he’s not “being evil”. He’s representing the coldly logical, algorithm based view of a search engine, and advertisement company. (Check out

In the age of blogs, Facebook, Twitter, MySpace and online medical records and a million other things, the logical extension is that, yes – privacy will be dead in a matter of years. Look at the train wreck the buzz rollout was – they shipped with the logical, auto-following and auto-public settings and features.

However, nuking years of email or delisting someone’s website with no human recourse is evil, and therefore, can be used as a competitive advantage against them. Be more private, be available to your customers. I know it’s expensive – but it’s how you can win. First mover advantage counts for a lot, but it doesn't count for anything if you fail your community and users.

I use lots - and I do mean lots - of Google projects. I live in the lap of Google luxury as they give me free things that have "enough" features to sate my needs and requirements. They're pretty enough - sort of like my code editor. I'm not passionate about them, they're functional utilities (albeit incredibly useful ones) - and at this point I'd probably been inoperable without a few of them. Google is a verb - JGI (Just Google it) leaves my mouth an innumerable number of times through the day. I have lots of friends who work at Google. Google has released an amazing amount of open source software, and continue to work on changing the face of the Internet, and society as a whole.

But would I say their UIs are beautiful? No. Would I ever be convinced that sending an email about my account being broken or disabled to Google's support line would be met with anything but metallic robot silence? Do I think pleas to relist my website in their index or reinstate an adwords account would be any more effective then yelling at my garbage disposal? No.

No, none of these are true. Github (despite it being git) and bitbucket are the better UIs for code hosting - Wordpress and others are better hosted blogging systems then Blogger, and so on, and so on. These services probably don't scale as well, or they can't calculate the velocity of an unladen swallow if you hit control-m-x-y-*, but they compete with Google where it counts.

Compete with Google where it hurts the most: Being Human.

Miscellanea - Python Sprints, Nasuni, etc.

by jesse in , , ,

I've obviously been quiet here on my personal blog - as everyone who reads regularly knows I'm neck-deep in a pretty exciting startup call Nasuni as well as doing other projects, like the PSF Sponsored sprints thing. That combined with twitter means my time for other additional long-form content is minimal. So here's a small roundup of interesting things:


Yup, still running Python and Django! We're actually pretty proud to be a sponsor for DjangoCon 2010 coming up in September - I'll be attending, so I hope to see all the familiar Django faces I know, and meet some new ones.

I've been blogging semi-regularly for the Nasuni blog itself - my posts are focused on product-things more than anything else. Here's a small list of posts which I've done:

  • The Road to Release - Feature Previews - this is actually my latest one, and the first in a series where I'll be showing off some of the new features we're adding in the latest release.
  • Looking at OpenStack, a Rackspace and NASA initiative - For those of you who don't know, Rackspace and NASA announced OpenStack - the awesome part? It's all python - I had the swift component (which powers Rackspace's cloudfiles system) of OpenStack running pretty quickly. I'd recommend snagging the code from launchpad and taking a look. Swift (the storage component) uses eventlet - and Nova (the compute part) uses Tornado and Twisted.
  • Storage Switzerland Test Drives the Filer - This is a response to an article written about the product - I actually used it to preview some of the work going into the next release of the Filer.
  • Thanks to Django - This piece goes into some detail about our use of Django, it's one of our ways of saying thanks. I still need to rework it so we can send it over for the Django Success Stories page.
  • Thanks to the Supporting Cast - This is an earlier thank you post - but to the other people who have helped out a ton, including Greg Newman, Lincoln Loop, and Revsys.
  • The Donut Solution - This was a fun one, mainly to show that yes - we're listening hard to customer feedback, and we're improving/iterating quickly. Also, I get to show off UI improvements.
  • Finally - The Nasuni Blog team - this is the rosetta stone for the authors of the blog, describing who we are. I didn't write this piece, but it's good reading to figure out who is who.

If you're interested in Nasuni - or cloud storage in general - I'd encourage you to sign up for the RSS feed. We're trying to keep the information useful outside of "just us" (despite my urge and predilection to churning out completely product-related posts) - and if you ever have feedback, drop us a line.

PSF Sponsored Sprints

The project continues on - we've funded two sprints so far, and have several more coming down the pike. We're always in need of volunteers to help us do things like the manuals and site maintenance/content authoring. Here's some highlights:

  • The call for applications is open - The call for applications is open - and now I suspect we won't be closing it. Originally, I thought we'd have to do things in waves of apply-approve. As time has progressed, I no longer think this is the case.
  • Montreal Python Packaging sprint wrap up - the wrap up report for our first sprint!
  • Europython core sprint report - another wrap up report for the core sprint we provided funds to.
  • Just added the locations page - we now have people/companies offering up space for sprinters! Check it out!
  • Finally - Sprints at PyOhio - PyOhio is going on this weekend, if you're in the area you should really go check it out! Catherine has gone above and beyond with the entire "become a contributor" effort going on.

Please! If you're thinking about holding a sprint - send us an application! Heck, even if we're not sponsoring it, we'll help promote you via the blog, and the sprint calendar we have up. A little fact? The sprints we've funded so far, and that are on deck for funding are all outside of the US, which is both awesome, and surprising!

PSF Board

Some of you probably know that I'm currently on the board of directors for the PSF - things progress well here, but I mainly wanted to call out the excellent blog Doug Hellmann has been authoring for PSF news. You should really be watching that because yes - we do do things, and hopefully over the next year, we'll be doing more awesome things.

I've actually got a bigger post in the works for what I think the ultimate mission of the PSF is/should be as well as "how do you get money from us" as well. Must find the time!