jessenoller.com

I’ve spent an incred­i­ble amount of time deal­ing with a few hun­dred machines — and by deal­ing, I mean clus­ter­ing, admin­is­ter­ing (simul­ta­ne­ously) QA’ing soft­ware on them, so on and so forth.

I’ve really ramped up my own expe­ri­ences in this space over the last few years too, espe­cially since I picked up the Python hat and started doing every­thing in well, Python.

One of the harder things to do when you’re inter­act­ing with greater than 5 simul­ta­ne­ous machines include flow con­trol, han­dling excep­tions prop­erly, man­ag­ing return data — etc.

In a per­fect world, you’d just make your own client and server — but why do that if you have some­thing as ubiq­ui­tous as SSH?

So, I do all my work through SSH — this is a bless­ing, and a curse in a lot of respects. The first good thing is that, well, SSH runs every­where. As long as I do the ini­tial work of get­ting SSH Keyring infor­ma­tion setup prop­erly every­place, I don’t need to deal with user authen­ti­ca­tion and can strictly scope myself to com­mand execution.

And here is where the curse comes in.

When you’re writ­ing appli­ca­tions that sub­process() or popen() on the local sys­tem, it’s triv­ial to track the pid, exit sta­tus and other infor­ma­tion of that spawned process from the local machine, gen­er­ally speak­ing you don’t have to worry about hang con­di­tions or not get­ting a “ratio­nal” exit status.

SSH doesn’t pre­vent that — not at all. How­ever, it does make life a lit­tle more inter­est­ing. You have to worry about hang con­di­tions a lot more, you have to worry how you pipe data back to the client — you have to be sure to close things like stdin() (oth­er­wise the SSH process never returns — as long as a stdin() pipe exists, SSH just chills out), etc. Then when you throw the fact you have to man­age this across a few hun­dred machines the sit­u­a­tion gets even more messy.

It’s not an impos­si­ble sit­u­a­tion — use select() to man­age the pipe from the spawned SSH process — make sure you’re cap­tur­ing the exist sta­tus of the com­mand you’re run­ning through ssh — and the exit sta­tus of the ssh com­mand itself (a process might exit 2 on the remote host, but ssh would always return an exit 1 — both are fail­ures, but you have to har­vest both) — make sure you han­dle simul­ta­ne­ous exe­cu­tion through threads — make sure those threads have timers on them, and that they can’t bring down the par­ent process, etc, etc.

Over­all, Python and SSH is a god-send for peo­ple like me who have to man­age, script and test across large farms of machines. Luck­ily, I don’t have to worry about the topol­ogy of each machine being dif­fer­ent — mine are all clones.

Using Python and SSH together — intel­li­gently can save you a ton of time, and ulti­mately make you more pro­duc­tive inter­nally, as your code should be fairly portable (and cross plat­form). Like I said — SSH is everywhere.

Hope­fully in the next few months I’ll carve off some projects I can share with the world (time is always pre­cious) that revolve around this. Until then, take a peek at these tools and utilities:

PuSSH — Pythonic, Ubiq­ui­tous SSH :
PuSSH is a rel­a­tive new­comer to the space — and has a lot of good ideas inside the code, and well, it works. I’ve peeked at the code and got­ten my hands dirty with it, hope­fully I can get more run­time with it over the next few weeks to learn more about it.

PSSH:
PSSH is, in my mind, the de-facto tool in this space. It’s flex­i­ble, and has a lot of thought and work put into it, mak­ing it incred­i­bly flex­i­ble and use­ful. It does all the cor­rect flow con­trol, and is highly read­able and flex­i­ble code.

Brent Chun (the cre­ator) also has some other tools he’s made (see the page I linked) in this space, so he knows what he’s doing. He’s also put out a lot of papers on the sub­ject of large scale installations.

The Dis­tribu­la­tor:
A type of dis­trib­uted shell — another tool in the tool­box, but when I first used it, it was on its first release, and fairly bug-ridden. I’ve used it a few times since then, but the XML for­mat of the con­fig­u­ra­tion and a few other nits made me feel like it’s a lit­tle heav­ier, and what it’s try­ing to do could be in a lighter, more effi­cient manner.

For a more recent update regard­ing pro­gram­matic SSH manip­u­la­tion with Paramiko, see this post: SSH Pro­gram­ming with Paramiko