jessenoller.com

(I know, it’s been a lit­tle bit since my last update — been crazy busy!)

I spend most of my time writ­ing tools and soft­ware against large num­bers of Linux servers. I’ve sank a great deal of time into mak­ing multithreaded/multi-node code. But one of the things I’ve been notic­ing recently is that I have to start track­ing the indi­vid­ual per­for­mance of nodes in the clusters.

Typ­i­cally, on a Linux sys­tem, you’d use some com­bi­na­tion of top/vmstat to track mem­ory usage and swap activ­ity over time. This doesn’t scale well unless you spend a good deal of time look­ing at the indi­vid­ual sta­tis­tics of each machine after a given run or date/time period. (And if you have to selec­tively track attrib­utes of a process on a per process basis)

When we’re talk­ing about track­ing things across 100 nodes in a given clus­ter — or more, any inter­ac­tive pro­gram (or non-network enabled appli­ca­tion with built in aggre­ga­tion) falls on it’s face.

In any case, yes, there are pack­ages out there that sup­port this type of track­ing. But that wouldn’t be fun, now would it? Not to men­tion, I need to selec­tively track sta­tis­tics on a per-process basis.

Instead, I ran across PyProcPs a few weeks ago, and never had a chance to bump back into it until the last cou­ple of days. It’s a really cool pack­age — it gives you a nice handy mod­ule that exposes the /proc/ filesys­tem to python. This allows you to attach to a given PID and track the infor­ma­tion of that PID.

For exam­ple:

[jesse@wasabi /tmp/pyprocps-0.2]# python
Python 2.4.1 (#1, May 16 2005, 15:19:29)
[GCC 4.0.0 20050512 (Red Hat 4.0.0–5)] on linux2
Type “help”, “copy­right”, “cred­its” or “license” for more infor­ma­tion.
»> import pyprocps
»> jvmpid = open(‘/var/run/guard/jvm.pid’, ‘r’).read().rstrip(‘\n’)
»> jvminfo = pyprocps.pidinfo(jvmpid)
»> jvminfo.vsize
‘311906304’
»> jvminfo.rss
‘39113’
»> jvminfo.nswap
‘0’
»> jvminfo.cnswap
‘0’
»> jvminfo.size
’76149

This is fairly neat. Remem­ber to read the docs: here. Obvi­ously, you can lever­age a tool like SAR on Linux hosts, but again, I enjoy doing things via Python 99% of the time.

Hope­fully, more to come later. I’d be inter­ested to see if set­ting up some­thing like this with PyRo or another sim­i­lar python pack­age would work well.

(I know, it’s been a lit­tle bit since my last update — been crazy busy!) I spend most of my time writ­ing tools and soft­ware against large num­bers of Linux servers....