jessenoller.com

My RSS reader betrays me each week. Every day, I get noti­fi­ca­tions on new items on the python cheese­shop and each day, I “mark as flagged” (inside of Net­NewsWire) for future followup.

Each week, my “flagged items” queue (and book­mark folder labeled “TO READ”) gets larger and larger. Maybe ‘log­ging about some of the ones that have caught my eye would help.

So here is an inter­est­ing pieces of cheese:

Hive:

This is a basic con­cur­rency mod­ule that uses only depen­den­cies avail­able in the Python 2.5 stan­dard library. It allows the cre­ation of a job­file for uses to queue work that any num­ber of worker processes with access to the job­file can pull from the queue and run.

Hive looks inter­est­ing (except it’s 2.5 only — I can’t touch it, have to keep my head in 2.4 com­pat mode only). Calvin’s cre­ated a lit­tle mod­ule that accepts in a text file of jobs which then loops back on itself spawn­ing the subprocesses:

1
2
3
4
5
6
7
8
9
10

    def worker(self, globalcallable=None):
        """Create a worker process. Optionally associate it with a global
        callable, which it will process and ignore others.
        """
        worker = subprocess.Popen(['python', __file__, '-j', self.filename], shell=False,
            stdout=subprocess.PIPE,
            stdin=subprocess.PIPE,
            stderr=subprocess.PIPE)
        self._workermap.setdefault(globalcallable, []).append(worker)
        return worker.pid

Calvin also made Hive aware of the plat­form (unix vs. win32) which is nice — the one thing I do need to fig­ure out is the for­mat of the job file. (I think it’s “func­tion­Name *args **kwargs”). Calvin parses the job and inserts it into a sqlite data­base, which is actu­ally pretty smart given you can then pass the DB around to the var­i­ous subprocesses.

My RSS reader betrays me each week. Every day, I get notifications on new items on the python cheeseshop and each day, I "mark as flagged" (inside of NetNewsWire) for future followup. Each week, my "flagged items" queue (and bookmark folder labeled "TO READ") gets larger and larger. Maybe 'logging about some of the ones ...