{"id":675,"date":"2012-11-05T22:31:12","date_gmt":"2012-11-05T22:31:12","guid":{"rendered":"http:\/\/tech.avant.net\/q\/?p=675"},"modified":"2019-04-30T16:12:46","modified_gmt":"2019-04-30T16:12:46","slug":"locking-and-concurrency-in-python-part-2","status":"publish","type":"post","link":"https:\/\/tech.avant.net\/q\/locking-and-concurrency-in-python-part-2\/","title":{"rendered":"locking and concurrency in python, part 2"},"content":{"rendered":"

Previously, I created a “MultiLock” class<\/a> for managing locks and lockgroups across a shared file system. Now I want to create a simple command-line utility that uses this functionality.<\/p>\n

To start, we can create a simple runone<\/strong>() function that leverages MutliLock, e.g.,<\/p>\n

def _runone(func, lockname, lockgroup, basedir, *args, **kwargs):\n    ''' run one, AND ONLY ONE, instance (respect locking)\n\n        >>> \n        >>> _runone(print, 'lock', 'locks', '.', 'hello world')\n        >>> \n    '''\n    lock = MultiLock(lockname, lockgroup, basedir)\n    if lock.acquire():\n        func(*args, **kwargs)\n        lock.release()\n<\/pre>\n
Any python function (with its *args and **kwargs) will be called if (and-only-if) the named lock was acquired. At a minimum, this guarantees that one (and only one) instance of the function can be called at a given time.<\/p>\n
To make this slightly more magic, we can wrap this as a decorator function — a decorator that accepts arguments,<\/p>\n
def runone(lockname='lock', lockgroup='.locks', basedir='.'):\n    ''' decorator with closure\n        returns a function that will run one, and only one, instance per lockgroup\n    '''\n    def wrapper(fn):\n        def new_fn(*args, **kwargs):\n            return _runone(fn, lockname, lockgroup, basedir, *args, **kwargs)\n        return new_fn\n    return wrapper\n<\/pre>\n
The closure is used so that we can pass arguments to the decorator function, e.g.,<\/p>\n
@runone('lock', 'lockgroup', '\/shared\/path')\ndef spam():\n    #do work, only if we acquire \/shared\/path\/lockgroup\/lock \n<\/pre>\n
Putting this all together, we can create a command-line utility that will execute any command-line program if (and only if) it acquires a named lock in the lockgroup. With such a utility we can add concurrency and fault-tolerance to any shell script that can be executed across all nodes in a cluster. This code is also available in this github repo<\/a>.<\/p>\n
import time, sys, subprocess, optparse, logging\nfrom multilock import MultiLock\n\ndef runone(lockname='lock', lockgroup='.locks', basedir='.'):\n    ''' decorator with closure\n        returns a function that will run one, and only one, instance per lockgroup\n    '''\n    def wrapper(fn):\n        def new_fn(*args, **kwargs):\n            return _runone(fn, lockname, lockgroup, basedir, *args, **kwargs)\n        return new_fn\n    return wrapper\n\n\ndef _runone(func, lockname, lockgroup, basedir, *args, **kwargs):\n    ''' run one, AND ONLY ONE, instance (respect locking)\n\n        >>> \n        >>> _runone(print, 'lock', 'locks', '.', 'hello world')\n        >>> \n    '''\n    lock = MultiLock(lockname, lockgroup, basedir)\n    if lock.acquire():\n        func(*args, **kwargs)\n        lock.release()\n\n\nif __name__ == '__main__':\n\n    p = optparse.OptionParser('usage: %prog [options] cmd [args]')\n    p.add_option('--lockname', '-l', dest=\"lockname\", default='lock', help=\"the lock name, should be unique for this instance\")\n    p.add_option('--lockgroup', '-g', dest=\"lockgroup\", default='.locks', help=\"the lockgroup, a collection of locks independent locks\")\n    p.add_option('--basedir', '-d', dest=\"basedir\", default='.', help=\"the base directory where the lock files should be written\")\n    p.add_option('--wait', '-w', dest=\"wait\", default=None, help=\"optional, wait (up till the number of seconds specified) for all locks to complete in the lockgroup\")\n    options, args = p.parse_args()\n\n    if options.wait:\n        lock = MultiLock(options.lockname, options.lockgroup, options.basedir)\n        lock.wait(options.wait)\n        sys.exit()\n    \n    @runone(options.lockname, options.lockgroup, options.basedir)\n    def _main():\n        subprocess.call(args)\n\n    _main() \n<\/pre>\n","protected":false},"excerpt":{"rendered":"
Previously, I created a “MultiLock” class for managing locks and lockgroups across a shared file system. Now I want to create a simple command-line utility that uses this functionality. To start, we can create a simple runone() function that leverages MutliLock, e.g., def _runone(func, lockname, lockgroup, basedir, *args, **kwargs): ”’ run one, AND ONLY ONE, […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[6,14],"tags":[],"_links":{"self":[{"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/posts\/675"}],"collection":[{"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/comments?post=675"}],"version-history":[{"count":10,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/posts\/675\/revisions"}],"predecessor-version":[{"id":954,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/posts\/675\/revisions\/954"}],"wp:attachment":[{"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/media?parent=675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/categories?post=675"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tech.avant.net\/q\/wp-json\/wp\/v2\/tags?post=675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}