[{"user_id": 927, "stars": [], "topic_id": 11618, "date_created": 1299637690.3131349, "message": "multiprocessing: the only task I have where it matters is image scaling, which is a perfect fit for that model", "group_id": 292, "id": 301102}, {"user_id": 2875, "stars": [], "topic_id": 11618, "date_created": 1299636387.3890941, "message": "how are you guys getting it done? parallel python? multiprocessing? voodoo magic?", "group_id": 292, "id": 301020}, {"user_id": 11647, "stars": [], "topic_id": 11618, "date_created": 1299637306.0279629, "message": "The best hack around it is using an event-oriented paradigm like Tornado and Twister do. Though it doesn't really solve the problem (and python Threads still suck), it is a rather elegant workaround.", "group_id": 292, "id": 301078}, {"user_id": 5981, "stars": [], "topic_id": 11618, "date_created": 1299638737.404022, "message": "Mostly I'd been using a combination of queue, threading, AND multiprocessing modules. In almost every single case I've done anything \"parallel\" I've had to back down because of some external resource not playing nice (mostly database connections).", "group_id": 292, "id": 301219}, {"user_id": 12791, "stars": [], "topic_id": 11618, "date_created": 1299637724.868176, "message": "I've toyed with multiprocessing, but haven't done any significant multicore work in Python as of yet. I too am interested to know what others are doing.", "group_id": 292, "id": 301109}, {"user_id": 5981, "stars": [], "topic_id": 11618, "date_created": 1299638921.690527, "message": "But like I said, it's been rare that any of this nonsense yields an actual performance increase.", "group_id": 292, "id": 301223}, {"user_id": 1152, "stars": [{"date_created": 1299640243.846559, "user_id": 1874}, {"date_created": 1299645133.483407, "user_id": 1243}, {"date_created": 1299651982.954124, "user_id": 3354}], "topic_id": 11618, "date_created": 1299640005.4945309, "message": "multiprocessing. but I'm biased. I also use a lot of threads, despite everyone insisting they suck", "group_id": 292, "id": 301278}, {"user_id": 5981, "stars": [], "topic_id": 11618, "date_created": 1299638880.348242, "message": "Usually it's a game of starting threads for specific tasks like polling externals, collecting info, feeding data into a queue, and finally processing. The great thing about the queues available is that they are compatible with each other in terms of interface. It's simple to setup a queue that works with threads, or works with multiprocessing.", "group_id": 292, "id": 301222}, {"user_id": 5981, "stars": [], "topic_id": 11618, "date_created": 1299638969.131845, "message": "The added complexity is not worth the theoretical gain in my experience.", "group_id": 292, "id": 301226}, {"user_id": 1874, "stars": [], "topic_id": 11618, "date_created": 1299639711.4705999, "message": "we used multiprocessing for some heavy-weight database benchmarking (lots of concurrent jobs and the like) ... worked pretty well except for issues with the database connection driver, which would only work correctly if imported after spawning processes.", "group_id": 292, "id": 301253}, {"user_id": 1289, "stars": [], "topic_id": 11618, "date_created": 1299640156.3292589, "message": "threading/multiprocessing", "group_id": 292, "id": 301284}, {"user_id": 18972, "stars": [], "topic_id": 11618, "date_created": 1299645616.642822, "message": "c++ sadly.", "group_id": 292, "id": 301956}, {"user_id": 9229, "stars": [{"date_created": 1299717974.601027, "user_id": 4937}], "topic_id": 11618, "date_created": 1299644196.6461201, "message": "concurrent.futures", "group_id": 292, "id": 301880}, {"user_id": 11039, "stars": [{"date_created": 1299654889.119031, "user_id": 12791}], "topic_id": 11618, "date_created": 1299649437.8442781, "message": "Stackless Python if I must, but when I need/want to code for a multicore/multiprocess environment I code it in Erlang. Luckily, interfacing the two isn't insurmountable. Do the right job with the right tool, say I.", "group_id": 292, "id": 302230}, {"user_id": 218, "stars": [], "topic_id": 11618, "date_created": 1299651418.922286, "message": "http://www.youtube.com/watch?v=w0nFpwPuk8E Multicore? Pffft. Hexacore.", "group_id": 292, "id": 302278}, {"user_id": 3354, "stars": [], "topic_id": 11618, "date_created": 1299652020.256953, "message": "@jessenoller Yup, the threading library is in danger of being prematurely optimized out of existence", "group_id": 292, "id": 302302}, {"user_id": 1874, "stars": [], "topic_id": 11618, "date_created": 1299653526.134675, "message": "@SquidLord I agree, but adding another language in crunching phases of a project might be too much of a distraction for some people.", "group_id": 292, "id": 302379}, {"user_id": 1874, "stars": [], "topic_id": 11618, "date_created": 1299653449.2419209, "message": "@justinlilly I currently get to play with a 4xoctocore system :D", "group_id": 292, "id": 302377}, {"user_id": 1455, "stars": [], "topic_id": 11618, "date_created": 1299665338.211561, "message": "execnet http://codespeak.net/execnet/", "group_id": 292, "id": 303332}, {"user_id": 4356, "stars": [{"date_created": 1299686094.747637, "user_id": 12791}, {"date_created": 1299725581.8500569, "user_id": 2108}], "topic_id": 11618, "date_created": 1299670043.2070971, "message": "multiprocessing + zeromq.  let's me write data processing jobs that span multiple cores on multiple servers with relative ease.", "group_id": 292, "id": 303822}, {"user_id": 18539, "stars": [{"date_created": 1299738371.650944, "user_id": 6671}], "topic_id": 11618, "date_created": 1299673168.972023, "message": "Threads are fine as long as you give each a separate process ;-) In all seriousness though when I had to care about it the system was so large I ended up structuring it as a message-queuey thing. txAMQP, Ampoule. Works a charm.", "group_id": 292, "id": 303953}, {"user_id": 10377, "stars": [{"date_created": 1299684467.6322739, "user_id": 12404}], "topic_id": 11618, "date_created": 1299681791.0668869, "message": "another vote for multiprocessing. I have a bunch of tasks that don't depend on each other on a multicore machine--a perfect case for multiprocessing. Watching the system tap out each cpu during processing is a simple pleasure.", "group_id": 292, "id": 305015}, {"user_id": 2875, "stars": [], "topic_id": 11618, "date_created": 1299697902.0984991, "message": "My issue is getting processes to return a value to the process that spawned them. I'm looking into queues and pipes but there must be a better way?", "group_id": 292, "id": 307553}, {"user_id": 10377, "stars": [], "topic_id": 11618, "date_created": 1299699405.3735721, "message": "can you put the logic of whatever you need down with those return values into the process itself? I had a similar situation where I needed to update a database with the results of the process, so I moved the database update into the process itself instead of doing it after the processes had finished.", "group_id": 292, "id": 307708}, {"user_id": 1243, "stars": [], "topic_id": 11618, "date_created": 1299699585.4965041, "message": "@keehyun Another approach is to have the worker queue another job to handle the results, although that can be overkill for some applications.", "group_id": 292, "id": 307719}, {"user_id": 10377, "stars": [], "topic_id": 11618, "date_created": 1299699423.9371071, "message": "umm, need *done* not 'need down'.", "group_id": 292, "id": 307709}, {"user_id": 13954, "stars": [], "topic_id": 11618, "date_created": 1299700163.8724279, "message": "@gthank the whole worker queue process is beautiful. +1 on that recommendation", "group_id": 292, "id": 307746}, {"user_id": 2875, "stars": [], "topic_id": 11618, "date_created": 1299702078.5801661, "message": "@tiarno I was initially thinking of that but in my case it wouldn't work\n@gthank I ended up doing something along those lines", "group_id": 292, "id": 307991}, {"user_id": 6671, "stars": [], "topic_id": 11618, "date_created": 1299738292.5861809, "message": "IReactorProcess.spawnProcess: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorProcess.spawnProcess.html", "group_id": 292, "id": 311402}, {"user_id": 6671, "stars": [], "topic_id": 11618, "date_created": 1299738343.4331861, "message": "Currently it's unfortunately challenging to set up all the required state, and each project I work on ends up having its own ad-hoc process pool, but Ampoule might help with that: https://launchpad.net/ampoule", "group_id": 292, "id": 311407}, {"user_id": 18539, "stars": [{"date_created": 1300808401.8062539, "user_id": 20783}, {"date_created": 1302103154.1040151, "user_id": 20326}], "topic_id": 11618, "date_created": 1300761886.9560599, "message": "@voidspace Yes! Now is the time to subvert the last 10 years of CPU architecture development", "group_id": 292, "id": 404606}, {"user_id": 15977, "stars": [], "topic_id": 11618, "date_created": 1300761330.009666, "message": "and then try: from __future__ import GIL", "group_id": 292, "id": 404571}, {"user_id": 15977, "stars": [], "topic_id": 11618, "date_created": 1300761292.3490779, "message": "IronPython or Jython and threads. :-)", "group_id": 292, "id": 404566}, {"user_id": 15977, "stars": [], "topic_id": 11618, "date_created": 1300809099.6575019, "message": "@lvh you should actually try that __future__ import with jython or ironpython - see what happens.", "group_id": 292, "id": 408613}, {"user_id": 3354, "stars": [], "topic_id": 11618, "date_created": 1300832232.672641, "message": "@voidspace If it doesn't say something like \"Never gonna happen\" I'll be a monkey's uncle. Maqybe I am anyway ...", "group_id": 292, "id": 412351}, {"user_id": 13854, "stars": [], "topic_id": 11618, "date_created": 1300882424.7699709, "message": "the new multiprocessing module is very trivial especially if you are use to writing thread code. I spent a fair amount of time writing a nonblocking web server where all WSGI app code would run in another processes than the web server socket which would just communicate with the clients. Based on my experimentation the bottleneck existed if the app code was not CPU intensive (not much a point of using multicore if you don't need the CPU) since most of the CPU was pushing data from once process to another. Using PIL to resize images then the IPC (inter processor communication) was not the bottleneck. I <3 multiprocess for a couple reasons over threads:", "group_id": 292, "id": 416523}, {"user_id": 13854, "stars": [], "topic_id": 11618, "date_created": 1300883585.3260651, "message": "If you know you will need multinode not just multicore upfront then I would start hacking parallel python right away the little I played with it I found it to be fantastic.", "group_id": 292, "id": 416657}, {"user_id": 13854, "stars": [], "topic_id": 11618, "date_created": 1300882818.9394829, "message": "WSGI app code can fail(segfalt) without bringing the rest of the server with it, processes that are using excessive memory can be killed after response is generated, processes don't have to exists on a single node so it is easier to add more processes say then threads. YMMV I good luck", "group_id": 292, "id": 416575}, {"user_id": 15977, "stars": [], "topic_id": 11618, "date_created": 1301938722.32482, "message": "@holdenweb yeah, you're right... :-)", "group_id": 292, "id": 525632}]