convore.json/groups/pycon-2011/distributed-tasks-with-celery/messages.json

[{"user_id": 13604, "stars": [], "topic_id": 12326, "date_created": 1299856906.540179, "message": "i'm more of a backend guy. right now I'm doing a lot of hadoop.", "group_id": 373, "id": 323027}, {"user_id": 20611, "stars": [], "topic_id": 12326, "date_created": 1299857050.6714931, "message": "@eloy I was looking for the snacks too. All I heard out there was the clink of dirty dishes.", "group_id": 373, "id": 323034}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857291.8429079, "message": "s/yeha/yeah/", "group_id": 373, "id": 323070}, {"user_id": 1, "stars": [], "topic_id": 12326, "date_created": 1299857567.452579, "message": "I like \"FailWhaleError\"", "group_id": 373, "id": 323150}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299856850.301909, "message": "@mjamesruggiero do you use python for web development?", "group_id": 373, "id": 323015}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857230.691185, "message": "@mjamesruggiero or if you're ever thinking about using amqp directly, look at Celery first", "group_id": 373, "id": 323060}, {"user_id": 20291, "stars": [], "topic_id": 12326, "date_created": 1299857620.476299, "message": "@neil what are some other examples?", "group_id": 373, "id": 323166}, {"user_id": 19652, "stars": [], "topic_id": 12326, "date_created": 1299858450.02175, "message": "@rytis sorry the webapps as @bitprophet said", "group_id": 373, "id": 323431}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299856777.711509, "message": "@mjamesruggiero because celery is really useful for offloading processor intensive tasks from your main server/machine.", "group_id": 373, "id": 323008}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858337.7850139, "message": "Luckily, it's possible to turn on event notifications with celeryctl", "group_id": 373, "id": 323394}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857299.087343, "message": "@elpargo \"done right\" is debatable. Dead simple is more like it. :)", "group_id": 373, "id": 323071}, {"user_id": 1, "stars": [{"date_created": 1299858191.9318261, "user_id": 6431}, {"date_created": 1299858372.910512, "user_id": 1979}], "topic_id": 12326, "date_created": 1299858182.4018581, "message": "I love CELERY_ALWAYS_EAGER for testing", "group_id": 373, "id": 323340}, {"user_id": 560, "stars": [], "topic_id": 12326, "date_created": 1299856895.0391331, "message": "What SSID are you guys using for Internet?", "group_id": 373, "id": 323022}, {"user_id": 560, "stars": [], "topic_id": 12326, "date_created": 1299857533.2579, "message": "Nice to see eventlet suport", "group_id": 373, "id": 323137}, {"user_id": 9153, "stars": [], "topic_id": 12326, "date_created": 1299858548.0715051, "message": "Ahhh .... iPad .. Slides?", "group_id": 373, "id": 323472}, {"user_id": 13604, "stars": [], "topic_id": 12326, "date_created": 1299856736.1744721, "message": "so why should I learn about Celery (as opposed to unit testing w/ mock, where I am now)?", "group_id": 373, "id": 323002}, {"user_id": 19652, "stars": [], "topic_id": 12326, "date_created": 1299857267.761735, "message": "So celery is basically  MQ done right?", "group_id": 373, "id": 323065}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857611.6265471, "message": "@dgouldin true, I usually use .delay() just because it's shorter and doesn't have an ugly underscore. :)", "group_id": 373, "id": 323162}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857797.8506451, "message": "@rytis they'll probably be on slideshare too, and I believe they put them up on the PyCon website last year.", "group_id": 373, "id": 323221}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299857901.6885769, "message": "Did I miss something?  I don't see how that architecture change sped up the processing time.", "group_id": 373, "id": 323258}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858837.096024, "message": "In order to distribute work over multiple nodes but still have the results back immediately.", "group_id": 373, "id": 323550}, {"user_id": 20611, "stars": [], "topic_id": 12326, "date_created": 1299857628.9563711, "message": "I hope the slides are going to be available for download later...", "group_id": 373, "id": 323172}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299857983.5883601, "message": "Old way:  ShootQ -> Calendar API -> Stale cache -> In-process calendar fetch -> Result to photographer", "group_id": 373, "id": 323288}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299858017.07919, "message": "New way: ShootQ -> Calendar API -> Stale cache -> Queue job -> calendar fetch -> result to photographer ???", "group_id": 373, "id": 323300}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299857307.8167889, "message": "@elpargo just MQ done.", "group_id": 373, "id": 323073}, {"user_id": 1979, "stars": [], "topic_id": 12326, "date_created": 1299858364.3955431, "message": "it's also a really short talk, it seems", "group_id": 373, "id": 323406}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858705.4379001, "message": "I think that's a goal.", "group_id": 373, "id": 323523}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299858926.234551, "message": "Is @gsiegman in here? I know he picked Redis over RabbitMQ for a recent project.", "group_id": 373, "id": 323577}, {"user_id": 8384, "stars": [], "topic_id": 12326, "date_created": 1299856865.624193, "message": "More importantly where are these elusive snacks?", "group_id": 373, "id": 323017}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299856880.529896, "message": "@eloy snacks are a lie :`(", "group_id": 373, "id": 323019}, {"user_id": 8384, "stars": [], "topic_id": 12326, "date_created": 1299856978.7876239, "message": "@dgouldin not even ... celery sticks?", "group_id": 373, "id": 323031}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299857336.647558, "message": "(it's one way to do it, and a good way, but there are other queue options available)", "group_id": 373, "id": 323076}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857499.433393, "message": ".delay is a simpler .apply_async", "group_id": 373, "id": 323126}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857572.2072599, "message": "@joshourisman yep, it's just sugar, \"good enough\" much of the time", "group_id": 373, "id": 323153}, {"user_id": 38, "stars": [], "topic_id": 12326, "date_created": 1299857945.3394899, "message": "@abeZaffra databases mostly (redis, any sql db via sqlalchemy or the django orm)", "group_id": 373, "id": 323276}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858006.6144199, "message": "Of course there are other ways to do it, but this is pretty Pythonic; it's one, obvious way to do the job.", "group_id": 373, "id": 323296}, {"user_id": 38, "stars": [], "topic_id": 12326, "date_created": 1299858032.8099301, "message": "AMQP is recommended though since it's more reliable", "group_id": 373, "id": 323304}, {"user_id": 6431, "stars": [{"date_created": 1299857216.6469071, "user_id": 13604}, {"date_created": 1299857234.039227, "user_id": 2024}, {"date_created": 1299858272.3249021, "user_id": 177}, {"date_created": 1299860604.7394271, "user_id": 1376}], "topic_id": 12326, "date_created": 1299857169.6614189, "message": "@mjamesruggiero if you ever want a asynchronous and/or distributed task queue and you're using python, you'll probably want to use celery ... reason I ask about web dev is it's a very common use case (Celery was originally developed for use with Django)", "group_id": 373, "id": 323052}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857339.5327139, "message": "anybody looking at this diagram right now will understand why Celery is so great once they see its api", "group_id": 373, "id": 323077}, {"user_id": 38, "stars": [], "topic_id": 12326, "date_created": 1299857345.500145, "message": "Celery doesn't have to use AMQP, it can use other messaging brokers", "group_id": 373, "id": 323080}, {"user_id": 1749, "stars": [], "topic_id": 12326, "date_created": 1299858914.1467969, "message": "@offby1 tight coupling is a much easier problem.  there's a bunch of such things out there; I might even just use ssh distribution.", "group_id": 373, "id": 323573}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857283.166419, "message": "@elpargo pretty much, yeha", "group_id": 373, "id": 323069}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857552.8777609, "message": "@dgouldin true but less semantically clear", "group_id": 373, "id": 323146}, {"user_id": 1169, "stars": [{"date_created": 1299857638.992687, "user_id": 1}], "topic_id": 12326, "date_created": 1299857622.2323411, "message": "ooh, hadn't seen web hooks in celery yet.", "group_id": 373, "id": 323167}, {"user_id": 19652, "stars": [], "topic_id": 12326, "date_created": 1299857727.603554, "message": "@Paul_Collins they will all be at the schedule app", "group_id": 373, "id": 323197}, {"user_id": 1979, "stars": [], "topic_id": 12326, "date_created": 1299858349.807951, "message": "not enough twisted to trust it's asynchronous support", "group_id": 373, "id": 323399}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299857640.805661, "message": "@donspaulding me either, contemplating potential of node + celery now", "group_id": 373, "id": 323176}, {"user_id": 20582, "stars": [], "topic_id": 12326, "date_created": 1299857770.796783, "message": "@elpargo what about those who don't have schedule app?...", "group_id": 373, "id": 323212}, {"user_id": 4156, "stars": [], "topic_id": 12326, "date_created": 1299857795.019829, "message": "@rytis think he means http://us.pycon.org/2011/schedule", "group_id": 373, "id": 323220}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299857923.6362481, "message": "@donspaulding more intelligent scheduling and allocation of resources.", "group_id": 373, "id": 323269}, {"user_id": 4156, "stars": [], "topic_id": 12326, "date_created": 1299858492.4509621, "message": ":)", "group_id": 373, "id": 323445}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299858582.2481339, "message": "wait, I do that.", "group_id": 373, "id": 323485}, {"user_id": 20611, "stars": [], "topic_id": 12326, "date_created": 1299857949.47984, "message": "@donspaulding serial processing to multi threaded", "group_id": 373, "id": 323278}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299858577.5773759, "message": "eww, logs in the DB", "group_id": 373, "id": 323484}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299861122.6798241, "message": "@jdaguilera I am pretty sure all slides get posted on the PyCon website on the schedule. If not, I am betting @ryanpetrello can email you a PDF :)", "group_id": 373, "id": 324039}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858593.9843709, "message": "If it's done asynchronously, it isn't too bad.", "group_id": 373, "id": 323491}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858668.0718341, "message": "Otherwise performance could hit the toilet.", "group_id": 373, "id": 323512}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858700.4843669, "message": "Probably.", "group_id": 373, "id": 323522}, {"user_id": 1, "stars": [], "topic_id": 12326, "date_created": 1299858847.525239, "message": "@offby1 just call the function?", "group_id": 373, "id": 323553}, {"user_id": 13954, "stars": [], "topic_id": 12326, "date_created": 1299858898.333936, "message": "He mentioned \"you probably don't want to use redis over rabbitmq\"... why is that?", "group_id": 373, "id": 323569}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858521.6832221, "message": "I'd have liked to see that.", "group_id": 373, "id": 323460}, {"user_id": 6431, "stars": [], "topic_id": 12326, "date_created": 1299858877.0017681, "message": "@offby1 you get a task handle back from .delay or .apply_async, if you call .wait() on it, it'll block until the task is done", "group_id": 373, "id": 323561}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861484.39045, "message": "@cleverdevil: There are ways, but not perfect. Last time I heard built-in simple queue replication was high on their priorities.", "group_id": 373, "id": 324115}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858816.9099891, "message": "Anyone have any insight into how to use celery for synchronous tasks?", "group_id": 373, "id": 323546}, {"user_id": 13954, "stars": [], "topic_id": 12326, "date_created": 1299858929.918865, "message": "The replication problem is a significant issue here", "group_id": 373, "id": 323580}, {"user_id": 13954, "stars": [], "topic_id": 12326, "date_created": 1299858942.8099279, "message": "Sounds like redis handles that though", "group_id": 373, "id": 323583}, {"user_id": 4156, "stars": [{"date_created": 1299858429.4301679, "user_id": 1169}], "topic_id": 12326, "date_created": 1299858418.7139039, "message": "@mishok13 A) is that sarcasm? B) it's hard to get talks exactly on time; at least he didn't run out of time!", "group_id": 373, "id": 323418}, {"user_id": 1169, "stars": [], "topic_id": 12326, "date_created": 1299858691.4414411, "message": "@offby1 does performance have to be out of the toilet prior to hitting it?", "group_id": 373, "id": 323520}, {"user_id": 9153, "stars": [], "topic_id": 12326, "date_created": 1299858527.5950849, "message": "Anyone know where they are posting these s", "group_id": 373, "id": 323462}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858660.404233, "message": "You'd probably want a separate DBMS for that, though", "group_id": 373, "id": 323508}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858691.2531011, "message": "Yeah, I'd agree there.", "group_id": 373, "id": 323519}, {"user_id": 3748, "stars": [], "topic_id": 12326, "date_created": 1299861076.736665, "message": "are the slides available somewhere now?", "group_id": 373, "id": 324032}, {"user_id": 1979, "stars": [], "topic_id": 12326, "date_created": 1299858473.700268, "message": "a) english is not my first language, so yes it is a sarcasm. it should of been funnier in russian but whatever", "group_id": 373, "id": 323441}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858695.9810939, "message": "Heh.", "group_id": 373, "id": 323521}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858875.251498, "message": "I'm thinking for doing things out-of-process.", "group_id": 373, "id": 323560}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858516.9070129, "message": "The logging aspect ... it's too bad they didn't have a good solution for it.", "group_id": 373, "id": 323456}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858896.777993, "message": "Yeah, I'm more interested in optimal queue configurations and task settings.", "group_id": 373, "id": 323568}, {"user_id": 13954, "stars": [], "topic_id": 12326, "date_created": 1299858914.993602, "message": "We use rabbitmq at our company and it seems to handle fan outs pretty well.", "group_id": 373, "id": 323575}, {"user_id": 1749, "stars": [], "topic_id": 12326, "date_created": 1299858937.314188, "message": "I think I need to play with celery to see why this syntax is a good idea.  I remain unconvinced just based on this one talk :).", "group_id": 373, "id": 323582}, {"user_id": 4748, "stars": [], "topic_id": 12326, "date_created": 1299858868.993017, "message": "That won't push the work off to another worker", "group_id": 373, "id": 323558}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299860509.7455981, "message": "@titus: What syntax are you discussing?", "group_id": 373, "id": 323929}, {"user_id": 2024, "stars": [], "topic_id": 12326, "date_created": 1299858682.3326521, "message": "Redis seems like a good choice for logging.", "group_id": 373, "id": 323516}, {"user_id": 1749, "stars": [], "topic_id": 12326, "date_created": 1299858980.5910261, "message": "The emphasis on testing is nice though, and speaks well of the project.", "group_id": 373, "id": 323599}, {"user_id": 1, "stars": [], "topic_id": 12326, "date_created": 1299858983.351429, "message": "We use Celery + Redis with Convore, but we're cool if a task gets lost here or there", "group_id": 373, "id": 323600}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299862451.80053, "message": "@asksol I looked at it a while back, haven't dived deep enough yet.", "group_id": 373, "id": 324306}, {"user_id": 3748, "stars": [], "topic_id": 12326, "date_created": 1299860813.2462389, "message": "@asksol are you at pycon?", "group_id": 373, "id": 323981}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299861618.2655411, "message": "We haven't reached out to them yet. I'll definitely put that on our list.", "group_id": 373, "id": 324136}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299861521.2538149, "message": "@asksol that's good to hear that they are away of the problem. This was a question in @ryanpetrello's talk, and we don't have a great solution at the moment.", "group_id": 373, "id": 324120}, {"user_id": 6279, "stars": [{"date_created": 1299861097.320296, "user_id": 1376}], "topic_id": 12326, "date_created": 1299860990.8164461, "message": "Yep, as @ask mentioned, there a couple of ways to define tasks.  I went with decorated functions because it was easiest to fit on a slide.", "group_id": 373, "id": 324016}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299860537.922214, "message": "you mean the task decorators, and the task class?", "group_id": 373, "id": 323936}, {"user_id": 3748, "stars": [], "topic_id": 12326, "date_created": 1299860846.507272, "message": "me too", "group_id": 373, "id": 323987}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299860837.6034589, "message": "in spirit, yes.  but not in person :(", "group_id": 373, "id": 323985}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861909.728981, "message": "see also http://www.rabbitmq.com/pacemaker.html", "group_id": 373, "id": 324175}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299862415.3550479, "message": "@cleverdevil: Ah, but did you try the HA guide? (linked above)", "group_id": 373, "id": 324294}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299860767.882854, "message": "syntax is not set in stone though, you could make up your own if you wanted to.  The core is about processing messages in various ways.", "group_id": 373, "id": 323976}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299861378.5424621, "message": "I'd really like to see RabbitMQ have replication/clustering.", "group_id": 373, "id": 324100}, {"user_id": 3748, "stars": [], "topic_id": 12326, "date_created": 1299861181.206913, "message": "@cleverdevil tks", "group_id": 373, "id": 324048}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861269.837291, "message": "but it's not so much about concurrency per se, more about responding to messages.  Lots of stuff can built on top of that", "group_id": 373, "id": 324071}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861219.4303091, "message": "e.g. I'm experimenting with implementing Actors in Celery,  ala Ruby's Nanite.  It fits quite well with what's already in Celery", "group_id": 373, "id": 324053}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299861532.271486, "message": "s/away/aware", "group_id": 373, "id": 324124}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861559.8971789, "message": "if you talk to them they I'm sure they would be happy to help", "group_id": 373, "id": 324127}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861786.7666509, "message": "There is great support for clustering, but it does not actually replicate the messages.  For that you need to run on NFS or similar afaik", "group_id": 373, "id": 324159}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299862133.905875, "message": "what are you needing the replication for?", "group_id": 373, "id": 324199}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861564.737129, "message": "-they", "group_id": 373, "id": 324129}, {"user_id": 6279, "stars": [], "topic_id": 12326, "date_created": 1299862261.3564379, "message": "I don't see my slides uploaded yet at http://us.pycon.org/2011/schedule/presentations/1/.  In the meantime, you can view them in PDF format at: http://cl.ly/440X3a2N3L2r353B140f", "group_id": 373, "id": 324245}, {"user_id": 257, "stars": [], "topic_id": 12326, "date_created": 1299861811.882849, "message": "but you could easily use multiple queues", "group_id": 373, "id": 324163}, {"user_id": 5699, "stars": [], "topic_id": 12326, "date_created": 1299862375.751538, "message": "@asksol I just don't like single points of failure :)", "group_id": 373, "id": 324279}, {"user_id": 13954, "stars": [], "topic_id": 12326, "date_created": 1299882785.687041, "message": "@ericflo any reason why you went with redis instead of rabbitmq? Any are you guys just using longpoll or communicating over any kind of comet protocol?", "group_id": 373, "id": 328199}]