convore.json/groups/django-community/with-rabbitmq-how-would-you-run-your-worker-processes-that-respond-to-published-messages/messages.json

[{"user_id": 209, "stars": [], "topic_id": 14026, "date_created": 1300592232.196336, "message": "@convy I've done similar things with management commands, those bootstrap the django environment nicely.  There are other methods as well", "group_id": 81, "id": 390415}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300592098.987685, "message": "is it possible for these python scripts to re-use a django apps models and other libs?", "group_id": 81, "id": 390413}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300592150.5090461, "message": "and how could you run multiple processes to respond to a high load environment? i.e. have the # of worker processes variable depending on load", "group_id": 81, "id": 390414}, {"user_id": 1736, "stars": [{"date_created": 1300694258.1937439, "user_id": 22279}], "topic_id": 14026, "date_created": 1300592434.6258831, "message": "Celery", "group_id": 81, "id": 390418}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300637559.659812, "message": "@coderanger wrong answer :)", "group_id": 81, "id": 392422}, {"user_id": 8119, "stars": [], "topic_id": 14026, "date_created": 1300637702.58762, "message": "As for responding to load, that's more server architecture/systems than Django/AMQP, IMO", "group_id": 81, "id": 392432}, {"user_id": 8119, "stars": [], "topic_id": 14026, "date_created": 1300637673.436116, "message": "If you want to re-use all the Django environment stuff you're best off with a long-running Django management command. while True: # wait for AMQP processes etc.", "group_id": 81, "id": 392429}, {"user_id": 1736, "stars": [], "topic_id": 14026, "date_created": 1300642230.1288531, "message": "@convy You want something the nicely integrates Django with RabbitMQ-backed worker processes. That describes Celery to a tee.", "group_id": 81, "id": 392871}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300717745.579951, "message": "@coderanger I see, sorry I guess I just don't understand it fully yet, I'll keep reading.", "group_id": 81, "id": 398414}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300717758.5818379, "message": "From django's side, I will simply be writing to the queue.", "group_id": 81, "id": 398417}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300717792.159034, "message": "I then need some .py scripts (with hopefully the ability to reference my django's libs and models etc) that will process the messages in a queue", "group_id": 81, "id": 398422}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300717808.7178631, "message": "and be able to have multiple .py scripts running at the same time not just one.", "group_id": 81, "id": 398425}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300719098.647562, "message": "ah ok, I thought celery was rabbitmq, but it just pulls messages on rabbitmq and processes them (read the celery site again after a long while)", "group_id": 81, "id": 398609}, {"user_id": 4219, "stars": [], "topic_id": 14026, "date_created": 1300717855.022644, "message": "e.g. say whenever someone uploads an image, I need to create thumbnails etc.  And many people are doing this at the same time, I want multiple processes to do this (this is just an example)", "group_id": 81, "id": 398433}, {"user_id": 2045, "stars": [], "topic_id": 14026, "date_created": 1300718651.2710431, "message": "@convy This is exactly what Celery provides. The built-in Django integration lets your write a tasks.py file which Celery will pick up and handle all of the serialization and routing for you. For simple things, you turn my_function(foo, bar) into my_function.delay(foo, bar).", "group_id": 81, "id": 398519}, {"user_id": 257, "stars": [], "topic_id": 14026, "date_created": 1300720192.2546539, "message": "it can also scale processes up/down based on load.  see --autoscale=max,min", "group_id": 81, "id": 398805}]