convore.json/groups/django-community/anyone-have-a-strategy-for-ensuring-celery-tasks-run-after-tx-commit/messages.json

[{"user_id": 521, "stars": [], "topic_id": 41777, "date_created": 1310509571.2592239, "message": "just run celery on post save", "group_id": 81, "id": 1623806}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310508666.659584, "message": "So.. Is there a handy strategy for situations where you fire a celery task with a pk for a model instance which has been saved, but whose transaction isn't committed yet? In some cases Celery can execute the task before the tx is committed causing obvious problems.", "group_id": 81, "id": 1623734}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310514704.840194, "message": "so not sure why that's not doing the trick...", "group_id": 81, "id": 1624279}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310514484.3298321, "message": "@chrisdrackett Makes sense.. I'd been subclassing QueuedSearchIndex to post to Celery in update_object().. Do you not use QueuedSearchIndex?", "group_id": 81, "id": 1624246}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310514694.2041969, "message": "@chrisdrackett err.. sorry. re-peeked.. we're following the pattern the Celery docs called out for implementing QueuedSearchIndex.. and they already include hooking into post_save... signals.post_save.connect(self.update_object, sender=model)", "group_id": 81, "id": 1624278}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310519223.0770171, "message": "@chrisdrackett k. dug some. post_save() doesn't help me, because as far as I can tell it's fired after the save is complete, but still before the transaction may have committed (in my case the tx is per-request). Ideal for me would be some sort of alternate to post_save() that fires post-commit.. but despite some tickets/patches it doesn't look like that exists.", "group_id": 81, "id": 1624734}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310519230.531903, "message": "Is there a useful pattern for pinning things to some sort of post-commit handler?", "group_id": 81, "id": 1624741}, {"user_id": 11827, "stars": [], "topic_id": 41777, "date_created": 1310537194.6547439, "message": "I don't think there's a specific pattern for that. However, you can take control of the transaction (@transaction.commit_manually) and add one. Another option would be to add a middleware for post-view handling (implement process_response) although I'm not sure if it will be out of the standard transaction.", "group_id": 81, "id": 1626720}, {"user_id": 11827, "stars": [], "topic_id": 41777, "date_created": 1310537277.33249, "message": "But I think the simplest part would be to submit the task with a few seconds of delay (.apply_async(countdown = 10))...", "group_id": 81, "id": 1626732}, {"user_id": 257, "stars": [], "topic_id": 41777, "date_created": 1310547907.3464761, "message": "the only solution is to handle transactions manually, and send the task after commit", "group_id": 81, "id": 1627299}, {"user_id": 257, "stars": [], "topic_id": 41777, "date_created": 1310547865.5167201, "message": "related to this django issue: https://code.djangoproject.com/ticket/14051", "group_id": 81, "id": 1627298}, {"user_id": 257, "stars": [], "topic_id": 41777, "date_created": 1310547937.811518, "message": "could probably make a decorator too", "group_id": 81, "id": 1627300}, {"user_id": 25337, "stars": [], "topic_id": 41777, "date_created": 1310628930.444386, "message": "Perhaps write your own database backend, subclassing your real backend, override _commit() to call super()._commit() and fire a custom signal. You can then write handlers for that. Not quite sure exactly what you'd usefully pass in sender though.", "group_id": 81, "id": 1635540}, {"user_id": 3580, "stars": [], "topic_id": 41777, "date_created": 1310665507.6310191, "message": "thanks for all the thoughts.. I saw that django signals ticket.. definitely hoping that happens. I also have issues with being attached to post-save where if something is updated more than once (not super uncommon on a complicated create use case) I can get duplicates.. so I'm thinking I might want to build up something where I maintain a set of instances that need updating with celery and send all the tasks after commit. I suppose if nothing else I could do it in middleware that sits outside of the tx middleware.", "group_id": 81, "id": 1638920}, {"user_id": 4383, "stars": [], "topic_id": 41777, "date_created": 1310714939.6185019, "message": "Personally I would attach the tasks to a list on the request object and process them in process_response on a custom middleware.", "group_id": 81, "id": 1643627}, {"user_id": 4383, "stars": [], "topic_id": 41777, "date_created": 1310715000.4369941, "message": "and put the custom middleware after the TransactionMiddleware", "group_id": 81, "id": 1643632}, {"user_id": 4383, "stars": [], "topic_id": 41777, "date_created": 1310715015.8680551, "message": "Of course that assumes that TransactionMiddleware is the problem.", "group_id": 81, "id": 1643633}, {"user_id": 4383, "stars": [], "topic_id": 41777, "date_created": 1310715150.634186, "message": "Actually, you would need your own custom TransactionMiddleware that processes the tasks only on commit and not when there is a 500 error or something.", "group_id": 81, "id": 1643640}, {"user_id": 43845, "stars": [], "topic_id": 41777, "date_created": 1322260826.6281559, "message": "http://ask.github.com/celery/userguide/tasks.html#database-transactions", "group_id": 81, "id": 2640320}, {"user_id": 43845, "stars": [], "topic_id": 41777, "date_created": 1322261265.725565, "message": "... on the other hand, this does not solve the problem of post_save signal :-( sorry", "group_id": 81, "id": 2640366}]