convore.json/groups/epio/problem-with-celery-workers/messages.json

[{"user_id": 5282, "stars": [], "topic_id": 42190, "date_created": 1310930349.650671, "message": "Looking into it - thanks Brad", "group_id": 2873, "id": 1657960}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1310929344.2443359, "message": "  File \"/mnt/bundles/port-8030/app_image/pyenv/lib/python2.7/site-packages/redis/connection.py\", line 180, in _send\n    self.connect()\n  File \"/mnt/bundles/port-8030/app_image/pyenv/lib/python2.7/site-packages/redis/connection.py\", line 127, in connect\n    raise ConnectionError(self._error_message(e))\nConnectionError: Error 111 connecting db-runner01.gb1.ep.io:9006. ECONNREFUSED.", "group_id": 2873, "id": 1657858}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1310929366.323833, "message": "Getting that error when executing celery tasks", "group_id": 2873, "id": 1657861}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1310936397.1876881, "message": "I tried turning it off and on again (re-upload) and still getting the same issue. I noticed if I run my own celeryd from the commandline (i.e. epio django celeryd) it is trying to connect on a different port (and seems to succeed) - redis://guest@db-runner01.gb1.ep.io:9627/", "group_id": 2873, "id": 1658679}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1310937213.372498, "message": "Sorry, I was being brain-dead and uploading the wrong app. Seems to have corrected it by re-uploading. Have added a check on queue function to my pingdom checks to see if it happens again...", "group_id": 2873, "id": 1658756}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1310937406.063714, "message": "Obviously, the workers are supposed to detect this, but that code isn't quite done yet (in an ideal world, of course, Redis doesn't die)", "group_id": 2873, "id": 1658785}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1310937417.6388111, "message": "explains why a re-upload fixed it", "group_id": 2873, "id": 1658789}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1310937374.602524, "message": "@bradwhittington I suspect what happened is that your redis server died while the worker was running - the workers don't detect this and just keep on trundling", "group_id": 2873, "id": 1658779}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1310937427.7926841, "message": "would have prompted the redis server to try starting again", "group_id": 2873, "id": 1658791}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1310937445.3030779, "message": "I'll go see if I can get that worker-detecting-dead-services code working", "group_id": 2873, "id": 1658794}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1310983743.1333001, "message": "Seems like there are issues with celery again. Seems to be putting onto the queue fine, but no celery workers seem to be able to connect: [2011-07-18 12:07:57,654: ERROR/MainProcess] \u001b[1;31mConsumer: Connection Error: Error 111 connecting db-runner01.gb1.ep.io:9908. Connection refused.. Trying again in 32 seconds...\u001b[0m", "group_id": 2873, "id": 1661319}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1311103932.6279581, "message": "@bradwhittington Sorry, it seems I told to forget to tell Ben to check Convore while I was away; is this still an issue?", "group_id": 2873, "id": 1674726}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311152489.9032111, "message": "thanks.", "group_id": 2873, "id": 1679464}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311152485.7429061, "message": "@andrewgodwin It disappeared with https://convore.com/epio/anybody-else-having-trouble-uploading-new-apps/", "group_id": 2873, "id": 1679463}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1311157380.501718, "message": "(name or numeric ID, either is fine)", "group_id": 2873, "id": 1679717}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1311157371.675771, "message": "@bradwhittington Hmm, alright. What was the app ID?", "group_id": 2873, "id": 1679716}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311237482.4976659, "message": "Oh, disappeared as in, came right, corrected itself.", "group_id": 2873, "id": 1687873}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1311254059.7987759, "message": "@bradwhittington Oh, excellent! That's better news that I expected.", "group_id": 2873, "id": 1688936}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311264426.2271881, "message": "@andrewgodwin looks like my celery workers are not running again on 'testfuss' and 'taskmaster' (and went down around 2 hours ago).", "group_id": 2873, "id": 1690616}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311264524.946331, "message": "If I try jog it from my side", "group_id": 2873, "id": 1690630}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311264472.219281, "message": "bradley@dev:~/src/git/lessfuss/TaskMaster/taskmaster$ epio django celeryd -- -l info\nEnter passphrase for key '/home/bradley/.ssh/id_rsa': \n\n\nep.io: Lost connection to runner; command exiting prematurely.\nConnection to upload.ep.io closed.", "group_id": 2873, "id": 1690626}, {"user_id": 29682, "stars": [], "topic_id": 42190, "date_created": 1311264718.5530679, "message": "[2011-07-21 11:46:43,725: WARNING/MainProcess] \u001b[1;33mcelery@app-8742-p8039.celery.ep.io has started.\u001b[0m\n[2011-07-21 13:41:48,454: WARNING/MainProcess] \u001b[1;33m\u001b[1;36m \n\n -------------- celery@app-8742-p8272.celery.ep.io v2.2.7\n---- **** -----\n--- * ***  * -- [Configuration]\n-- * - **** ---   . broker:      redis://guest@109.74.247.140:9070/\n- ** ----------   . loader:      djcelery.loaders.DjangoLoader\n- ** ----------   . logfile:     [stderr]@WARNING\n- ** ----------   . concurrency: 4\n- ** ----------   . events:      OFF\n- *** --- * ---   . beat:        OFF\n-- ******* ----\n--- ***** ----- [Queues]\n --------------   . celery:      exchange:celery (direct) binding:celery\n                  \n\u001b[0m\u001b[0m\u001b[0m\u001b[0m\n[2011-07-21 13:41:48,516: WARNING/MainProcess] \u001b[1;33mcelery@app-8742-p8272.celery.ep.io has started.\u001b[0m\n\n", "group_id": 2873, "id": 1690643}, {"user_id": 8740, "stars": [], "topic_id": 42190, "date_created": 1311276484.0159481, "message": "@bradwhittington Yes, more intermittent issues with the redis service, I'm afraid - they should be resolved now.", "group_id": 2873, "id": 1691711}]