convore.json/groups/celery/memory-usage/messages.json

[{"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300215576.864995, "message": "I'm using multiprocessing concurrency, so does that mean the numbers I'm seeing in ps for memory usage are including shared pages?", "group_id": 3886, "id": 359455}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300216736.265733, "message": "as far as celery is concerned, is that actually high? i'm not sure how to read it. we were hitting the web host's limits, so i expect that it is, but maybe we're just using underpowered hosting.", "group_id": 3886, "id": 359592}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300217620.771697, "message": "i guess if i wanted to save the Django overhead i'd have to make my celery tasks run outside the content of Django so i could run celery normally", "group_id": 3886, "id": 359676}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300216533.528986, "message": "(i'm still learning how to ops, sorry)", "group_id": 3886, "id": 359563}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300215639.737262, "message": "I'm also using django-celery so I guess all my django apps are preloaded into the workers. I guess that's not really an issue since they'll need to load Django and my apps for the workers to work anyway?", "group_id": 3886, "id": 359457}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300216628.548043, "message": "  PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND\n16495 pts/5    Sl+    0:11      0     1 89134 19600  0.2 python2.6 /home/gercheq/webapps/django/myproject/manage.py celeryd -B -v 2\n16516 pts/5    S+     0:39      0     1 29942 21124  0.2 python2.6 /home/gercheq/webapps/django/myproject/manage.py celeryd -B -v 2\n16517 pts/5    S+     0:46      0     1 29806 21372  0.2 python2.6 /home/gercheq/webapps/django/myproject/manage.py celeryd -B -v 2\n16523 pts/5    S+     0:00      0     1 80646 18884  0.2 python2.6 /home/gercheq/webapps/django/myproject/manage.py celeryd -B -v 2\n", "group_id": 3886, "id": 359573}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300215494.3799651, "message": "Seems like my celery workers are taking up a lot of memory. Are there some approaches I can take to address that?", "group_id": 3886, "id": 359447}, {"user_id": 257, "stars": [], "topic_id": 13231, "date_created": 1300217098.597249, "message": "if using the eventlet pool you can save the memory usage spent by the worker processes, but it's not always a magic bullet (e.g. not all libs are compatible)", "group_id": 3886, "id": 359636}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300217798.8414049, "message": "and yeah, i'm using httplib2 to do some http requests so i don't know that i can switch to eventlet concurrency right now. though i guess since it uses the underlying libraries they'll get monkeypatched...", "group_id": 3886, "id": 359693}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300216581.491955, "message": "with CELERYD_CONCURRENCY = 2, i get:", "group_id": 3886, "id": 359568}, {"user_id": 257, "stars": [], "topic_id": 13231, "date_created": 1300217019.667979, "message": "You only need to look at RSS. These are one MainProcess, one celerybeat process and two worker processes", "group_id": 3886, "id": 359617}, {"user_id": 257, "stars": [{"date_created": 1300217741.699091, "user_id": 141}], "topic_id": 13231, "date_created": 1300217036.4575319, "message": "and the usage is not high, with django you have to expect ~20MB RSS", "group_id": 3886, "id": 359620}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300217401.2310691, "message": "great, that's good to know", "group_id": 3886, "id": 359664}, {"user_id": 141, "stars": [], "topic_id": 13231, "date_created": 1300217488.155468, "message": "is there a way to tell which process is which? i guess they would need to set $0 and don't or can't", "group_id": 3886, "id": 359670}, {"user_id": 257, "stars": [{"date_created": 1300259326.763896, "user_id": 141}], "topic_id": 13231, "date_created": 1300231980.469171, "message": "yeah, install the setproctitle module, and it will :)", "group_id": 3886, "id": 361064}, {"user_id": 3748, "stars": [], "topic_id": 13231, "date_created": 1300283330.403971, "message": "there's a new kid on the async python block, \"meinheld\", based on picoev.  It's supposed to be faster than eventlet and gevent ( that's what others say not me ).  May be soon this can be as an alternative to multiprocessing, eventlet and gevent.", "group_id": 3886, "id": 365762}, {"user_id": 257, "stars": [], "topic_id": 13231, "date_created": 1300720876.0378799, "message": "@jdaguilera It seems this is just the even loop implementation, so possibly there could be a new eventlet hub using picoev", "group_id": 3886, "id": 398933}]