convore.json/groups/python/using-redis-with-eventlet/messages.json

[{"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306184823.1212981, "message": "So I have been playing around with a long polling app which uses redis to do pubsub. I have been using eventlet, and it works perfectly for the long polling parts (just doing subscribes and listen in redis). I recently added a history feature, where I store all messages which are published in a list.", "group_id": 292, "id": 1157124}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306184923.540252, "message": "Unfortunately redis + eventlet for the normal requests such as LLEN or LRANGE screw up when having multiple green threads use the same redis connection. This seems to do with the threads sending requests, and then due to the eventlet scheduler the responses are read out of order, so what the response for a call becomes out of order.", "group_id": 292, "id": 1157142}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306185162.288985, "message": "(1) seems like there would be overhead some unnecessary overhead involved (although I could use a pool). I am strugglying to get (2) working. I am using gunicorn which seems to always monkey patch the socket library. Is there some method for importing redis without the monkey patched socket library? (I tried the method eventlet.patcher.original but it doesnt seem to work for me)", "group_id": 292, "id": 1157196}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306185006.0039351, "message": "There are two solutions to this that I can think of.\n1) Each green thread gets a new connection to redis.\n2) Somehow have a redis instance which uses monkey patched sockets (for subscribe/listen), and one which uses the usual blocking sockets for things like LRANGE etc.", "group_id": 292, "id": 1157169}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306187726.3359749, "message": "Ok I have found a way to do the redis calls which are blocking. I use eventlet.tpool.execute on the redis functions which allows me to block the current coroutine/greenthread until the method completes.", "group_id": 292, "id": 1157604}, {"user_id": 1175, "stars": [], "topic_id": 36498, "date_created": 1306218608.961395, "message": "You don't really want your app to block on your redis connection, do you (if I read the thread correctly)?", "group_id": 292, "id": 1162428}, {"user_id": 1175, "stars": [], "topic_id": 36498, "date_created": 1306218691.0102651, "message": "Sounds like you need a connection pool, though I haven't used eventlet so I'm just guessing", "group_id": 292, "id": 1162441}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1306232582.9582441, "message": "well i am assuming that the redis responses will be very fast.", "group_id": 292, "id": 1164729}, {"user_id": 1265, "stars": [{"date_created": 1307516338.2337689, "user_id": 11592}], "topic_id": 36498, "date_created": 1307487706.7309, "message": "Just create multiple redis connections on monkey-patched sockets. you can put them in a eventlet.queue.Queue as a simple connection pool. then `q.get().get(key)` would possibly block on the first get() until a connection is available, and on the second one for the redis I/O.", "group_id": 292, "id": 1330224}, {"user_id": 6587, "stars": [], "topic_id": 36498, "date_created": 1307516586.596833, "message": "So my previous issue was that spawning did not monkey patch the threading library, so threading.local did not work. But I like the idea of using a Queue instead, and will switch to that where necessary.", "group_id": 292, "id": 1334502}, {"user_id": 1265, "stars": [], "topic_id": 36498, "date_created": 1307559970.511812, "message": "actually my `q.get().get(key)` is a terrible example because it drops the reference to the connection. you need to be sure to re-queue the connection, even in the event of exceptions (this is a probably a pretty familiar pattern, http://pastebin.com/08xCr6fM)", "group_id": 292, "id": 1340767}, {"user_id": 21011, "stars": [], "topic_id": 36498, "date_created": 1307598996.4389689, "message": "There was an interesting discussion on the Gevent mailing list on this as well: http://groups.google.com/group/gevent/browse_thread/thread/5808ebf3ce5647e9 . Gevent uses Greenlet.", "group_id": 292, "id": 1346710}]