convore.json/groups/redis/auto-increment-and-lpush/messages.json

[{"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300291819.722188, "message": "problem is, I sometimes have a state where the global counter has a higher index than what is pushed into the list, e.g. if a process did the INCR command but didn't have time to push yet.", "group_id": 832, "id": 367261}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300291724.885741, "message": "I'm not sure how to do the following in Redis:", "group_id": 832, "id": 367246}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300292815.144145, "message": "errr more specifically, like you said there's no sync between the counters, meaning the value of the counter the client remembers doesn't help me at all.", "group_id": 832, "id": 367364}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300291737.064326, "message": "I have a list, to which I want to push items", "group_id": 832, "id": 367250}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300291901.4812541, "message": "Is there any way I can make sure the two stay consistent, i.e. a form of auto-incrementing list? I can't rely on list indices, by the way, since I delete parts of the list from time to time.", "group_id": 832, "id": 367267}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292693.6214099, "message": "... you have no guarantee that the counters will be in sync though, so that sucks. And you can't just SET the value you got when you incremented the first counter as that would race when you had multiple writers.", "group_id": 832, "id": 367344}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292468.6806121, "message": "So, Redis has no support for transactions, something you implicitly signed up to when you stopped using a \"real\" database. Don't worry, it's not signposted too much in the documentation.", "group_id": 832, "id": 367317}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300292705.594768, "message": "I'm not sure how option 2 solves this", "group_id": 832, "id": 367346}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300291769.7757659, "message": "I also have a separate counter, which I INCR to get a global auto-incremented key for messages that go into the list", "group_id": 832, "id": 367254}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300292767.6185081, "message": "doesn't it cause the exact same problems?", "group_id": 832, "id": 367355}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292493.0316551, "message": "I can think of a couple of tricks to get this working though.", "group_id": 832, "id": 367320}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292551.398243, "message": "1) Lock access to this list with another key (eg. http://chris-lamb.co.uk/2010/06/07/distributing-locking-python-and-redis/) - that way your reader shouldn't see an inconsistent state", "group_id": 832, "id": 367324}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292576.2703381, "message": "2) Have another global counter that you increment after you have inserted into the list. Adjust your readers to read from this one.", "group_id": 832, "id": 367326}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292940.079241, "message": "You increment the second counter after you have pushed the item to the list, so the item will be in the list. (Modulo the counter-sync issues I've mentioned above)", "group_id": 832, "id": 367378}, {"user_id": 7129, "stars": [], "topic_id": 13369, "date_created": 1300292944.1622491, "message": "I'm just trying to trade-off one race condition with another.", "group_id": 832, "id": 367379}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300293100.2064619, "message": "I think we're going to end up going with using \"WATCH\" and transactions, and just trying to INCR the timer then PUSH until we succeeed", "group_id": 832, "id": 367405}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300293119.599036, "message": "I hoped there was an atomic \"auto-increment\"-like functionality in Redis, but it doesn't look like it", "group_id": 832, "id": 367409}, {"user_id": 2588, "stars": [], "topic_id": 13369, "date_created": 1300723426.684521, "message": "Is there a reason pipelining won't solve this problem?", "group_id": 832, "id": 399371}, {"user_id": 3946, "stars": [], "topic_id": 13369, "date_created": 1300814304.3480489, "message": "Unless I'm misunderstanding either you or redis, you should be ok wrapping a MULTI/[..]/EXEC around LPUSH and INCR with a WATCH in case sane failure is possible (Though, the only failure I can imagine for push+incr is some sort of system failure..)", "group_id": 832, "id": 409225}, {"user_id": 3946, "stars": [], "topic_id": 13369, "date_created": 1300816926.6743269, "message": "You can also consider trying to sidestep the problem by using digests of the data rather than numeric indexes for uniqueness, LLEN for counts and leaning a bit more on Hashes (H* commands) for storing related datas.", "group_id": 832, "id": 409591}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300979987.102972, "message": "@sshirokov Yeah, MULTI/EXEC is what we ended up doing. I first thought it wasn't a good idea but I since became convinced that it is.", "group_id": 832, "id": 427677}, {"user_id": 3409, "stars": [], "topic_id": 13369, "date_created": 1300979934.6616099, "message": "@unbracketed I'm not sure what pipelining is. Are you talking about MULTI/EXEC commands?", "group_id": 832, "id": 427664}]