convore.json/groups/django-community/anyone-got-any-opinions-on-how-best-to-manage-connections-to-redis-from-django/messages.json

[{"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395188.23612, "message": "urls.py", "group_id": 81, "id": 78221}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395204.454684, "message": "didn't we come up with a better solution at E?DC", "group_id": 81, "id": 78225}, {"user_id": 56, "stars": [{"date_created": 1297395258.12292, "user_id": 3019}, {"date_created": 1297398053.5587089, "user_id": 141}, {"date_created": 1297399312.5888979, "user_id": 4156}, {"date_created": 1297414995.859457, "user_id": 4893}, {"date_created": 1297416979.5689089, "user_id": 5582}, {"date_created": 1297431742.1158171, "user_id": 2087}, {"date_created": 1297435591.688782, "user_id": 960}, {"date_created": 1297440232.756762, "user_id": 521}, {"date_created": 1297459459.8899479, "user_id": 6439}, {"date_created": 1297481255.2775519, "user_id": 7381}, {"date_created": 1297492316.135473, "user_id": 1822}, {"date_created": 1297610691.5223579, "user_id": 209}], "topic_id": 4121, "date_created": 1297395242.1675129, "message": "personally I think Django needs a documented startup process", "group_id": 81, "id": 78235}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395192.6321161, "message": "it's why I don't use signals as often as I should", "group_id": 81, "id": 78223}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395261.3926129, "message": "as in \"this gets loaded, then this, then it loops through INSTALLED_APPS importing auto_import.py in turn\" - or something like that", "group_id": 81, "id": 78244}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395252.0382409, "message": "NOW I remember", "group_id": 81, "id": 78239}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395191.7850759, "message": "yeah :(", "group_id": 81, "id": 78222}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395235.611743, "message": "I think we did, I just don't know if we implemented it :D", "group_id": 81, "id": 78234}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395068.9000189, "message": "See this quora question: http://www.quora.com/What-are-the-best-practices-for-managing-connections-to-Redis-from-a-Django-app", "group_id": 81, "id": 78189}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395101.2262781, "message": "one of hte few valid places for a thread local", "group_id": 81, "id": 78201}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395121.297446, "message": "lazily open connection somewhere, close it using the signal at the end of a request", "group_id": 81, "id": 78204}, {"user_id": 56, "stars": [{"date_created": 1297416250.083854, "user_id": 1}, {"date_created": 1297431750.3219881, "user_id": 2087}], "topic_id": 4121, "date_created": 1297395125.357522, "message": "I think I'll use Eric's gist from here https://gist.github.com/609238", "group_id": 81, "id": 78206}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395186.5530181, "message": "I usually put it in urls.py, but that still feels pretty dirty", "group_id": 81, "id": 78220}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395306.8714609, "message": "where's @freakboy3742 when you need him", "group_id": 81, "id": 78261}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395175.3668511, "message": "Is there a decent place to import something so that it will only register for the request_finished handler once?", "group_id": 81, "id": 78217}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395261.773499, "message": "we were going to do it as part of logging, but then we did logging without it", "group_id": 81, "id": 78245}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395161.3734059, "message": "damn me and him are sync'd apparently", "group_id": 81, "id": 78215}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395219.101059, "message": "k, thanks - just making sure no one had invented a better solution while I wasn't paying attention", "group_id": 81, "id": 78229}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297395278.4226799, "message": "so it never happened", "group_id": 81, "id": 78252}, {"user_id": 56, "stars": [], "topic_id": 4121, "date_created": 1297395457.7535651, "message": "well, at least I'm not mad and it is still an open issue :)", "group_id": 81, "id": 78294}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297415622.336391, "message": "That gist confuses me, you don't need to store the client object in a thread local since Redis already inherits from it. Granted I'm not really sure why Redis() does that since I don't think it is valid to use the connection across threads right now.", "group_id": 81, "id": 79200}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416763.7544849, "message": "@coderanger Yeah I see nothing but good intentions, but I'd rather have something that does less (that I understand) than something that does more (that I don't understand)", "group_id": 81, "id": 79224}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416773.490665, "message": "But that's probably just being ignorant/dumb", "group_id": 81, "id": 79225}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416819.5144191, "message": "hmm, may have had too much to drink tonight :P", "group_id": 81, "id": 79227}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416366.7392399, "message": "@coderanger I have no idea what inheriting from threadlocal does and I don' trust it...I mean I've been told it's safe, but it seems really shady to me.  No idea why I feel that way.  In any case it can't hurt to put it in another threadlocal.  At least that's what I think.", "group_id": 81, "id": 79217}, {"user_id": 213, "stars": [{"date_created": 1297414543.558706, "user_id": 141}, {"date_created": 1297417022.282578, "user_id": 5582}], "topic_id": 4121, "date_created": 1297414457.6377521, "message": "@simonw that startup process idea was implemented in last year's gsoc about refactoring the app loading mechanism and is planned to be merged in the 1.4 release cycle", "group_id": 81, "id": 79174}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416295.4135039, "message": "FWIW we're using that gist for Convore in production", "group_id": 81, "id": 79214}, {"user_id": 1736, "stars": [{"date_created": 1297416722.4578891, "user_id": 1}], "topic_id": 4121, "date_created": 1297416676.1547971, "message": "@ericflo Yeah, I have a feeling Andy was trying to make it unnecessary and have it handle the connection logic internally, but from the looks of it it isn't there yet. Might be a decent perf boost to remove the local parent class from Redis and ConnectionPool since they have move overhead than a normal __dict__.", "group_id": 81, "id": 79222}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297416810.3037231, "message": "err just *me* being ignorant/dumb", "group_id": 81, "id": 79226}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297416972.5306959, "message": "On a related note, has anyone else been using miniredis for testing? We integrated it with our test runner and it has been very nice to have.", "group_id": 81, "id": 79233}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297429307.595531, "message": "@ericflo uh yeah, it's safe, django's ORM connections inherit from it", "group_id": 81, "id": 80170}, {"user_id": 4808, "stars": [], "topic_id": 4121, "date_created": 1297434658.226764, "message": "I've been using this Gist without any problems: https://gist.github.com/822419", "group_id": 81, "id": 80475}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297434877.826421, "message": "@coderanger on pypy, very soon, local objects will be as efficient as normal for attribute access, because we can hoist the get_dict() part of it, because the dict for this thread doesn't change across the execution of a loop", "group_id": 81, "id": 80483}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297463376.10361, "message": "And then ConnectionPool is also a local subclass, and thats about where my eyes glaze over", "group_id": 81, "id": 84399}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297462508.3550031, "message": "It's irrational", "group_id": 81, "id": 84334}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462961.2841091, "message": "http://www.giantbomb.com, http://www.tested.com, http://www.screened.com, http://www.comicvine.com (all the Whiskey Media sites)", "group_id": 81, "id": 84370}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297463021.5926571, "message": "@andy I guess I'm just unclear how that happens :-) Aren't all the connection params stored in the client object so wouldn't they only be visible to the thread that created it? Maybe I don't entirely understand the semantics of subclassing local.", "group_id": 81, "id": 84373}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462566.5862861, "message": ";)", "group_id": 81, "id": 84338}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462797.7225001, "message": "@coderanger so there's no weirdness in terms of requests/replies getting trampled on.", "group_id": 81, "id": 84360}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297463464.767411, "message": "@andy I just don't see how the other thread can know what host/port/db the original client was configured with", "group_id": 81, "id": 84409}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462470.976825, "message": "@ericflo The Redis connection is modeled after Django's database connections wrt using threading.local.", "group_id": 81, "id": 84330}, {"user_id": 1, "stars": [], "topic_id": 4121, "date_created": 1297462499.5816269, "message": "@andy I know, I know, I don't know why I don't trust things that subclass threading.local", "group_id": 81, "id": 84333}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462580.140142, "message": "If you have a clearer way to be thread-safe, I'm open to suggestions.", "group_id": 81, "id": 84340}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297462670.421201, "message": "@andy If you open a connection on one thread can you use it on another given the current code? Not sure that the thread-safety of redis-py is a replacement for the snippet mentioned though it does seem to overlap.", "group_id": 81, "id": 84351}, {"user_id": 723, "stars": [{"date_created": 1297462801.2377009, "user_id": 1}], "topic_id": 4121, "date_created": 1297462739.71876, "message": "@ericflo @simonw Here's how we connect/disconnect from Redis in production: http://dpaste.com/406797/", "group_id": 81, "id": 84356}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297462773.49506, "message": "@coderanger You can. The underlying tcp socket get's re-connected in the new thread.", "group_id": 81, "id": 84357}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297463459.9150989, "message": "@coderanger The idea above would at least eliminate the need for threading.local on the Client class.", "group_id": 81, "id": 84408}, {"user_id": 4219, "stars": [], "topic_id": 4121, "date_created": 1297462922.172116, "message": "@andy that's in production where? thanks for that.", "group_id": 81, "id": 84367}, {"user_id": 723, "stars": [{"date_created": 1297463494.178735, "user_id": 1736}], "topic_id": 4121, "date_created": 1297463376.309027, "message": "@coderanger @ericflo I think what I'm leaning toward is actually doing away with the cached \"current connection\" object on the client in favor of replacing it with a self.connection_pool.get_connection() type call. I *think* then that only the Connection class would need to be a threading.local... and the client could be passed around to other threads just like it is now.", "group_id": 81, "id": 84400}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297463434.720217, "message": "@coderanger If it wasn't, then those references would be held onto on separate threads... which is what we don't want.", "group_id": 81, "id": 84405}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297463200.481338, "message": "@coderanger Kind of. They're actually stored on the ConnectionPool/Connection object. The client object has the current connection object cached on it, which gets replaced by a new connection object when the client is passed to a separate thread.", "group_id": 81, "id": 84388}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297464107.8529871, "message": "@coderanger sec, looking now", "group_id": 81, "id": 84469}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297463409.388777, "message": "@coderanger Right... the pool has to be, because it's what holds on to the actual connections.", "group_id": 81, "id": 84403}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297463471.86642, "message": "Since it is all stored on local instances", "group_id": 81, "id": 84410}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297463522.083395, "message": "Something needs that data to create the new connection on the other thread", "group_id": 81, "id": 84413}, {"user_id": 723, "stars": [{"date_created": 1297465468.8346059, "user_id": 1736}], "topic_id": 4121, "date_created": 1297465103.7697129, "message": "@coderanger it's because when the Client is copied across threads, it's __init__ is called on the separate thread with the original *args, **kwargs", "group_id": 81, "id": 84504}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297465142.609473, "message": "@coderanger which is totally admit is weird/creepy/etc.", "group_id": 81, "id": 84506}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297465467.328969, "message": "Ahhh, definitely didn't know about that semantic :-) Nifty trick though, thanky!", "group_id": 81, "id": 84526}, {"user_id": 1736, "stars": [], "topic_id": 4121, "date_created": 1297465556.1450601, "message": "To quote from the docstring for others that find this later: \"This can be useful to support default values, methods and\ninitialization.  Note that if you define an __init__ method, it will be\ncalled each time the local object is used in a separate thread.  This\nis necessary to initialize each thread's dictionary.\"", "group_id": 81, "id": 84536}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297465635.8199129, "message": "aye.", "group_id": 81, "id": 84540}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297465684.8641491, "message": "And i've probably had to open/read _threading_local.py 5 times now ... basically once each time I use it :)", "group_id": 81, "id": 84544}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297465782.447922, "message": "Again, I think I want to do away with most of this stuff. I'd like to keep the client thread-safe by default, but I think it can be simplified.", "group_id": 81, "id": 84548}, {"user_id": 7, "stars": [], "topic_id": 4121, "date_created": 1297480071.436579, "message": "@andy if you want to know how threading.local works just read the pypy implementation :)", "group_id": 81, "id": 85551}, {"user_id": 723, "stars": [], "topic_id": 4121, "date_created": 1297488261.8699811, "message": "@alex heh ya it's not terribly difficult to understand, there's just a few intricacies that escape me since I rarely use it.", "group_id": 81, "id": 86266}]