convore.json/groups/django-community/setting-up-django-profilesearch-analytics/messages.json

[{"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307127470.6887219, "message": "Firstly is DB backend the best way to go with? I'm thinking of getting a NoSQL store to handle the analytics end like redis/mongo. Which is better suited?", "group_id": 81, "id": 1279834}, {"user_id": 28981, "stars": [{"date_created": 1307212067.888236, "user_id": 1081}], "topic_id": 38016, "date_created": 1307127381.865016, "message": "I want to be able to track things like (a) User appeared X number of times on search results (b) There were X searches for keyword 'Y' in Florida (b) Your profile was viewed X times (c) User was contacted X times.", "group_id": 81, "id": 1279816}, {"user_id": 1736, "stars": [], "topic_id": 38016, "date_created": 1307127935.8271251, "message": "Redis is basically built exactly for this kind of bucketed metrics.", "group_id": 81, "id": 1279904}, {"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307127613.2822311, "message": "One concern is the growing dependence on all the daemons. Solr/Tomcat for search(geo), MySQL, memcached, celery, and now a datastore. Right now i'm just running it on 2 servers(i.e. RDS(mysql) + prod( with everything else)", "group_id": 81, "id": 1279845}, {"user_id": 1736, "stars": [], "topic_id": 38016, "date_created": 1307127962.097841, "message": "Depending on volume, you could just store them in the same SQL backend though", "group_id": 81, "id": 1279909}, {"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307127987.2880721, "message": "Researching some stuff on choice on NoSQL: http://www.quora.com/MongoDB/Is-MongoDB-and-others-alike-key-value-pair-a-good-storage-backend-for-analytics-data", "group_id": 81, "id": 1279912}, {"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307129491.7728651, "message": "I was using a common model to store different types of hits with a JSON Field. But it's grown to over a million rows now, and growing. Uptil now my concern was just to log data, and not query back. But now i'm working on using that data in a more useful manner.", "group_id": 81, "id": 1280234}, {"user_id": 1127, "stars": [], "topic_id": 38016, "date_created": 1307151382.4998591, "message": "Disqus stores analytics in Redis. Assuming you have less traffic then us you should be able to do it without too much effort", "group_id": 81, "id": 1283779}, {"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307161533.7991359, "message": "@zeeg, and ofcourse i assume almost everyone here has less traffic than disqus :-p", "group_id": 81, "id": 1284547}, {"user_id": 28981, "stars": [], "topic_id": 38016, "date_created": 1307161489.8321421, "message": "@zeeg, Just out of curiosity, what made you go with Redis instead of say MongoDB. Are there advantages/disadvangates for either choice?", "group_id": 81, "id": 1284543}, {"user_id": 1127, "stars": [{"date_created": 1307173010.741055, "user_id": 1736}, {"date_created": 1307212436.33776, "user_id": 22109}], "topic_id": 38016, "date_created": 1307171988.1127219, "message": "We used MongoDB before Redis. Suffice to say, we werent satisfied (we didnt use it for anything nearly as heavy as our analytics). At this point we're widely deploying Redis for all sorts of cases. I think we have 4, maybe more, clusters of servers for it now. It powers things like our bayesian classifier, analytics, materialized views, counters, anything that can be write heavy, and the data can be rebuilt", "group_id": 81, "id": 1285027}, {"user_id": 33347, "stars": [], "topic_id": 38016, "date_created": 1307171526.043371, "message": "a little less traffic than disqus", "group_id": 81, "id": 1285016}, {"user_id": 25736, "stars": [{"date_created": 1307212170.123899, "user_id": 1081}], "topic_id": 38016, "date_created": 1307176124.400645, "message": "@zeeg, when doing analytics, do you do any type of \"shortcut\" (performance wise) not to go all the way through django just to update a counter?", "group_id": 81, "id": 1285139}, {"user_id": 1081, "stars": [], "topic_id": 38016, "date_created": 1307212210.7352769, "message": "@sidmitra Can you tell us what approach you're using? are you using any django re-usable app to help? or did you just create a model that logs everyting to redis?", "group_id": 81, "id": 1287843}, {"user_id": 1127, "stars": [{"date_created": 1307220200.546232, "user_id": 1081}], "topic_id": 38016, "date_created": 1307215221.194782, "message": "@liorsion redis.incr(foo) -- there's no Django involved at all. We have a bunch of signal-type things that just watch for changes (triggers) to keep the counters and pretty much everything else listed above in check", "group_id": 81, "id": 1288221}, {"user_id": 1081, "stars": [], "topic_id": 38016, "date_created": 1307222496.216593, "message": "@zeeg how do you then get the data from redis into django for display ... via templates, etc? as json ?", "group_id": 81, "id": 1289124}, {"user_id": 12030, "stars": [{"date_created": 1307245140.6971841, "user_id": 1081}], "topic_id": 38016, "date_created": 1307241389.7088709, "message": "This sounds like a great use for Redis. Especially the counter. @rasiel you use redis-py from Andy McCurdy. Get the data you need just like the ORM and pass it into the template or build a template tag depending on your needs.", "group_id": 81, "id": 1290862}, {"user_id": 1081, "stars": [], "topic_id": 38016, "date_created": 1307245158.661516, "message": "@chrisjones thanks for the tip, i'm about to try this", "group_id": 81, "id": 1291058}]