convore.json/groups/django-community/high-level-design-of-how-convore-works/messages.json

[{"user_id": 2024, "stars": [{"date_created": 1297457808.163003, "user_id": 7}, {"date_created": 1297463929.6526511, "user_id": 4156}, {"date_created": 1298010634.0620949, "user_id": 12675}], "topic_id": 4394, "date_created": 1297457763.1619811, "message": "I'm going to go ahead and guess that they aren't using MySQL...", "group_id": 81, "id": 83313}, {"user_id": 4747, "stars": [], "topic_id": 4394, "date_created": 1297457772.591383, "message": "Yea.", "group_id": 81, "id": 83317}, {"user_id": 1, "stars": [{"date_created": 1298371792.7828839, "user_id": 13597}], "topic_id": 4394, "date_created": 1297457865.109468, "message": "So someone sends a message, which goes through a pipeline that searches it for links, images, and @-references.", "group_id": 81, "id": 83334}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297457865.8677349, "message": "@josh why not?  cache is king", "group_id": 81, "id": 83335}, {"user_id": 1, "stars": [{"date_created": 1298371795.989208, "user_id": 13597}], "topic_id": 4394, "date_created": 1297457898.0175509, "message": "Oh, first it's logged", "group_id": 81, "id": 83340}, {"user_id": 1, "stars": [{"date_created": 1298371800.082989, "user_id": 13597}], "topic_id": 4394, "date_created": 1297457916.4442871, "message": "Then, once it's gone through that processing pipeline, it gets pushed into a message queue", "group_id": 81, "id": 83342}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297458186.939482, "message": "I have an idea on what it'll do, yeah, but we'll see.", "group_id": 81, "id": 83378}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297458189.718379, "message": "Thanks, @convy!", "group_id": 81, "id": 83379}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297459066.6350231, "message": "if you had to re-write it again, how long would it take you? (just you!)", "group_id": 81, "id": 83536}, {"user_id": 4219, "stars": [{"date_created": 1298192314.0216961, "user_id": 14022}], "topic_id": 4394, "date_created": 1297458168.4905469, "message": "this idea is way better than pownce! i think its a winner, really i do!", "group_id": 81, "id": 83377}, {"user_id": 1, "stars": [{"date_created": 1298371803.2141719, "user_id": 13597}], "topic_id": 4394, "date_created": 1297458047.9030261, "message": "Pretty straightforward stuff, all in all.", "group_id": 81, "id": 83362}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297458109.5841391, "message": "yeah in theory", "group_id": 81, "id": 83369}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297458935.935905, "message": "so you are still using mysql/postgres right? for groups, users, etc", "group_id": 81, "id": 83494}, {"user_id": 1, "stars": [{"date_created": 1298371794.3989129, "user_id": 13597}], "topic_id": 4394, "date_created": 1297457876.5546329, "message": "If it finds images, it will download those images", "group_id": 81, "id": 83338}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297457792.7384901, "message": "@convy At a high level that's basically how it works", "group_id": 81, "id": 83320}, {"user_id": 1, "stars": [{"date_created": 1297682431.4332869, "user_id": 1782}, {"date_created": 1297696797.1970999, "user_id": 5703}, {"date_created": 1297705429.491643, "user_id": 978}, {"date_created": 1297928466.939249, "user_id": 12817}, {"date_created": 1298371785.1616459, "user_id": 13597}], "topic_id": 4394, "date_created": 1297457973.0030949, "message": "We have celery that picks up the message, updates unread counts for everyone in the group, updates mention counts as well, and publishes it to redis pub/sub", "group_id": 81, "id": 83349}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297458140.7075901, "message": "any idea what your current server setup can handle in terms of user usage, messages etc?", "group_id": 81, "id": 83372}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297458980.2030599, "message": "@convy PostgreSQL, yep.", "group_id": 81, "id": 83510}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297459133.6723361, "message": "I have no idea :)", "group_id": 81, "id": 83553}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297457726.2073121, "message": "so what's your take onit?  i.e. I submit a reply to a question, it goes into mysql, then adds a message to a queue, the message gets popped off the queue and puts a message into redis, then tornado (or whatever eventful??) pushes the message to all users who are subscribed if there have an open ajax open", "group_id": 81, "id": 83306}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297457740.6310239, "message": "is there any truth to what I say? lol", "group_id": 81, "id": 83309}, {"user_id": 1, "stars": [{"date_created": 1297682419.8936369, "user_id": 1782}, {"date_created": 1297696797.2461369, "user_id": 5703}, {"date_created": 1297705430.7788799, "user_id": 978}, {"date_created": 1297928468.7904119, "user_id": 12817}, {"date_created": 1298371787.4895229, "user_id": 13597}], "topic_id": 4394, "date_created": 1297458022.7733469, "message": "Then we have a lightweight eventlet http server that is holding open connections to the clients, and also holding open a connection to redis pub/sub, and when it gets the message from redis, it passes it along to the user.", "group_id": 81, "id": 83355}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297457902.3560491, "message": "ok that's the message parsing ofit.", "group_id": 81, "id": 83341}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297460549.6118791, "message": "i think the only problem with this sort of site is, you can't count on getting a full nights sleep.  allot of moving parts that have to keep moving :)", "group_id": 81, "id": 83919}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297460939.138799, "message": "@coderanger how many people are managing it though?", "group_id": 81, "id": 84028}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297462225.7731099, "message": "@coderanger in eventlet (like in convore's case)", "group_id": 81, "id": 84306}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297461254.663141, "message": "Managing moving parts? Chef/puppet mean you spin up new servers easily so you don't have to worry about individual ones going down. Within the app, Django and Celery both provider a pretty amazing base for the kind of functionality you don't have to write yourself anymore", "group_id": 81, "id": 84127}, {"user_id": 548, "stars": [{"date_created": 1297462616.9796059, "user_id": 1}, {"date_created": 1297462891.2816041, "user_id": 177}, {"date_created": 1297463097.016274, "user_id": 719}, {"date_created": 1297464048.091213, "user_id": 4156}, {"date_created": 1297466878.697639, "user_id": 5749}, {"date_created": 1297467408.979352, "user_id": 1081}, {"date_created": 1297610446.714462, "user_id": 7512}, {"date_created": 1297705472.8528261, "user_id": 978}, {"date_created": 1298012657.503475, "user_id": 12126}, {"date_created": 1298030010.383672, "user_id": 12274}], "topic_id": 4394, "date_created": 1297461574.330502, "message": "I think it's pretty awesome how open & transparent @ericflo is about all this. Thumbs up!", "group_id": 81, "id": 84205}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297461860.0787809, "message": "From eventlet or from a normal WSGI app?", "group_id": 81, "id": 84275}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297460734.9326191, "message": "@convy ~5 types of servers is a lot of moving parts? /me eyes his companies status board which has >50 types listed.", "group_id": 81, "id": 83981}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297461293.596952, "message": "When writing a new background task I can assume that Celery will Just Work and handle all the IO/mashalling/etc. for me", "group_id": 81, "id": 84138}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297461340.7628739, "message": "I wouldn't see running celeryd as a big risk factor at this point", "group_id": 81, "id": 84151}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297460772.1139629, "message": "Also stuff like Celery make some bits of it a lot easier at least conceptually", "group_id": 81, "id": 83996}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297461518.4779811, "message": "@convy If launching your startup isn't stressful as hell you probably aren't doing it right ;-)", "group_id": 81, "id": 84192}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297461471.6943059, "message": "ok then you guys have proven me wrong :) but the proof is in the bags under ericflo's eyes lol", "group_id": 81, "id": 84184}, {"user_id": 603, "stars": [], "topic_id": 4394, "date_created": 1297461139.696908, "message": "i'd agree with @coderanger, background jobs and asynch processes mitigate risk from moving parts.", "group_id": 81, "id": 84091}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297462604.647619, "message": "</making things up>", "group_id": 81, "id": 84344}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297461614.7371521, "message": "how would you keep a open connection to redis?", "group_id": 81, "id": 84218}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297462477.5507729, "message": "@convy I don't know of an async-oriented Redis client lib in Python so I would guess the Convore folks wrote one.", "group_id": 81, "id": 84331}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297462491.5807149, "message": "but this is an open connection, not using a pool i'd imagine.", "group_id": 81, "id": 84332}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297464229.712657, "message": "but it's lacking examples", "group_id": 81, "id": 84474}, {"user_id": 548, "stars": [], "topic_id": 4394, "date_created": 1297462269.3463161, "message": "doesn't the last version of redis-py have connection pooling?  I thought I saw they added it with the hiredis stuff?", "group_id": 81, "id": 84314}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297462572.088202, "message": "Or maybe they are just using redis-py since I think eventlet can monkeypatch the socket library. Using redis-py to make a native Eventlet client wouldn't be that hard I don't think", "group_id": 81, "id": 84339}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297462594.640573, "message": "redis-py provides all the parser logic, you just need to hook up the eventlet side of the comms", "group_id": 81, "id": 84343}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297464071.1936929, "message": "for msg in redis_client.listen():...", "group_id": 81, "id": 84468}, {"user_id": 2363, "stars": [{"date_created": 1297463953.6978099, "user_id": 1736}], "topic_id": 4394, "date_created": 1297463820.677151, "message": "@coderanger  redis=eventlet.import_patched('redis')", "group_id": 81, "id": 84448}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297463842.2531769, "message": "that's all that has to be done to have async redis client", "group_id": 81, "id": 84449}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297464165.4295521, "message": "yes, it is", "group_id": 81, "id": 84473}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297465243.6174071, "message": "@ericflo can you tell how many eventlet.wsgi processes do you roun per server ?", "group_id": 81, "id": 84511}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297465261.325743, "message": "s/roun/run/", "group_id": 81, "id": 84512}, {"user_id": 1, "stars": [{"date_created": 1297464779.175525, "user_id": 2363}, {"date_created": 1297464975.2919691, "user_id": 1736}], "topic_id": 4394, "date_created": 1297464733.8308439, "message": "@forgems We have no cross-domain needs right now, and in my tests the error conditions were more consistent in XHR vs. script tag.", "group_id": 81, "id": 84492}, {"user_id": 2363, "stars": [{"date_created": 1297463954.32938, "user_id": 1736}], "topic_id": 4394, "date_created": 1297463929.4379771, "message": "after that you just eventlet.spawn_n(redis_subscriber)", "group_id": 81, "id": 84455}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297463962.4786201, "message": "and read the messages from redis in a loop", "group_id": 81, "id": 84459}, {"user_id": 1736, "stars": [{"date_created": 1297464157.2881811, "user_id": 1}], "topic_id": 4394, "date_created": 1297464134.4291911, "message": "And this is why eventlet is awesome :-)", "group_id": 81, "id": 84470}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297464633.9989691, "message": "@ericflo why have you decided to use xmlhttprequest long pooling instead of script tag long pooling ?", "group_id": 81, "id": 84488}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297465622.2192039, "message": "@forgems 4 for the Django stack, 8 for the live endpoint.", "group_id": 81, "id": 84539}, {"user_id": 2363, "stars": [], "topic_id": 4394, "date_created": 1297465826.1481259, "message": "that's not much :)", "group_id": 81, "id": 84550}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297478931.308778, "message": "what does the long polling xmlhttprequest look like?", "group_id": 81, "id": 85511}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297480157.4680271, "message": "@convy Same as a normal one, the server just doesn't send back a response until it has data.", "group_id": 81, "id": 85552}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297603259.031867, "message": "so if you say 8 process for live, you would create 8 of these, one on each port?", "group_id": 81, "id": 90708}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297603226.5457981, "message": "@forgems so how would you start the evenlet process, like this? http://eventlet.net/doc/examples.html#wsgi-server", "group_id": 81, "id": 90705}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297602994.4660299, "message": "so is that a single celery task that does all that?", "group_id": 81, "id": 90679}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297603316.9475629, "message": "so that means there are 8 open connections into redis, I guess you have to make things transactional somehow so you don't sub/pub one to many times then...", "group_id": 81, "id": 90709}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297602980.363102, "message": "@ericflo what message queue are you using?", "group_id": 81, "id": 90678}, {"user_id": 1736, "stars": [{"date_created": 1297625928.435998, "user_id": 1}, {"date_created": 1297805286.91642, "user_id": 275}], "topic_id": 4394, "date_created": 1297620608.1376729, "message": "@convy Given previous statements, I would guess they are using the Redis virtual transport backend to Celery/Kombu.", "group_id": 81, "id": 92508}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297625939.245487, "message": "@coderanger is totally right", "group_id": 81, "id": 92958}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297632686.323386, "message": "@coderanger oh, you are saying they are using redis as the backend instead of say rabbitmq gotcha.", "group_id": 81, "id": 93880}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297632232.4359901, "message": "@coderanger any links that might help me understand that?", "group_id": 81, "id": 93816}, {"user_id": 133, "stars": [], "topic_id": 4394, "date_created": 1297680756.3768139, "message": "@ericflo if you're using redis as the celery broker, how are you dealing with persistence?", "group_id": 81, "id": 97925}, {"user_id": 133, "stars": [], "topic_id": 4394, "date_created": 1297682294.080348, "message": "e.g. what happens if your redis server dies? All messages in flight are lost?", "group_id": 81, "id": 98116}, {"user_id": 9013, "stars": [], "topic_id": 4394, "date_created": 1297689236.8104241, "message": "redis has support for file-based persistence", "group_id": 81, "id": 99117}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297695768.150619, "message": "and it's pretty fast as reloading the data", "group_id": 81, "id": 100334}, {"user_id": 1126, "stars": [], "topic_id": 4394, "date_created": 1297698050.6382351, "message": "is there a particular reason you guys went for redis as a backend instead of rabbit/zeromq? /cc @ericflo", "group_id": 81, "id": 100744}, {"user_id": 1736, "stars": [], "topic_id": 4394, "date_created": 1297704499.276083, "message": "@pewpewarrows I would guess because redis is fast enough as a broker for all but the highest loads, and redis has a lot of other (awesome) uses so it is nice to get double duty out of it.", "group_id": 81, "id": 101773}, {"user_id": 133, "stars": [], "topic_id": 4394, "date_created": 1297716909.089123, "message": "@simonpantzare yeah, it does, but that comes with some tradeoffs. Given that they have one dedicated redis server, I'm not sure they would want it to work that way. @ericflo, share some deets! ^^", "group_id": 81, "id": 104907}, {"user_id": 1, "stars": [], "topic_id": 4394, "date_created": 1297906927.621295, "message": "@convy Not really, I just kind of muddled through it, it's not that bad once you get the hang of it.", "group_id": 81, "id": 128242}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297906900.927002, "message": "@ericflo did you get help to setup puppet? what did you use to learn, any tutorial that helped you out the most? book?", "group_id": 81, "id": 128231}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297909350.0957739, "message": "@bitprophet gotcha, thanks for info", "group_id": 81, "id": 128657}, {"user_id": 4156, "stars": [], "topic_id": 4394, "date_created": 1297908468.3906641, "message": "@convy There's a bunch of stuff out there. You can also download/run the Puppet docs site locally, which is great for plane/train/etc at the cost of installing a few Rubygems.", "group_id": 81, "id": 128487}, {"user_id": 4156, "stars": [], "topic_id": 4394, "date_created": 1297908560.2116871, "message": "@convy In my experience, the tools are the ones that often provide readymade cookbooks/whatever. Chef has a knowledgebase of cookbooks and I'm 99.999% sure puppet does too.", "group_id": 81, "id": 128508}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297908525.063241, "message": "there must be some templates for setting up servers etc., surprised rackspace doesn't have their own repo for this stuff to make things easier for their clients!", "group_id": 81, "id": 128500}, {"user_id": 4156, "stars": [], "topic_id": 4394, "date_created": 1297908598.021632, "message": "I think that makes more sense than having each individual hosting provider build up such things -- though some do. (RimuHosting has a big wiki with lots of info, for example)", "group_id": 81, "id": 128518}, {"user_id": 12937, "stars": [], "topic_id": 4394, "date_created": 1297947113.363497, "message": "eeeeeeee", "group_id": 81, "id": 131272}, {"user_id": 6415, "stars": [{"date_created": 1297970538.8061061, "user_id": 4219}, {"date_created": 1297986760.880065, "user_id": 141}], "topic_id": 4394, "date_created": 1297966506.095988, "message": "@convy http://twitter.com/ericflo/status/24390721339392000", "group_id": 81, "id": 133783}, {"user_id": 4219, "stars": [{"date_created": 1297981276.521955, "user_id": 6415}], "topic_id": 4394, "date_created": 1297970557.9462881, "message": "@davidpaccoud merci!", "group_id": 81, "id": 134460}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297975127.2185309, "message": "puppet is powerful stuff, i can be a makeshift sysadmin.....hardcore!", "group_id": 81, "id": 135639}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1297975143.1302431, "message": "just as long as everything places nice :)", "group_id": 81, "id": 135644}, {"user_id": 6415, "stars": [], "topic_id": 4394, "date_created": 1297981277.7415681, "message": "@convy you're welcome", "group_id": 81, "id": 137025}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1298150813.398257, "message": "wonder what the difference is between puppet and puppet enterprise...I thought all good thins in life were free! :(", "group_id": 81, "id": 150594}, {"user_id": 4219, "stars": [], "topic_id": 4394, "date_created": 1298258877.6026361, "message": "any ideas what tinychat does behind the scenes?", "group_id": 81, "id": 155861}]