convore.json/groups/django-community/tips-for-a-big-django-project/messages.json

[{"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299189118.8671539, "message": "We're going to work on an application that may grow indefinitely if everything goes well, and this is the reason because I'm looking for some advice from the experts (aka you).\n\nLet me tell you a bit about the project:\nIt's mainly a webservice, where mobile devices will query to an API (read only).\nBut, there will be a web based application where users will load data, two kinds: one that will be around 5000 users, and the other with around 10000 users.\nIn total, there will be 3 big models: \nusers: with those ~15k users\nmodelA: with ~20k records\nmodelB: with ~250k records (dependent of modelA, ForeignKey)\nAnd few more models that acts as metadata of models A and B\n\nThese are the numbers for a speculated start.\n\nNow, what are your suggestions about everything:\nFramework: should we tie to Django? we love Django. If yes, which apps would be a must?\nDatabases: PostgreSQL is ok? Should we use some no-SQL for any specific data?\nServers: do we need load balancers? proxies? \nCache: we plan to use memcached, and since webservice will be read only, most of the requests will be cached, do you suggest anything better?\n\nI hope it also help others in our same situation", "group_id": 81, "id": 264210}, {"user_id": 2062, "stars": [], "topic_id": 10598, "date_created": 1299190198.537919, "message": "agreed the number of records sounds very reasonable. getting caching invalidation working properly will be important.   piston might be useful for the api depending on the required features", "group_id": 81, "id": 264517}, {"user_id": 7129, "stars": [{"date_created": 1299198685.923418, "user_id": 16570}, {"date_created": 1299241316.3668351, "user_id": 7179}], "topic_id": 10598, "date_created": 1299189986.8105299, "message": "Unless you feel the need to use something cool, I would stick with Postgres - 250k rows is really nothing big and you'll have zero surprises. Don't worry about any of the other stuff until you have your application up and running, it's mostly a distraction. The exception to this might be caching as it often takes some thinking ahead to get a sane invalidation scheme. Anyway, good luck with your project :)", "group_id": 81, "id": 264441}, {"user_id": 2062, "stars": [], "topic_id": 10598, "date_created": 1299190199.9638669, "message": "https://bitbucket.org/jespern/django-piston/wiki/Home", "group_id": 81, "id": 264519}, {"user_id": 7, "stars": [], "topic_id": 10598, "date_created": 1299190685.2686019, "message": "postgres will be fine, typewar has a table over several million records", "group_id": 81, "id": 264591}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299190590.9185171, "message": "Great, thanks guys. And what about the concurrent users loading data? Should I care about that in any special way? Do you think Postgres will handle it ok?", "group_id": 81, "id": 264582}, {"user_id": 16570, "stars": [{"date_created": 1299230102.682622, "user_id": 9650}, {"date_created": 1299239303.0562279, "user_id": 7179}], "topic_id": 10598, "date_created": 1299198669.896179, "message": "Also keep in mind that running with a stable, well known DB platform makes it easier to 1) scale 2) fix bugs (most people have probably already run into the same ones) and 3) hire people.\n\nAs for load balancers/proxies: how do you expect to deploy? Apache2 + mod_wsgi? If so, stick an nginx proxy in front, and just make sure you're serving your static media off another instance. Then tweak the hell out of your caching schemes :)", "group_id": 81, "id": 265255}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299241055.7703149, "message": "Thank you!", "group_id": 81, "id": 269085}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299241099.9723649, "message": "What do you prefer Piston or the new Tastypie? I've seen that Piston core wasn't updated for several months", "group_id": 81, "id": 269087}, {"user_id": 13814, "stars": [], "topic_id": 10598, "date_created": 1299246451.972666, "message": "If your mobile app needs to track location at some point, the postgresl is by far the best choice for geolocation (i.e., GeoDjango.)", "group_id": 81, "id": 269503}, {"user_id": 13814, "stars": [], "topic_id": 10598, "date_created": 1299246594.7575569, "message": "If data loading becomes a headache, moving to queuing system (celery and rabbitmq or redis) is a snap.", "group_id": 81, "id": 269528}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299246645.2311399, "message": "@rockhoward yes, this another thing I'm considering. At the moment it doesn't needs geolocation, but will be a possibility. And yes, pg does a great work with geo spatial searches.", "group_id": 81, "id": 269537}, {"user_id": 13814, "stars": [], "topic_id": 10598, "date_created": 1299246383.10181, "message": "I made several changes to piston before the changes stopped. It was on the way to being great, but the lack of updates is making me start to look elsewhere.", "group_id": 81, "id": 269492}, {"user_id": 13814, "stars": [], "topic_id": 10598, "date_created": 1299246779.4317529, "message": "The decision to use noSQL boils down to details about your data model, data sizes and the types of queries you require. Generally you can ignore it in the early stages of any app and add it later as necessary for optimization, However if the data is highly integrated such as in a social app with millions of users, then noSQL really does become a required piece of infrastructure.", "group_id": 81, "id": 269545}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299246923.387033, "message": "@rockhoward I'll keep in mind the queuing systems! And yes, I don't think noSQL is a requirement for now, specially since our model has many interrelations between submodels. And also, we're not having sessions between devices and server, API requests are made using simple tokens, then we don't even think in store sessions on an external db.", "group_id": 81, "id": 269553}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299249232.951715, "message": "@tomchristie oh, looks interesting! Will give it a try..", "group_id": 81, "id": 269729}, {"user_id": 18316, "stars": [], "topic_id": 10598, "date_created": 1299248255.1181469, "message": "Also worth taking a look at http://django-rest-framework.org for APIs.  New but looking v nice.  Well tested and with an awesome admin style API browser.  http://api.django-rest-framework.org (Okay I'll get off my sales pitch now!) :)", "group_id": 81, "id": 269686}, {"user_id": 11405, "stars": [], "topic_id": 10598, "date_created": 1299251099.926353, "message": "You may want to have a look at this topic: https://convore.com/django-community/rest-in-django-piston-or-tastypie/ There are a lot more options for REST than mentioned here... If you decided on one of the many available options, would be cool if you release some blog post or so on your decision :)", "group_id": 81, "id": 269971}, {"user_id": 7179, "stars": [], "topic_id": 10598, "date_created": 1299255766.3532529, "message": "@mbrochh great, thanks!", "group_id": 81, "id": 270888}]