convore.json/groups/django-community/how-can-i-measure-the-speed-of-my-django-app-over-time/messages.json

[{"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302435653.185225, "message": "People often complain about the django-cms being slow, while this might be the case, I'd love to have hard metrics about this, but have no idea how to do it", "group_id": 81, "id": 609129}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302435691.8252821, "message": "I want to have some sort of tool that shows me that whatever improvements I do actually make it faster", "group_id": 81, "id": 609131}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302435715.767698, "message": "Something like speed.pypy.org would be cool, and the tech behind it seems to be open source but is lacking documentation on how the heck one should use it", "group_id": 81, "id": 609133}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453803.5371771, "message": "well that's a web app isn't it?", "group_id": 81, "id": 610695}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453938.0406749, "message": "timeit doesn't really seem appropriate here", "group_id": 81, "id": 610748}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453966.0793369, "message": "you could use django debug toolbar", "group_id": 81, "id": 610757}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454106.0301731, "message": "like 'whole page rendering'?", "group_id": 81, "id": 610805}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453780.139133, "message": "it's not about a web app", "group_id": 81, "id": 610686}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453984.909976, "message": "and ddt is just 'snapshots'", "group_id": 81, "id": 610768}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454110.771666, "message": "more granular would be more useful", "group_id": 81, "id": 610807}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453994.351917, "message": "doesn't provide real comparable metrics", "group_id": 81, "id": 610776}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453980.123981, "message": "it shows how many queries you are running", "group_id": 81, "id": 610764}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454063.979835, "message": "I want something that proofs whatever fix I come up with is valid", "group_id": 81, "id": 610794}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453435.729795, "message": "I use siege", "group_id": 81, "id": 610627}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453770.96789, "message": "on reference hardware", "group_id": 81, "id": 610684}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453804.8518031, "message": ":D", "group_id": 81, "id": 610698}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453821.4082839, "message": "or web site, whatever you like to call it", "group_id": 81, "id": 610706}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454203.7902181, "message": "graphs are not important, just something I can see the progress over time (eg: for every commit)", "group_id": 81, "id": 610823}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453823.6962521, "message": "yea but just req/sec sounds a bit too simple", "group_id": 81, "id": 610708}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453835.526516, "message": "it's not for a website, it's for a django app", "group_id": 81, "id": 610714}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453864.9597991, "message": "a reusable app? or what?", "group_id": 81, "id": 610723}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453876.999788, "message": "yea it's a reusable app", "group_id": 81, "id": 610729}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453907.9102941, "message": "if it exposes views you could still do that", "group_id": 81, "id": 610737}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453924.759172, "message": "if it exposes other code profile it with timeit or something", "group_id": 81, "id": 610743}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302454014.282433, "message": "in my experience, most of the time it's because you're running queries in a for loop without even knowing it", "group_id": 81, "id": 610781}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453756.1294899, "message": "when it's about the web application as a whole it's best to measure how much requests/sec it can serve on different concurrent loads", "group_id": 81, "id": 610679}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453790.062649, "message": "it's for the django-cms project", "group_id": 81, "id": 610689}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453873.2639611, "message": "want stuff like 'how long does it take to render the menu', 'how long does it take to render a placeholder' etc", "group_id": 81, "id": 610726}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453943.561964, "message": "there's too much 'setup'", "group_id": 81, "id": 610750}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302453949.4993789, "message": "but most of the time speed bottlenecks come from database access codes", "group_id": 81, "id": 610752}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454100.692884, "message": "wouldn't that be way too general?", "group_id": 81, "id": 610804}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302453975.662467, "message": "not in a continuous and automated way", "group_id": 81, "id": 610763}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302454161.469867, "message": "you could make a graph out of it :), since you want graphs", "group_id": 81, "id": 610815}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454050.7517381, "message": "as stated above: I'm not looking for a fix for the perfomance issues yet", "group_id": 81, "id": 610792}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302454072.868001, "message": "and doesn't introduce new regressions", "group_id": 81, "id": 610795}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302454089.0588191, "message": "setup a sample project and do it with a load testing tool like siege", "group_id": 81, "id": 610798}, {"user_id": 13912, "stars": [], "topic_id": 18355, "date_created": 1302454367.4739339, "message": "something like this would tell you specifics about which parts of code take what amount of time http://code.djangoproject.com/wiki/ProfilingDjango", "group_id": 81, "id": 610863}, {"user_id": 257, "stars": [], "topic_id": 18355, "date_created": 1302459517.5211191, "message": "if the tests suddenly start taking much longer then you know your changes had a negative effect on performance (or that the environment changed)", "group_id": 81, "id": 611941}, {"user_id": 257, "stars": [{"date_created": 1302566067.508826, "user_id": 141}], "topic_id": 18355, "date_created": 1302459307.3997359, "message": "@ojiidotch: Just write functional tests and time them", "group_id": 81, "id": 611907}, {"user_id": 257, "stars": [], "topic_id": 18355, "date_created": 1302459783.6596589, "message": "don't know specifics about the codespeed project (which is what speed.pypy.org uses), but it looks like you post the runtimes of your tests to this website", "group_id": 81, "id": 611977}, {"user_id": 257, "stars": [{"date_created": 1302459798.8898561, "user_id": 1736}, {"date_created": 1302462444.830153, "user_id": 5778}, {"date_created": 1302491016.2672341, "user_id": 4078}], "topic_id": 18355, "date_created": 1302459389.1280079, "message": "https://github.com/tobami/codespeed/", "group_id": 81, "id": 611920}, {"user_id": 257, "stars": [], "topic_id": 18355, "date_created": 1302459792.5130229, "message": "seen https://github.com/tobami/codespeed/blob/master/tools/save_single_result.py", "group_id": 81, "id": 611981}, {"user_id": 257, "stars": [], "topic_id": 18355, "date_created": 1302459794.7361441, "message": "-n", "group_id": 81, "id": 611983}, {"user_id": 257, "stars": [], "topic_id": 18355, "date_created": 1302459575.956079, "message": "if you run the tests as a commit hook then you can even see what changeset caused the degredation", "group_id": 81, "id": 611946}, {"user_id": 9896, "stars": [], "topic_id": 18355, "date_created": 1302466052.429481, "message": "the big thing about codespeed is how do I get the data? as far as i can see codespeed only *displays* stuff but gives no framework/help to time stuff", "group_id": 81, "id": 613129}, {"user_id": 205, "stars": [{"date_created": 1302487472.4251931, "user_id": 5778}, {"date_created": 1302693149.358259, "user_id": 2341}], "topic_id": 18355, "date_created": 1302465950.8345461, "message": "@acdha also set up one for django: http://django-speedcenter.improbable.org/", "group_id": 81, "id": 613115}, {"user_id": 7, "stars": [], "topic_id": 18355, "date_created": 1302466717.8332429, "message": "Way too noisy to be really looked at though: http://django-speedcenter.improbable.org/changes/?tre=10&rev=791e716e90fde3aa1e3af129a8c7183eb6efd681&exe=6&env=OptiPlex+755", "group_id": 81, "id": 613250}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302520321.773504, "message": "@alex Some of that's due to the way djangobench works - microbenchmarks are notoriously prone to noise like that. There's some work underway to improve codespeed's display of those as well - I find the timeline views are currently the most useful", "group_id": 81, "id": 622911}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302520491.850893, "message": "I've been working on a suite for haystack: https://github.com/acdha/django-haystack-bench", "group_id": 81, "id": 622919}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302520540.3592949, "message": "One goal is to measure major operation runtime so minor fluctuations don't dominate", "group_id": 81, "id": 622923}, {"user_id": 927, "stars": [{"date_created": 1302573508.2931631, "user_id": 141}], "topic_id": 18355, "date_created": 1302520692.6792951, "message": "For what @ojiidotch is talking about, I'd probably go with a stable test project and just measure view performance over time - that could be as simple as using django-crawler to record query counts & heap usage and plotting those over time", "group_id": 81, "id": 622927}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302521425.3302469, "message": "But really, the big thing with something like codespeed is getting away from time-based measurements where possible (e.g. counting queries, function calls, etc.) and making time based measurements run for an appreciable amount of time. Codespeed will plot anything you can give it a value, (min, value, max) or (value, stddev) so you have a lot of leeway in that regard", "group_id": 81, "id": 623002}, {"user_id": 2375, "stars": [], "topic_id": 18355, "date_created": 1302546939.182919, "message": "so... develop benchmark suite?", "group_id": 81, "id": 627702}, {"user_id": 2375, "stars": [], "topic_id": 18355, "date_created": 1302546942.1019681, "message": "oh sorry, mispost", "group_id": 81, "id": 627704}, {"user_id": 7, "stars": [], "topic_id": 18355, "date_created": 1302547619.1074769, "message": "@jacobian I'd start by bumping the iterations up a ton, if a benchmark doesn't take at least a few seconds it'll be victim to TONS of noise", "group_id": 81, "id": 627866}, {"user_id": 5852, "stars": [], "topic_id": 18355, "date_created": 1302547862.073736, "message": "I know there's a way to choose a number of trials mathmatically. But that's about all I know about that :)", "group_id": 81, "id": 627891}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302548546.2467589, "message": "@jacobian: one way might simply be running for a constant time (e.g. 5s)", "group_id": 81, "id": 628031}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302549474.3586421, "message": "@jacobian I'll do some more research on that - for something like djangobench currently I believe it could be somewhat arbitrary as long as it's multiple seconds total since we're concerned about interference on a smaller time-scale", "group_id": 81, "id": 628288}, {"user_id": 5852, "stars": [{"date_created": 1302573531.7892389, "user_id": 141}], "topic_id": 18355, "date_created": 1302547530.179949, "message": "djangobench is what happens when someone (i.e. me) who sucks at math writes a benchmark suite. I've been hoping that someone with a better grounding in statistics and measurements will come along and school me, but so far hasn't happened.", "group_id": 81, "id": 627846}, {"user_id": 5852, "stars": [], "topic_id": 18355, "date_created": 1302548732.0580909, "message": "@acdha again, how do you chose that time such that it'll yield significant results? You can't just pick arbitrarily, right?", "group_id": 81, "id": 628086}, {"user_id": 5852, "stars": [], "topic_id": 18355, "date_created": 1302547836.618979, "message": "@alex there's already the --trials option; think I should just bump up the default number to something higher?", "group_id": 81, "id": 627889}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302549539.213695, "message": "That said, I really do need to look at more sophisticated ways to look for outliers - I was wondering about something like doing an initial n calibration runs and reporting when the test runs are more substantially different", "group_id": 81, "id": 628301}, {"user_id": 927, "stars": [{"date_created": 1302573574.9295521, "user_id": 141}], "topic_id": 18355, "date_created": 1302568991.32341, "message": "I checked with the resident scientist (my wife). She suggested that for our needs even a basic median is probably fine (i.e. keep it simple) but that the best way to avoid errors would be pairing control & experiment runs and then do a signed-rank test. I like the idea of alternating runs to avoid a temporary anomaly affecting only one side", "group_id": 81, "id": 633285}, {"user_id": 14923, "stars": [], "topic_id": 18355, "date_created": 1302593363.2356279, "message": "I've always preferred to monitor the performance of production in greater detail rather than use/rely on constructed benchmarks.  Not only does that data help you answer a lot of important questions about your application and your whole system setup, but you still get to catch weird one-off outliers, track classes of requests (specific views, etc) that are too slow or getting slower, and in the worst case you at least have enough info to postmortem performance related issues.", "group_id": 81, "id": 638854}, {"user_id": 927, "stars": [], "topic_id": 18355, "date_created": 1302609027.946738, "message": "@jmoiron That's definitely the better approach for real applications - for a framework like Django I think it makes sense to have both some microbenchmarks and a suite of selected real applications which could be measured at a high-level (e.g. assume we created a static django-cms project and then simply crawled it with each Django release looking for regressions)", "group_id": 81, "id": 640847}]