convore.json/groups/python/faster-json-library/messages.json

[{"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300676067.965847, "message": "I'm not going to claim that I require the absolute fastest JSON performance. Mostly I'm just curious. So, I did some comparisons of the four JSON libraries I was aware of: the builtin json, simplejson, jsonlib, and cjson.", "group_id": 292, "id": 394926}, {"user_id": 1, "stars": [], "topic_id": 14143, "date_created": 1300676359.301944, "message": "No idea if that's true or not", "group_id": 292, "id": 394952}, {"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300676292.2994871, "message": "But cjson is broken and abandoned! So is there anything else available operating at such speeds?", "group_id": 292, "id": 394946}, {"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300676181.012167, "message": "With just a simple test object on my laptop, I got these results with the timeit module:", "group_id": 292, "id": 394929}, {"user_id": 12791, "stars": [{"date_created": 1300680264.732157, "user_id": 14562}, {"date_created": 1300793330.5901871, "user_id": 831}], "topic_id": 14143, "date_created": 1300676186.814142, "message": "Encoding with json\n10000 loops, best of 3: 38.6 usec per loop\nEncoding with simplejson\n100000 loops, best of 3: 8.36 usec per loop\nEncoding with jsonlib\n100000 loops, best of 3: 5.66 usec per loop\nEncoding with cjson:\n100000 loops, best of 3: 4 usec per loop\nDecoding with json\n10000 loops, best of 3: 136 usec per loop\nDecoding with simplejson\n100000 loops, best of 3: 5.36 usec per loop\nDecoding with jsonlib\n100000 loops, best of 3: 12 usec per loop\nDecoding with cjson\n1000000 loops, best of 3: 1.68 usec per loop", "group_id": 292, "id": 394932}, {"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300676253.038538, "message": "As you can see, cjson looks like the fastest in this non-rigorous test, especially considering both encoding and decoding (see jsonlib's time double for decoding)", "group_id": 292, "id": 394939}, {"user_id": 1, "stars": [{"date_created": 1300677120.9116271, "user_id": 12791}, {"date_created": 1300681929.8616059, "user_id": 1982}, {"date_created": 1300735871.6258581, "user_id": 11663}], "topic_id": 14143, "date_created": 1300676338.6100321, "message": "Someone was mentioning to me that py-yajl was really fast https://github.com/rtyler/py-yajl/", "group_id": 292, "id": 394949}, {"user_id": 12791, "stars": [{"date_created": 1300677025.9198711, "user_id": 1}, {"date_created": 1300678493.0730281, "user_id": 4578}, {"date_created": 1300680305.2862599, "user_id": 14562}, {"date_created": 1300681952.8843949, "user_id": 1982}], "topic_id": 14143, "date_created": 1300676865.030242, "message": "Looks true! With the same data and test setup, add on py-yajl:\nEncoding with py-yajl:\n100000 loops, best of 3: 2.59 usec per loop\nDecoding with py-yajl:\n100000 loops, best of 3: 2.98 usec per loop", "group_id": 292, "id": 394969}, {"user_id": 9307, "stars": [], "topic_id": 14143, "date_created": 1300681766.912858, "message": "I heard codetalker was fast, too.  http://pypi.python.org/pypi/CodeTalker/0.5", "group_id": 292, "id": 395256}, {"user_id": 9307, "stars": [], "topic_id": 14143, "date_created": 1300681793.1004701, "message": "but haven't tried it", "group_id": 292, "id": 395259}, {"user_id": 927, "stars": [{"date_created": 1300766069.1040821, "user_id": 12791}], "topic_id": 14143, "date_created": 1300708357.38991, "message": "Ultrajson was pretty fast in my testing: https://github.com/jskorpan/ultrajson", "group_id": 292, "id": 397241}, {"user_id": 5630, "stars": [{"date_created": 1300766723.873409, "user_id": 12791}], "topic_id": 14143, "date_created": 1300712147.472688, "message": "Just run on pypy :-p", "group_id": 292, "id": 397442}, {"user_id": 7, "stars": [{"date_created": 1300766072.872155, "user_id": 12791}], "topic_id": 14143, "date_created": 1300710653.1754439, "message": "Please don't use cjson, last I check it is riddled with bugs", "group_id": 292, "id": 397367}, {"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300764839.709738, "message": "Indeed, with the same machine/test, ultrajson comes in as the fastest", "group_id": 292, "id": 404713}, {"user_id": 18539, "stars": [], "topic_id": 14143, "date_created": 1300763906.3431449, "message": "I'm glad ultrajson is fast, but its documentation seems somewhat sparse -- I couldn't figure out if it does iterative parsing for example. (my main rationale for iterative parsing is \"not blocking the reactor\")", "group_id": 292, "id": 404698}, {"user_id": 12791, "stars": [{"date_created": 1300783162.5524261, "user_id": 14438}], "topic_id": 14143, "date_created": 1300764857.0058999, "message": "Encoding with ujson:\n1000000 loops, best of 3: 1.05 usec per loop\nDecoding with ujson:\n1000000 loops, best of 3: 1.4 usec per loop", "group_id": 292, "id": 404714}, {"user_id": 12791, "stars": [], "topic_id": 14143, "date_created": 1300766705.513258, "message": "Looks like yajl (the library used by py-yajl) is more well tested and documented than ultrajson though.", "group_id": 292, "id": 404839}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300792710.4210219, "message": "Though I average my results and ran it on a relatively large JSON file (6.4k)", "group_id": 292, "id": 406339}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300792577.5655949, "message": "Running tests locally I get comparable results with the newest version of simplejson and cjson", "group_id": 292, "id": 406331}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300792595.7133319, "message": "I haven't tested py-yajl though", "group_id": 292, "id": 406332}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300792630.6982789, "message": "https://bitbucket.org/IanLewis/jsonlib-test", "group_id": 292, "id": 406336}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300792768.2008989, "message": "Running 10 loops with 500 iterations with 2 processes\nPython 2.6.5 (r265:79063, Apr 16 2010, 13:09:56) \n[GCC 4.4.3]\n\nencode\n========================================\n\nsimplejson:       3175.44574 /s\njsonlib2:         2925.55263 /s\ncjson:            2901.22826 /s\njsonlib:          2860.49601 /s\njson:             1489.21863 /s\ndemjson:           250.03410 /s\n\ndecode\n========================================\n\nsimplejson:       3224.92945 /s\njsonlib2:         2980.32737 /s\ncjson:            2940.61543 /s\njsonlib:          2847.47568 /s\njson:              673.22381 /s\ndemjson:           145.67908 /s", "group_id": 292, "id": 406342}, {"user_id": 19606, "stars": [], "topic_id": 14143, "date_created": 1300799583.2898891, "message": "Total  Read   Write  Implementation\n-----------------------------------\n0.657  0.283  0.373  django.utils.simplejson\n0.691  0.311  0.380  simplejson\n0.799  0.518  0.281  yajl\n0.853  0.297  0.556  cjson\n0.926  0.406  0.520  jsonlib2\n1.771  1.293  0.478  jsonlib\n10.671  4.369  6.302  jsonpickle\n19.567  13.157  6.410  json", "group_id": 292, "id": 406811}, {"user_id": 19606, "stars": [], "topic_id": 14143, "date_created": 1300799670.830513, "message": "I suspect django selects an installed implementation rather than using its own these days, it used to be quite slow.", "group_id": 292, "id": 406819}, {"user_id": 19606, "stars": [], "topic_id": 14143, "date_created": 1300799578.989435, "message": "The benchmarks I use in anyjson for selecting the fastest available implementation currently looks like this:", "group_id": 292, "id": 406810}, {"user_id": 10716, "stars": [], "topic_id": 14143, "date_created": 1300801100.237524, "message": "@ianmlewis can you run that same test with yajl? and interesting that simplejson won by a far margin...", "group_id": 292, "id": 407128}, {"user_id": 18539, "stars": [], "topic_id": 14143, "date_created": 1300805939.667176, "message": "And importantly, ultrajson. Tests and docs can be added :)", "group_id": 292, "id": 407915}, {"user_id": 3354, "stars": [], "topic_id": 14143, "date_created": 1300832468.143333, "message": "Well if something doesn't knock simplejson off its perch that'll do!", "group_id": 292, "id": 412388}, {"user_id": 7, "stars": [], "topic_id": 14143, "date_created": 1300860680.6517749, "message": "are you guys seriously that bottlenecked on JSON performance?", "group_id": 292, "id": 414744}, {"user_id": 1992, "stars": [], "topic_id": 14143, "date_created": 1300861246.3882451, "message": "mostly json and netstrings: http://cr.yp.to/proto/netstrings.txt", "group_id": 292, "id": 414792}, {"user_id": 1992, "stars": [], "topic_id": 14143, "date_created": 1300861168.1269901, "message": "I would bet it comes from the mongrel2 mailing list, they've been bikeshedding over serialization all week", "group_id": 292, "id": 414782}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300876914.386673, "message": "As if it's bugs and abandonment weren't enough :(", "group_id": 292, "id": 415999}, {"user_id": 4383, "stars": [], "topic_id": 14143, "date_created": 1300876878.6930821, "message": "@alex No. simplejson is good enough. A client requested using cjson and I needed some ammunition for not using it.", "group_id": 292, "id": 415997}, {"user_id": 10716, "stars": [], "topic_id": 14143, "date_created": 1300895945.5177431, "message": "@alex in my case yes,we store in JSON and do millions of JSON API calls a day, so the less time we can spend processing JSON the better", "group_id": 292, "id": 418440}, {"user_id": 18539, "stars": [], "topic_id": 14143, "date_created": 1300905967.7031901, "message": "@alex Yes, I have been blocked on serialization where excessively large particularly deeply nested structures blocked the reactor so long it was a real problem. Especially so when encoding custom types. Maybe Pypy will fix that? ;-)", "group_id": 292, "id": 419674}, {"user_id": 7, "stars": [], "topic_id": 14143, "date_created": 1300920555.8808861, "message": "@lvh maybe, beating a hand optimized C extension is tough, but we're working on it :)", "group_id": 292, "id": 421497}, {"user_id": 927, "stars": [], "topic_id": 14143, "date_created": 1300935653.85937, "message": "simplejson on PyPy is close to CPython and wins overall with any real processing, probably due to GC. One spare-time project is producing a simpler benchmark suite.", "group_id": 292, "id": 423521}, {"user_id": 927, "stars": [], "topic_id": 14143, "date_created": 1300935534.7245851, "message": "@alex Not always but it's close: we're getting huge piles of data (100M+ records/day). One use-case is doing MapReduce queries over it", "group_id": 292, "id": 423515}, {"user_id": 927, "stars": [], "topic_id": 14143, "date_created": 1300939776.870347, "message": "Was talking with him about that, too, since I think our data causes more object churn", "group_id": 292, "id": 423942}, {"user_id": 7, "stars": [], "topic_id": 14143, "date_created": 1300939293.668241, "message": "@acdha nice, might want to coordinate with Bob Ippollito (spelling?) who was working on some benchmarks, and some nice optimizations for pypy on simplejson to boot", "group_id": 292, "id": 423906}]