convore.json/groups/hacker-news/creating-a-really-really-simple-large-2gb-file-upload-app/messages.json

[{"user_id": 8668, "stars": [], "topic_id": 18897, "date_created": 1302614112.5533929, "message": "We need to give clients a way to upload very large files to us... at the moment they upload via either http or ftp, but http is not suitable for uploading 2GB files (correct me if I am wrong!) and ftp is proving too tricky for some clients", "group_id": 95, "id": 642064}, {"user_id": 26065, "stars": [], "topic_id": 18897, "date_created": 1302614661.7727411, "message": "checkout plupload.com", "group_id": 95, "id": 642243}, {"user_id": 8668, "stars": [], "topic_id": 18897, "date_created": 1302614149.255585, "message": "We were thinking of doing something like an Adobe Air application, cross-platform drag-and-drop, perhaps building it up into a complete upload / download / file manager thing over time", "group_id": 95, "id": 642075}, {"user_id": 8668, "stars": [], "topic_id": 18897, "date_created": 1302614165.939224, "message": "I was wondering what approaches people would recommend for this, and which protocol for the upload? (Resuming is a *must*)", "group_id": 95, "id": 642083}, {"user_id": 4169, "stars": [], "topic_id": 18897, "date_created": 1302614660.781884, "message": "http://streamfile.com/", "group_id": 95, "id": 642242}, {"user_id": 9420, "stars": [], "topic_id": 18897, "date_created": 1302614930.664217, "message": "Video files? http://latakoo.com ", "group_id": 95, "id": 642344}, {"user_id": 8327, "stars": [{"date_created": 1302626233.316983, "user_id": 22043}], "topic_id": 18897, "date_created": 1302616909.478734, "message": "If you're looking at a DIY solution, you could perhaps do something with the BitTorrent protocol; offer clients a very cut down abstracted interface that would generate a torrent file, upload it to a tracker you host expressly for the purpose, add a hook to the tracker that says when torrent file is received (typically a few kbs) begin grabbing file from peer. Resumable, and each chuck is hash checked. Biggest problem could potentially come from a client's firewall.", "group_id": 95, "id": 642952}, {"user_id": 25720, "stars": [{"date_created": 1302907582.534966, "user_id": 1}], "topic_id": 18897, "date_created": 1302618668.8168571, "message": "try http://sendoid.com/", "group_id": 95, "id": 643251}, {"user_id": 4149, "stars": [{"date_created": 1302680561.9165881, "user_id": 9262}], "topic_id": 18897, "date_created": 1302618817.0361359, "message": "i think he wants a way to receive 2gb files and i feel plupload.com would be your best bet, it is a framework that is easy to install and setup", "group_id": 95, "id": 643264}, {"user_id": 15929, "stars": [], "topic_id": 18897, "date_created": 1302621866.133949, "message": "dropbox?", "group_id": 95, "id": 644023}, {"user_id": 8668, "stars": [], "topic_id": 18897, "date_created": 1302770726.232415, "message": "Dropbox is entirely unsuitable for this kind of workflow. Really we want to build our own solution (for various reasons). I already considered bittorrent @mttwrnr but that would require getting clients through their NAT, and also adds a lot of overhead for an A to B transfer", "group_id": 95, "id": 681158}, {"user_id": 4859, "stars": [{"date_created": 1302907871.6552701, "user_id": 8327}], "topic_id": 18897, "date_created": 1302907560.743717, "message": "rsync?", "group_id": 95, "id": 706311}, {"user_id": 4149, "stars": [], "topic_id": 18897, "date_created": 1302927817.487159, "message": "@SquidLord they are using that currently but they want it over HTTP now if you read up they said that", "group_id": 95, "id": 708799}, {"user_id": 11039, "stars": [], "topic_id": 18897, "date_created": 1302928232.232388, "message": "@italiano40 Actually, you only said ftp is currently too complicated for them. Which is why I suggested just building a lightweight, simplified interface over a protocol that you already know works, has worked for decades, and is easy to interface with due to tonnes of extant, simple APIs. You can build a glassy, dead simple interface over the top of it in half the time you'd spend fighting with new protocol implementations like PLUpload. You can put all the development effort into building the UI, which is the part your clients are having trouble with, rather than the protocol, which with ftp is one you already *know* works.\n\nI recognize the implicit hacker response to anything is \"reimplement it from scratch,\" but after a while, you get to the point you only want to wrestle with the code of the part what's broken.\n\nSending 2+GB files over HTTP is a lousy idea, frankly, even if it does work. Or can be made to. HTTP is not defined as a sustained-transfer protocol; its not built to do that. So why bang your head against a wall trying to milk a pig when you can just go grab a cow?", "group_id": 95, "id": 708846}, {"user_id": 11039, "stars": [], "topic_id": 18897, "date_created": 1302927669.319102, "message": "How about the obvious simplest solution: Lightweight graphic front-end for FTP. There are tonnes of FTP libraries and its easy enough to gloss it with Adobe Air on the front end.\n\nNo need to reimplement a solution that's worked for 30 years, really.", "group_id": 95, "id": 708785}, {"user_id": 23304, "stars": [], "topic_id": 18897, "date_created": 1302958357.336817, "message": "Transferbigfiles.com", "group_id": 95, "id": 710422}, {"user_id": 23304, "stars": [], "topic_id": 18897, "date_created": 1302958340.4328389, "message": "Transferbigf", "group_id": 95, "id": 710421}, {"user_id": 11358, "stars": [], "topic_id": 18897, "date_created": 1302966318.7110629, "message": "I think many people do it using flash.", "group_id": 95, "id": 711180}, {"user_id": 8327, "stars": [], "topic_id": 18897, "date_created": 1302978898.213876, "message": "Burn it, then UPS it, in my experience even the least tech savvy clients can manage that ;)", "group_id": 95, "id": 712924}, {"user_id": 30049, "stars": [], "topic_id": 18897, "date_created": 1303119344.6934991, "message": "You can do it with a combination of HTML 5 FileReader, HTML 5 TypedArrays, streaming SHA1 implementation, content-dependent chunking, sliding window hashing (use XOR rather than Rabin-Karp) and data de-duplication theory. You need a server endpoint for uploading chunks and an endpoint for downloading files (aggregations of chunks). Something like this would upload more than three times faster than Google Docs (you would be uploading multiple chunks of a file in parallel) and would be able to handle at least 4GB. It would by definition support resumable uploads.", "group_id": 95, "id": 724431}, {"user_id": 4935, "stars": [{"date_created": 1303158979.4913881, "user_id": 4149}], "topic_id": 18897, "date_created": 1303158643.0211289, "message": "http://transloadit.com/", "group_id": 95, "id": 730677}]