convore.json/groups/django-community/how-do-i-setup-my-deployment-process-using-git-fabric-south-and-virtualenv/messages.json

[{"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297811592.285413, "message": "the one time i messed with fabric i had it \u201cgit archive\u201d into a tarball to push out, so i didn't have to worry about an unsynced checkout on the server", "group_id": 81, "id": 116618}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297812687.1547489, "message": "I always like to have a deploy box... as doing a deploy from your machine is asking for fail.", "group_id": 81, "id": 116764}, {"user_id": 7151, "stars": [{"date_created": 1297827737.960896, "user_id": 2458}], "topic_id": 6232, "date_created": 1297811786.306699, "message": "Draw out your ideal deployment plan and then write it in code.  So for example, you can do a git-clone of your project, rename your settings.py-prod to settings.py (or what have you), rsync this into a new directory on the server, on the server use your virtualenv'd pip to install all your new dependencies, set up a \"maintenance page\", run south, and then symlink your /prod to point at the new directory you uploaded.  then restart the server", "group_id": 81, "id": 116637}, {"user_id": 275, "stars": [], "topic_id": 6232, "date_created": 1297813210.7782459, "message": "@rlofthouse what do you mean by a deploy box vs local machine? How does that work?", "group_id": 81, "id": 116855}, {"user_id": 7378, "stars": [{"date_created": 1297813963.5846701, "user_id": 141}, {"date_created": 1297825962.9869699, "user_id": 6415}], "topic_id": 6232, "date_created": 1297813899.660707, "message": "a) even if development we're always using the \"collected\" dependencies. We have our own internal \"pypi\".. that has all the right versions. I don't think we actually use pypi anymore... unless every now and then we have a new dependency, or we need a newer version (which is rare) b) I'm buggered whether I'm deploying from my laptop or a deploy box in that case. In the case of moving from staging to live... it's fairly quick, so less of a window for my connection to screw up... Also, if the internet goes... well, it's always handy to have a dongle to stick in the side of your laptop, or just always do deploys from work :)", "group_id": 81, "id": 116885}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297812383.4028921, "message": "i was going to ask about example fabfiles but the one in http://www.clemesha.org/blog/modern-python-hacker-tools-virtualenv-fabric-pip seems like a basicmost start", "group_id": 81, "id": 116727}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297812401.658411, "message": "and is probably why i started with git archive", "group_id": 81, "id": 116729}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297814152.6514471, "message": "yeah, i guess i was thinking you'd use screen/tmux on the deploy box so if you were interrupted in the middle of a long automated deploy, it could keep going and when you did get network back you could rejoin your regularly scheduled deployment already in progress", "group_id": 81, "id": 116899}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297814437.198374, "message": "You were correct with your assumption in the previous message. I'm also working on building a little web app to keep track of metadata around deploys at work, so we know what went into each version, what is currently living on each environment... and so you can do a one click push if you want to move things across for those who like a nice GUI ;) can't get enough metadata when doing deploys, or investigating.", "group_id": 81, "id": 116920}, {"user_id": 7151, "stars": [], "topic_id": 6232, "date_created": 1297812239.566741, "message": "pip only installed compiled packages.", "group_id": 81, "id": 116715}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297811462.5885279, "message": "i usually do it manually (without fabric) cause i haven't had to use multiple servers, so i'd like to hear answers to this too", "group_id": 81, "id": 116597}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297812741.266284, "message": "+1 for tarballs (with prebuilt dependencies from your own internal pypi server) and rsync..", "group_id": 81, "id": 116770}, {"user_id": 141, "stars": [{"date_created": 1297827723.1862521, "user_id": 2458}, {"date_created": 1297831487.8674819, "user_id": 3705}], "topic_id": 6232, "date_created": 1297813519.291924, "message": "so you're, say, sshed into a deploy server where you use fabric to collect sdists for all your dependencies and push them to the prod servers, where (a) if PyPI is down or unreachable or suddenly missing the version of whatever you're using, you still have copies of all your dependencies, and (b) if your connection from your laptop to the internet goes away you don't end up partially deployed?", "group_id": 81, "id": 116873}, {"user_id": 214, "stars": [{"date_created": 1297812186.37041, "user_id": 141}, {"date_created": 1297827734.941227, "user_id": 2458}], "topic_id": 6232, "date_created": 1297811947.7284851, "message": "I've also decided that it's worth the effort to maintain a directory of tarball sdists for all  your dependencies. I maintain this in its own git repo, there could be various approaches. It makes the \"pip install\" portion of deployment (if you use --no-index, and --find-links to point at that directory of tarballs) blazingly fast, without the multiple SPOFs you get with a normal pip install from the net.", "group_id": 81, "id": 116659}, {"user_id": 7151, "stars": [{"date_created": 1297831430.177628, "user_id": 3705}], "topic_id": 6232, "date_created": 1297812218.9735501, "message": "We use a /vendor library that goes into the $ROOT of our app see: https://github.com/jbalogh/zamboni-lib", "group_id": 81, "id": 116711}, {"user_id": 214, "stars": [{"date_created": 1297811743.3151829, "user_id": 141}], "topic_id": 6232, "date_created": 1297811723.2670341, "message": "My current fabric scripts push to a checkout on the server, but I've been wanting to rewrite them to export+rsync; less fiddly. Uploading a tarball can be annoyingly slow for minor changes, which is why I'd go with rsync instead.", "group_id": 81, "id": 116634}, {"user_id": 7151, "stars": [], "topic_id": 6232, "date_created": 1297811824.985522, "message": "@carljm makes a good point about the slowness for one-off changes... so if this doesn't work for you, incrementally adjust it :) til it makes sense and is fast.", "group_id": 81, "id": 116639}, {"user_id": 7378, "stars": [{"date_created": 1297813769.992059, "user_id": 7722}, {"date_created": 1297855158.955843, "user_id": 978}], "topic_id": 6232, "date_created": 1297813263.2819171, "message": "@caseywstark Ideally at build time... which isn't on your live server. Failing that, ensuring you're not relying on anything outside of your network is a decent step. No idea why people think it's a wise idea to pull from pypi/github etc at deploy to live time... especially if they don't specify versions in requirements", "group_id": 81, "id": 116858}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297812808.5864339, "message": "If you're doing a pip install during deployment (moving from staging to live), you're doing it wrong.", "group_id": 81, "id": 116795}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297814466.4514921, "message": "*hands the thread back to everyone else*", "group_id": 81, "id": 116922}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297814356.383801, "message": "cool :)", "group_id": 81, "id": 116912}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297813333.820348, "message": "It's just asking for a kick in the nuts", "group_id": 81, "id": 116861}, {"user_id": 7378, "stars": [{"date_created": 1297813558.733341, "user_id": 141}, {"date_created": 1297813773.3047619, "user_id": 7722}], "topic_id": 6232, "date_created": 1297813432.3573551, "message": "and if your OS is different in development, staging and live... a kick in the nuts is probably the least of your problems :)", "group_id": 81, "id": 116862}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297813988.4537771, "message": "we live in the futurez people... your connection dies, another one is an unprotected wireless network away :)", "group_id": 81, "id": 116891}, {"user_id": 7911, "stars": [], "topic_id": 6232, "date_created": 1297813065.8395829, "message": "@rlofthouse so when do you install dependencies?", "group_id": 81, "id": 116840}, {"user_id": 275, "stars": [], "topic_id": 6232, "date_created": 1297813034.234817, "message": "I need to be able to bOokmark a topic in convore. Too much going on!", "group_id": 81, "id": 116834}, {"user_id": 7722, "stars": [], "topic_id": 6232, "date_created": 1297813825.2623651, "message": "I like the idea of deploying to a fresh folder and symlinking each time - provides an easier fallback path.", "group_id": 81, "id": 116884}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297813910.587847, "message": "that should be *even IN development", "group_id": 81, "id": 116886}, {"user_id": 7378, "stars": [{"date_created": 1297814226.351721, "user_id": 141}], "topic_id": 6232, "date_created": 1297814196.3468339, "message": "@markpasc Yeah, very true. I usually do use screen. Midnight blues setting in here for me in the UK :)", "group_id": 81, "id": 116903}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297813704.8729661, "message": "(i guess normally you'd already have collected the dependencies so that part wouldn't usually happen right at deploy time)", "group_id": 81, "id": 116881}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297814197.824065, "message": "is that at all what you meant by that, or is it more just the place that keeps the dependency archive?", "group_id": 81, "id": 116904}, {"user_id": 141, "stars": [], "topic_id": 6232, "date_created": 1297814497.1979899, "message": "nice!", "group_id": 81, "id": 116923}, {"user_id": 6415, "stars": [], "topic_id": 6232, "date_created": 1297826162.085551, "message": "@rlofthouse What server do you use for your internal \"pypi\"? Is it open source?", "group_id": 81, "id": 117474}, {"user_id": 205, "stars": [], "topic_id": 6232, "date_created": 1297827680.54741, "message": "pip/setuptools is pretty easy to please", "group_id": 81, "id": 117513}, {"user_id": 205, "stars": [{"date_created": 1297846484.840606, "user_id": 6415}, {"date_created": 1297873119.9472311, "user_id": 214}], "topic_id": 6232, "date_created": 1297827674.7819541, "message": "chishop works, as do tarballs in a directory listing", "group_id": 81, "id": 117512}, {"user_id": 214, "stars": [], "topic_id": 6232, "date_created": 1297873177.3644891, "message": "And using version-pinned dependencies everywhere is a no-brainer. If you have a single non-pinned dependency in your requirements file, you're just asking for trouble.", "group_id": 81, "id": 121086}, {"user_id": 214, "stars": [], "topic_id": 6232, "date_created": 1297873021.984935, "message": "And it allows you to use compiled packages without having to deploy from exactly the same architecture.", "group_id": 81, "id": 121070}, {"user_id": 214, "stars": [{"date_created": 1297885492.319037, "user_id": 594}], "topic_id": 6232, "date_created": 1297873220.8925769, "message": "Also (and this bites people who think they have everything pinned), make sure your requirements file contains _everything_, including dependencies of your dependencies. Otherwise the latter are effectively unpinned.", "group_id": 81, "id": 121088}, {"user_id": 214, "stars": [], "topic_id": 6232, "date_created": 1297872996.784162, "message": "@rlofthouse If you have a local directory with all your dependency tarballs, and you use --no-index, a pip install on the live server at deploy time becomes about as fast and as reliable as doing it on the deploy box, IMO.", "group_id": 81, "id": 121068}, {"user_id": 2588, "stars": [], "topic_id": 6232, "date_created": 1297875810.0065341, "message": "@carljm that last point is a great tip. Are you aware of any negative side effects to just using the output of `pip freeze` for a requirements file?", "group_id": 81, "id": 121426}, {"user_id": 214, "stars": [], "topic_id": 6232, "date_created": 1297877355.8415301, "message": "@unbracketed Nope, in fact that's what I'd recommend, once you have a working environment set up the way you want it. (This may get the stdlib's \"wsgiref\" in your requirements file; doesn't hurt to have it there, but you can also remove it).", "group_id": 81, "id": 121711}, {"user_id": 927, "stars": [], "topic_id": 6232, "date_created": 1297888087.4102931, "message": "+1 on using pip freeze to avoid surprises. It also helps provide quick correction when someone uses a no-site-packages venv by mistake", "group_id": 81, "id": 123848}, {"user_id": 1126, "stars": [], "topic_id": 6232, "date_created": 1297885677.1263731, "message": "it should also go without saying that you should be setting up your virtualenv without global site-packages, keep it completely sandboxed", "group_id": 81, "id": 123410}, {"user_id": 7378, "stars": [{"date_created": 1300295184.0241759, "user_id": 5778}], "topic_id": 6232, "date_created": 1297888246.1073151, "message": "You'd be amazed at what you assume is a no-brainer and people still do it wrong :)", "group_id": 81, "id": 123865}, {"user_id": 7378, "stars": [], "topic_id": 6232, "date_created": 1297888396.0823081, "message": "When it comes to deployment (and because I'm not always around) I prefer to reduce the steps to one click.... otherwise PEBKAC comes into play. The more commands you have to run, or individual components there are... the more likely it is that a human error will fuck you up :P", "group_id": 81, "id": 123896}]