convore.json/groups/python/no-serious-project-should-ever-depend-on-the-os-distros-python-package/messages.json

[{"user_id": 6543, "stars": [{"date_created": 1299386102.799036, "user_id": 1}, {"date_created": 1299387210.6339381, "user_id": 8391}, {"date_created": 1299396318.7908549, "user_id": 11592}, {"date_created": 1299414666.2889071, "user_id": 18347}, {"date_created": 1299417338.9736509, "user_id": 5778}, {"date_created": 1299422703.4397161, "user_id": 927}, {"date_created": 1299447632.7017169, "user_id": 6415}, {"date_created": 1299980663.49453, "user_id": 12791}], "topic_id": 11053, "date_created": 1299383865.6579189, "message": "virtualenv rules", "group_id": 292, "id": 278510}, {"user_id": 18972, "stars": [{"date_created": 1299601573.2071681, "user_id": 1376}], "topic_id": 11053, "date_created": 1299383005.111706, "message": "...or anything else significant in the distro. It locks you into one version of one distro and makes changing that in the future painful. Control the _entire_ software stack for your application.", "group_id": 292, "id": 278470}, {"user_id": 1152, "stars": [], "topic_id": 11053, "date_created": 1299383561.694257, "message": "Yes. 1000x yes. There's some benefit to using what \"a million other people are using\" e.g. - kernels in major distros - there's stability in that, but it locks you into their \"one way\" of doing things.", "group_id": 292, "id": 278494}, {"user_id": 1152, "stars": [{"date_created": 1299383647.691057, "user_id": 1289}, {"date_created": 1299387206.5878971, "user_id": 8391}, {"date_created": 1299396315.981282, "user_id": 11592}, {"date_created": 1299417337.019419, "user_id": 5778}, {"date_created": 1299601571.9064519, "user_id": 1376}, {"date_created": 1299980658.8401461, "user_id": 12791}], "topic_id": 11053, "date_created": 1299383576.479969, "message": "Also, shit is *always* out of date. Always.", "group_id": 292, "id": 278496}, {"user_id": 5701, "stars": [{"date_created": 1299392129.7229581, "user_id": 1736}, {"date_created": 1299423035.5927191, "user_id": 927}, {"date_created": 1299429407.8754399, "user_id": 4156}, {"date_created": 1299454277.662617, "user_id": 6671}], "topic_id": 11053, "date_created": 1299389868.9144411, "message": "Or your serious project should work in a wide variety of environments and not required the version of Python released last Tuesday.", "group_id": 292, "id": 278699}, {"user_id": 18972, "stars": [{"date_created": 1299419125.519505, "user_id": 1152}], "topic_id": 11053, "date_created": 1299391797.2280021, "message": "By a \"serious project\" I'm thinking of custom server side applications. Not a library or framework. Don't tie yourself to a distro when you start. If your project evolves to need easy deployment by others or by spawning off component libraries for use by a wider variety of projects (open source or not) you can decide what their minimum requirements for stand alone use of those parts should be at that time.", "group_id": 292, "id": 278775}, {"user_id": 1736, "stars": [{"date_created": 1299422594.7156351, "user_id": 927}, {"date_created": 1299454682.5410521, "user_id": 9229}], "topic_id": 11053, "date_created": 1299392232.2838249, "message": "It is precisely this kind of thinking that has left Python web turnkey deployment in the dust. I very much hope people will learn to work with distros (example, use virtualenv for local pure-python deps but work with any PIL in recent packages) instead of in spite of them.", "group_id": 292, "id": 278794}, {"user_id": 18347, "stars": [{"date_created": 1299565774.853456, "user_id": 6661}], "topic_id": 11053, "date_created": 1299414745.11818, "message": "@chrismcdonough OS Distro < OS < virtualenv < ... what comes after virtualenv? :D", "group_id": 292, "id": 280003}, {"user_id": 1152, "stars": [], "topic_id": 11053, "date_created": 1299419157.691505, "message": "@jpcalderone what @gpshead said. We're not talking distributed apps here.", "group_id": 292, "id": 280082}, {"user_id": 214, "stars": [{"date_created": 1299422602.939677, "user_id": 927}, {"date_created": 1299429461.66225, "user_id": 4156}, {"date_created": 1299436885.3283651, "user_id": 5778}, {"date_created": 1299454669.995218, "user_id": 9229}, {"date_created": 1299458582.1123099, "user_id": 3617}, {"date_created": 1299605601.7424719, "user_id": 12677}, {"date_created": 1299613989.243402, "user_id": 1963}, {"date_created": 1299639163.6564691, "user_id": 15291}], "topic_id": 11053, "date_created": 1299422382.855032, "message": "Blanket statements about what serious projects should or should not do are dumb. Assumptions about what constitutes a \"serious\" project based on your personal work experience, likewise. Know your situation, know what axes you need flexibility on, and choose the right solution for your job.", "group_id": 292, "id": 280177}, {"user_id": 927, "stars": [], "topic_id": 11053, "date_created": 1299422859.5769701, "message": "What @coderanger and @carljm said: your local conventions and needs are not universal. I've seen a few projects where someone's desire to run the latest and greatest burned weeks of support time and had absolutely no benefit beyond over the OS packages.", "group_id": 292, "id": 280183}, {"user_id": 927, "stars": [{"date_created": 1299423330.9745171, "user_id": 214}], "topic_id": 11053, "date_created": 1299423131.6968901, "message": "There's a related issue which is that when you do need something different, such as the [rather rare] case where you actually require bleeding-edge Python, using the OS packaging mechanism is a much better choice than tossing everything in /usr/local: it's verifiable, repeatable and a LOT more appealing to any good sysadmin.", "group_id": 292, "id": 280191}, {"user_id": 1152, "stars": [], "topic_id": 11053, "date_created": 1299425452.677947, "message": "@acdha Sure, there's benefit to running what \"a million other people\" are running - the problem is that it locks you into the stack, which for something that's somewhat complex (isn't just python) getting locked into a single vendor's packages, versions and solutions can cause some serious pain.", "group_id": 292, "id": 280352}, {"user_id": 1152, "stars": [], "topic_id": 11053, "date_created": 1299425521.082629, "message": "@acdha Remember, @gpshead and I are not talking about pure python applications. When I say stack, I mean everything from the kernel all the way up to the version of python and apache (and dependencies) they use.", "group_id": 292, "id": 280361}, {"user_id": 1736, "stars": [], "topic_id": 11053, "date_created": 1299432415.2273409, "message": "@jessenoller I'm confused, how does using a slightly older version of something cause pain? I can't remember any time since 2.0 when I actually cared what version of Apache I was using. I can understand the point for either very young stuff (we build our own package for Redis because it changes so much) or major versions that won't be included because of LTS restrictions (Postgres 9.0, mod_wsgi 3) but we thought long and hard before committing to deviating from the OS packages for those and it is a non-trivial amount of work to do so (and we might even stop using mod_wsgi because of it).", "group_id": 292, "id": 281044}, {"user_id": 927, "stars": [], "topic_id": 11053, "date_created": 1299432875.4956119, "message": "@jessenoller I've just found that there's a rapidly decreasing likelihood that you'll need to make changes as you go down that stack and that doing so outside of the vendor's mechanism tends to cause problems later. e.g. it's easy to build a website with generic Apache config that'll work unmodified using Red Hat, CentOS or OS X's default Apache - taking on the responsibility of managing all of that yourself just to have the same version number hasn't been a win in my book", "group_id": 292, "id": 281091}, {"user_id": 927, "stars": [], "topic_id": 11053, "date_created": 1299433514.9591031, "message": "I also don't agree with the lock-in argument: if you have to support a platform, using their tools rather than duplicating them is almost a side issue to whatever compels you to actually upgrade a system package. e.g. if I need a custom Apache module, shipping it as a .rpm or .deb rather than rolling my own system using something like buildout or fabric shouldn't be more work than the actual custom module.", "group_id": 292, "id": 281140}, {"user_id": 927, "stars": [], "topic_id": 11053, "date_created": 1299433563.436028, "message": "The difference being that if you do something like fork a Debian/Ubuntu package, send the patch upstream and install from a PPA, you're a lot less likely to need to repeat the process every time some other update comes out", "group_id": 292, "id": 281144}, {"user_id": 6543, "stars": [{"date_created": 1299496990.7538431, "user_id": 18539}, {"date_created": 1299508634.7800419, "user_id": 214}, {"date_created": 1299511730.6382849, "user_id": 12404}, {"date_created": 1299554396.358444, "user_id": 18972}, {"date_created": 1299556868.2011449, "user_id": 927}], "topic_id": 11053, "date_created": 1299475504.427, "message": "if you dont have a way to repeatably build and deploy your software in an automated way, you've already lost.", "group_id": 292, "id": 285742}, {"user_id": 18539, "stars": [], "topic_id": 11053, "date_created": 1299497481.791575, "message": "My talk about Continuous Deployment talks about this subject. Distro packages certainly aren't all bad, but there are a few things that a sandbox (for Python, that pretty muh means a tox-built virtualenv to me) can do better, particularly falling back to an older version. Since distro packages are typically fairly global in scope, you'd need per-deployment virtual machines or chroots to be able to just drop one and go back to what you had instantly without having affected it. Unfortunately those typically have a bunch of performance/memory use/practicality problems on its own, plus it compounds the vendor lock-in since they're usually as if not more platform specific, not to mention that you'll probably need to hook python dependencies up to your distro package manager, unless suddenly pip et al and rpm et al become mutually aware -- something which I think is a WIP but doesn't actually work yet. This all becomes much more interesting when you, like my old project, start adding different stuff to the mix (node.js, Erlang, a bunch of JVM languages) which typically have their own ideas about building and package management and unfortunately don't always map as cleanly to distro packages as I'd have liked them to.", "group_id": 292, "id": 286980}, {"user_id": 18972, "stars": [], "topic_id": 11053, "date_created": 1299554581.891428, "message": "The reason I state that is that any \"serious\" project grows beyond what any distro will ever provide in terms of requirements, schedule, bugs, etc.  Yes, if you have to make changes to a component you should absolutely send those upstream. It reduces your long term maintenance costs. But by suggesting that you've already forked an existing distro package to apply your own patch(es) you are already effectively requiring that the distro packages be part of your project/organization/company's version control and are no longer blindly accepting them from the distro update mechanism. That is an important leap to make.", "group_id": 292, "id": 293168}, {"user_id": 1736, "stars": [{"date_created": 1299557111.8779421, "user_id": 927}], "topic_id": 11053, "date_created": 1299554769.830966, "message": "@gpshead To say that you might want to personally manage a small number of very critical dependencies I might buy, but you said \"anything\" and that is pure nonsense.", "group_id": 292, "id": 293178}, {"user_id": 1736, "stars": [{"date_created": 1299557113.120683, "user_id": 927}], "topic_id": 11053, "date_created": 1299554852.571913, "message": "@gpshead I don't compile my own kernel, or manage a patch queue for Apache, or track security issues for Postgres, and yet all of those components are quite critical to me. I trust in Ubuntu to Do The Right thing and they are a pretty good track record saying they will.", "group_id": 292, "id": 293184}, {"user_id": 927, "stars": [{"date_created": 1299557959.2946751, "user_id": 1736}], "topic_id": 11053, "date_created": 1299557148.3585849, "message": "In addition to @coderanger's points, the other major objection I have with the extremist position is the degree of difference: forking a distribution package still means that I'm getting e.g. Debian Apache with one mission-critical patch but I'm reusing the standard mechanisms for everything else, my sysadmin won't be confused by non-standard tools or config locations, etc. and if I do something like pull the next release out of the OS' testing repo I'll actually acquire that version at some point in the future rather than having to maintain it myself.", "group_id": 292, "id": 293331}, {"user_id": 18972, "stars": [], "topic_id": 11053, "date_created": 1299563472.0749359, "message": "I don't really consider it an extremist position but I can easily see how many people not accustomed to working on huge deployments or self contained environments see it that way. I've worked on both sides. When your own software becomes the most significant thing thing that you run, the rest diminishes in value and real production issues start happening on a large enough scale to make you need to control every last little bit for both sanity and cost.", "group_id": 292, "id": 294493}, {"user_id": 18972, "stars": [], "topic_id": 11053, "date_created": 1299563797.390017, "message": "Take convore for example. Today its a startup probably running on random rackspace machines (or, gasp, poor little VMs) most likely running stock distros with Python, Ruby or Java VMs installed. If it goes on to grow as large as twitter I guarantee you they'll stop relying on distros for their software stack and pull in everything they depend on into their code tree. I simply advocate doing that early on. For open source projects its still easy to release and say you've been tested on X, Y and Z versions A B and C while also stating the more lenient versions you believe the software works on (soon confirmed by others). But there is a difference between working at all and working like a well tuned machine.", "group_id": 292, "id": 294534}, {"user_id": 1736, "stars": [{"date_created": 1299569239.0554259, "user_id": 5778}, {"date_created": 1299571755.2543399, "user_id": 5582}, {"date_created": 1299583597.2910049, "user_id": 6415}, {"date_created": 1299592869.973156, "user_id": 214}, {"date_created": 1299600422.0321691, "user_id": 4156}], "topic_id": 11053, "date_created": 1299567589.6898279, "message": "@gpshead How deep does that rabbit hole go. Do you run your own kernel fork? I know for a fact that Google does, and since there is a non-zero probability that I might need some random kernel feature before Ubuntu packages it does that I mean I should start now? There is such a thing as proportional response. I'm not saying that no one needs to care about these things, but I certainly don't and the vast majority of Python programmers (startup or otherwise) don't. Trying to claim that this is some kind of magnanimous one-size-fits-all strategy is straight lunacy. For example, Twitter runs on a highly customized interpreter tuned to their exact needs, Convore would be out of their gourd if they forked Python and decided to \"fix\" things. They don't have the man power, the money, or the time. Maybe if they get as big as Twitter they will find they have all three of those things, and then it will make perfect sense.", "group_id": 292, "id": 294744}, {"user_id": 18972, "stars": [], "topic_id": 11053, "date_created": 1299610562.933511, "message": "I am on Google's Linux kernel team. :) But my reason for making this statement comes from experience prior to Google. Start with a distro's packages? Great, that gets you going! But don't insist that the distro Python is what you should always follow. Compiling and distributing your own is not difficult and should not be seen as a problem. Do it at the first sign that you need anything that the distro lacks. I'm not suggesting making major changes to it; just that you control it as part of your source tree rather than as part of the distro so that you evolve at your own pace. It makes for a much more hermetic build in the long run that is easier to stand up and run on any system.", "group_id": 292, "id": 298484}, {"user_id": 927, "stars": [], "topic_id": 11053, "date_created": 1299613652.885705, "message": "@gpshead I think we're not that far apart: I'd describe it as sticking with a stock package until you have an actual need. I haven't run as many systems as you have (low hundreds) but found it relatively rare to need to fork that much. Maybe a custom kernel (esp. for #$%@#$% driver issues) and a handful of other packages, and even there it's been a site-wide package rather than project specific.", "group_id": 292, "id": 298864}]