convore.json/groups/inscight/episode-6-cython/messages.json

[{"user_id": 20326, "stars": [], "topic_id": 13906, "date_created": 1300800518.5613019, "message": "The topic of \"How do I wrap my C, C++, and Fortran code?\" came up several times when scientific developers got together at PyCon, so I'm glad to see an episode of InScight dedicated to wrapping. I'm already convinced Cython is the best tool by far for wrapping C, but it will be interesting to hear what is discussed.", "group_id": 3435, "id": 406983}, {"user_id": 10411, "stars": [], "topic_id": 13906, "date_created": 1300817000.0893979, "message": "@mikegraham Yup just a week or so to go!", "group_id": 3435, "id": 409614}, {"user_id": 10348, "stars": [{"date_created": 1300822713.669801, "user_id": 10411}], "topic_id": 13906, "date_created": 1300819157.5527811, "message": "Cython's also quite good for providing bridges in the opposite direction; for instance, I'm currently using it to call Python functions from within C++.", "group_id": 3435, "id": 410189}, {"user_id": 20326, "stars": [], "topic_id": 13906, "date_created": 1300842335.5607009, "message": "@matthewturk: Sounds like a personal problem. ;-)", "group_id": 3435, "id": 413188}, {"user_id": 10411, "stars": [], "topic_id": 13906, "date_created": 1300847167.0469289, "message": "@mikegraham Yikes!", "group_id": 3435, "id": 413560}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1300896735.972903, "message": "@mikegraham back in summer of '09, two Cython GSoC projects addressed Cython-C++ and Cython-Fortran wrapping/interoperability.  Cython has the C/C++/Fortran trifecta!", "group_id": 3435, "id": 418488}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1300897033.4216001, "message": "What are the barriers to using Cython from a user's perspective?  IOW, what are the primary annoyances that prevents someone from cythonizing: documentation, compilation issues & distutils, not knowing enough C, debugging, solution X is better...", "group_id": 3435, "id": 418518}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1300897097.7885821, "message": "@kwmsmith then again, the C++ and Fortran interop are still in their infancy, and are at alpha/beta stage, while Cython's C support is mature.", "group_id": 3435, "id": 418525}, {"user_id": 10411, "stars": [], "topic_id": 13906, "date_created": 1300909474.099967, "message": "@kwmsmith The C++ stuff is 85% there.  And the 15% that is missing you can work around.", "group_id": 3435, "id": 420122}, {"user_id": 10348, "stars": [], "topic_id": 13906, "date_created": 1300922874.0878, "message": "@scopatz I had the same experience.  After 0.13, we have been able to use pretty heavily templated, namespaced and class-based code really, really easily with Cython.  Frankly I was a bit surprised how easy it was ...", "group_id": 3435, "id": 421758}, {"user_id": 20326, "stars": [], "topic_id": 13906, "date_created": 1300974433.095228, "message": "@scopatz Sometimes.", "group_id": 3435, "id": 426721}, {"user_id": 20326, "stars": [], "topic_id": 13906, "date_created": 1300974556.3884699, "message": "@kwmsmith I'm not sure why you were tagging me. I was just giving matthewturk a hard time\u2014note the emoticon", "group_id": 3435, "id": 426732}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301076073.03214, "message": "@mikegraham I was replying to your first post.", "group_id": 3435, "id": 440798}, {"user_id": 20326, "stars": [], "topic_id": 13906, "date_created": 1301248564.972162, "message": "@kwmsmith Oh, that makes more sense.\nThat being said, I abbreviated to \"C\" in part because of well-suitedness. The C++ ride can be a little bumpy depending on what you're wrapping. There are more than a few cases where the 15% mentioned above become a real problem; most of the ones I've encountered are really more about using a manual-memory-management library with a garbage collected end language, but they're resolved in well-written C API wrappers in ways that Cython won't let you.", "group_id": 3435, "id": 452248}, {"user_id": 22813, "stars": [], "topic_id": 13906, "date_created": 1301289866.7675769, "message": "Interestingly, I hadn't even thought about looking into what's borrowable in FWrap for this. It would be nice to a wrapper generator toolchain for Cython which supports the most common languages.", "group_id": 3435, "id": 455449}, {"user_id": 22813, "stars": [], "topic_id": 13906, "date_created": 1301289808.68047, "message": "I'm working on a side project at the moment which is attempting to autogenerate Cython wrappers from header files. Lots of things are straight forward, but there's that 10% area like opaque structs which are difficult to detect and expose in a Pythonic way. When hand-writing the wrappers, this is easy enough to take care of. But hand wrapper large libraries is prohibitive.", "group_id": 3435, "id": 455443}, {"user_id": 10411, "stars": [], "topic_id": 13906, "date_created": 1301449176.857075, "message": "@sccolbert What we should probably discuss in the next episode:\n\n* What is Cython (ie what a creole language is)\n* NumPy/SciPy for .NET\n* FWrap\n* Autowrapping idea\n* Why we love Cython.\n\nThis is supposed to be about the cool things that you can do with Cython, and an introduction to it.  We probably won't discuss how it is different from PyPy or other VMs.", "group_id": 3435, "id": 474892}, {"user_id": 26515, "stars": [], "topic_id": 13906, "date_created": 1301475319.0979769, "message": "I have a C++ project which controls humanoid robots and is used by students with no/little C++ experience. I'm thinking about porting it to Cyphon/Python and recycle the performance critical parts (vision, localization, motion control). Still, I'm a bit hesitant because I can't estimate possible problems with the GIL, multithreading in Cython/C++ applications in general and performance evaluation in such a system. So I'd really appreciate it if you talked about MT and performance evaluation.", "group_id": 3435, "id": 479855}, {"user_id": 20326, "stars": [{"date_created": 1301499627.1120961, "user_id": 10411}], "topic_id": 13906, "date_created": 1301498851.9589159, "message": "@otte Cython can call C/C++/Fortran functions while releasing the GIL, so that multiple threads execute at the same time (at most one of them touching Python objects). The implementation is pretty easy, too; it looks like a context manager\u2014put non-Python-touching code you want to execute with the GIL released in a \"with nogil:\" block.\nI hope you are also aware of the other parallelization and concurrency options, since threads end up being a suboptimal tool for almost every task. Threads are used for two things:\n1) Concurrency, doing multiple tasks at once. Concurrency allows us to speed up IO-bound code by not holding up the whole process for things that depend on one another. Threads were originally developed for this task and had in mind single-core processors. For IO-bound concurrency problems, the GIL doesn't really affect you, per se. \nIt turns out, though, that threads are a far-too-low-level, ill-performant way to do concurrency. It makes programs harder to extend, write, and debug. Event-driven, non-blocking IO is a superior approach to spawning threads for IO. \n2) Parallelization, spreading out a task's work on multiple processors. Parallelization allows us to speed up CPU-bound code by providing more resources for computations. This is probably the main use scientific programmers thinks of for multithreading. \nIn my experience and observation, threads are almost always the wrong thing for this. Threads are hard/impossible to control compared to processes and don't scale onto multiple machines when you need to spread out work further. Shared memory is error-prone so much that it is virtually always worth avoiding, at least in the uncontrolled, unmodular way it is done using threads. Shared memory (on modern architectures) is also not as helpful as we would imagine; I know I've seen more than a couple programs where MPI parallelization gave better performance than OpenMP parallelization.", "group_id": 3435, "id": 482315}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301501243.8249209, "message": "@sccolbert there was some talk on the Cython ML about this a while ago; *many* projects need a good, robust C/C++ header parser and AST generator written in Python & with a BSD-like license.  They seemed to indicate that robust C/C++ parsing was the bulk of the problem; once that's done, generating wrappers for X language would be doable.  Fwrap uses a Fortran parser that's pretty good, but doesn't yet do derived types, which get hairy really quickly.", "group_id": 3435, "id": 482634}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301501801.524585, "message": "@sccolbert regarding a Cython wraper-generator toolchain: it's a good idea.  Fwrap does really simple wrapper generation, nothing too sophisticated or general.  Fwrap currently is kind of a frankenstein project, as it had F77 support shoehorned in recently and needs some TLC.  I've wanted to generalize it so it's usable from other languages.  One issue is that there's much overlap with projects like http://code.google.com/p/pybindgen/ (does for C what fwrap does for fortran, but without a Cython layer) so I'm not sure about interest or adoption.", "group_id": 3435, "id": 482736}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301502462.442688, "message": "how do I get linebreaks in a chat post?  Hitting return submits what I've written so far...", "group_id": 3435, "id": 482816}, {"user_id": 10411, "stars": [], "topic_id": 13906, "date_created": 1301502780.8344319, "message": "@kwmsmith Shift+Enter", "group_id": 3435, "id": 482861}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301502328.0110171, "message": "@scopatz One thing I'd like to discuss is Cython usability, from an end-user perspective: what are those annoying aspects of Cython that require significant workarounds, boilerplate, etc?  And what are the Cython killer features that would benefit from more development and attention?  Cython's getting much attention and development, and I'm interested in putting in time on the usability side as contrasted with the core features side, which is the Cython core developer's focus.  For example: many people don't like having to mess with distutils and setup.py scripts for Cython, so they use pyximport, but pyximport has limitations.  Some people use Cython's 'pure' mode, but it isn't documented as well as it should be.  I've started a project in its infancy called 'cythonize' that is sort of a mashup of Cython, psyco, and scipy.weave -- it's focused on usability rather than Cython features.", "group_id": 3435, "id": 482792}, {"user_id": 22813, "stars": [], "topic_id": 13906, "date_created": 1301607068.491364, "message": "@kwmsmith I was playing around with PyParsing, PyCParser, Ply and various combinations of those. It was really difficult to get something robust. The AST generated by PyCParser, while nice and complete, is not really nice for converting to Cython. It also needs all the type defintions from all includes in order to parse a file and you lose locality information. \n\nI just built GCC-xml and now using that in conjuction with ctypeslib.codegen. The AST created there is very nice and workable. The only issue is that gcc xml parses the headers as if they were c++, so things like typedef'd anonymous structs get parsed incorrectly according to C standards, and I havent found a clever way of dealing with that, short of reading the line number of header and resolving it that way.", "group_id": 3435, "id": 496677}, {"user_id": 11542, "stars": [], "topic_id": 13906, "date_created": 1301607125.6361301, "message": "Discussion subtopics:\n\n * Cython used to fill the vectorization/for-loop gap in scipy/numpy, for the cases when an algorithm can't be easily or cleanly vectorized, or when memory concerns are an issue.  IOW, \"just give me a fast for loop, dammit!\"\n\n * Python + Cython + C: allows one to mostly ignore C++ and STL programming.  Superior language combo to C++?", "group_id": 3435, "id": 496690}, {"user_id": 22813, "stars": [], "topic_id": 13906, "date_created": 1301608020.3997011, "message": "Another subtopic:\n\n* Cython is useful for far more than just numeric code. Cython's support for python's builtins (lists, dicts, sets) is superb. There are optimizations in place for iterating, enumerating, etc... For example, you can iterate over a cdef'd list in Cython nearly as fast as traversing a pure C-Array in a loop.", "group_id": 3435, "id": 496844}, {"user_id": 22813, "stars": [], "topic_id": 13906, "date_created": 1301607902.3085461, "message": "@kwmsmith Cython doesn't complete replace the STL as it doesn't have templating support. That doesn't mean its not possible and I know there has been some discussion about that on the Cython lists. It would definitely be a feature I would like to see added.", "group_id": 3435, "id": 496814}]