convore.json/groups/scikitsimage/gsoc-meeting-5/messages.json

[{"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792090.6835849, "message": "You there Pieter?", "group_id": 9959, "id": 1560195}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792090.8807869, "message": "I thought he went to Berkely for a post-doc?", "group_id": 9959, "id": 1560196}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792091.060859, "message": "well, let's go ahead and get started then.", "group_id": 9959, "id": 1560197}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792101.9581101, "message": "Hey Chris", "group_id": 9959, "id": 1560206}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309792165.1871769, "message": "I see it.", "group_id": 9959, "id": 1560228}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792101.6125269, "message": "My responses might be a bit delayed, I have work going on in another remote session", "group_id": 9959, "id": 1560205}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309792174.263504, "message": "And some things twice.", "group_id": 9959, "id": 1560232}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309791660.974879, "message": "hello.  Is Stefan going to join us?", "group_id": 9959, "id": 1560145}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309791890.678885, "message": "ah, he's still in the US?", "group_id": 9959, "id": 1560172}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309791617.9446509, "message": "Hello guys.", "group_id": 9959, "id": 1560131}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309791730.9597199, "message": "howdy", "group_id": 9959, "id": 1560152}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309791752.631603, "message": "Stefan might be busy enjoying the festivities of the 4th", "group_id": 9959, "id": 1560155}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309791972.7599609, "message": "Stefan is out and about in the world, couldn't yet have a good meeting with him yet. Let's hope he can join us soon :)", "group_id": 9959, "id": 1560188}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792145.6586599, "message": "I'm seeing some delay in the conversation.", "group_id": 9959, "id": 1560225}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792146.134541, "message": "and I'm not quite sure if you guys are receiving what I type.", "group_id": 9959, "id": 1560226}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792225.3852701, "message": "ok", "group_id": 9959, "id": 1560241}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309791763.7924149, "message": "Me, I'm lame and work too much, so here I am :)", "group_id": 9959, "id": 1560158}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792105.7722771, "message": "Hehe we can shoot some fireworks for you", "group_id": 9959, "id": 1560208}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792106.73557, "message": "When I meet with Stefan we will discuss your suggestion of specifying a backend more explicitly in the __init__ file", "group_id": 9959, "id": 1560210}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792107.2162049, "message": "As I understand he's going to be there for quite a while :) Postdoc at Berkeley", "group_id": 9959, "id": 1560211}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792107.705631, "message": "And we will also then discuss how backends will operate with binary packaging .", "group_id": 9959, "id": 1560212}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792173.0697141, "message": "Do you guys have gtalk accounts?", "group_id": 9959, "id": 1560231}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792217.9497769, "message": "pieter, i'll let you set it up", "group_id": 9959, "id": 1560240}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792207.576194, "message": "mine is sccolbert@gmail.com", "group_id": 9959, "id": 1560239}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792090.106457, "message": "hopefully Convore cooperates better this week", "group_id": 9959, "id": 1560193}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792106.252543, "message": "I did fix the backend tester to skip on import errors. So at least that is dealt with.", "group_id": 9959, "id": 1560209}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792167.5036371, "message": "wow, ok convore is bunk. I'm gonna setup a gtalk session", "group_id": 9959, "id": 1560229}, {"user_id": 33626, "stars": [], "topic_id": 41116, "date_created": 1309792136.7919681, "message": "Stefan is out and about in the world, couldn't yet have a good meeting with him yet. Let's hope he can join us soon :)", "group_id": 9959, "id": 1560216}, {"user_id": 22813, "stars": [], "topic_id": 41116, "date_created": 1309792138.7684319, "message": "hopefully Convore cooperates better this week", "group_id": 9959, "id": 1560221}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309792295.6212909, "message": "yes.", "group_id": 9959, "id": 1560251}, {"user_id": 34630, "stars": [], "topic_id": 41116, "date_created": 1309792300.400579, "message": "thouis@gmail.com", "group_id": 9959, "id": 1560255}, {"user_id": 33559, "stars": [], "topic_id": 41116, "date_created": 1309845041.286787, "message": "5:11 PM Chris has joined\n  Thouis has joined\n me: Ok let's hope this goes better :)\n Thouis: yes.\n5:12 PM me: Ok so just repeating what I said in convore\n  I did fix the backend tester to skip on import errors. So at least\nthat is dealt with.\n5:13 PM when I meet with Stefan we will discuss your suggestion of\nspecifying a backend more explicitly in the _init_ file as Chris\nsuggested\n  and how backends will operate with binary packaging .\n5:14 PM This week I looked a bit at DFT work and got the pyfft library\ngoing in comparison with opencv\n5:15 PM Chris: which fft lib does pyfft wrap?\n me: but I'm getting the impression though that to get gains with gpu\ncomputations you would have to be set it up quite tailored for a\nspecific problem\n  CUDA and CL\n  yeah that is on the gpu\n  Data transfer to the gpu card is quite expensive, so for small data\nsets it will always be slower than say opencv\n5:16 PM Chris: yeah. I would be surprised for most normal sized images\nto see a big speedup over the stuff in OpenCV/IPP\n  Intel spent a lot of money making that stuff fast\n me: For a 4096x4096 image\n  opencl calculation time 0.0111520290375\nopencl time 0.226531028748\nopencv time 1.07513904572\n  You can see that opencl is getting faster\n Chris: well, 5x improvement is no slouch\n me: the calculation time is the time of actual calculation after\nbuffer setup and transfer\n5:17 PM but yes for smaller images it is quite slower\n Chris: is that using a generic routine, or hand tuned code?\n Thouis: are buffer setup/transfer and calculation pipeline-able?\n5:18 PM and if so, for thing like video applications is it something\nthat would be useful?\n5:19 PM me: It is a port of a fft opencl routine that Apple released\n  So I guess it would be fast\n5:20 PM Pipeline-able? It needs to happen at once. It is most\nefficient if you transfer multiple batches over and then do the\ncalculation on it all.\n5:21 PM Thouis: But can buffer setup, transfer, and calculation be\ndone separately?\n5:22 PM Perhaps I'm misunderstanding something.\n me: I think they do follow sequentially\n5:23 PM Chris: what about setups with mutli-gpus? Can you use them\nindividually, like from threads?\n me: Yeah that shouldn't be a problem.\n Thouis: And if I have a large stack of images to process, can I start\nthe setup phase of image 2 while image 1 is being processed?\n5:24 PM me: Not according to my understanding...but I'll check. That\ncould help for videos yes.\n5:26 PM I'm starting to wonder if GPU versions of scikits.image\nfunctions would give us much use. It seems very specific to certain\nproblems.\n Chris: What size images are people typically processing?\n me: But yeah I'll play around some more and find something. Fourier\nroutines would be helpful for the scikit in any case.\n5:27 PM Chris: I never do much above 2048\n Thouis: That's about our typical upper limit, too\n Chris: I would be most interested in FFT and Convolution\n me: Well I'm used to process video hanging around 640x480\n  but getting disappointed with my gpu workings on that res ;)\n Thouis: But we've done some work on 10k x 10k histology sections\n5:28 PM Chris: Pieter, what might help is looking into which CPU-based\nlibs we can wrap via Cython and release the GIL while it does the\nwork, thus allowing real threading.\n5:29 PM at least on separate images\n  (to help with video)\n me: Would opencv still hold the GIL?\n5:31 PM I'm also interested in convolution work. Using it for template\nmatching and so.\n5:32 PM Chris: Pieter I don't know if the offical OpenCV Python\nbindings hold the GIL, the ones scikits.image do not\n  in fact, if you run Scivi on a large image and plan around with the\nColor correction, you'll notice that it utilizes multi-core\n me: If I understand it correctly, something like sobel uses direct\ncorrelation, while Opencv uses the DFT for template matching routines\n5:33 PM Chris: it does that by chunking the image into multi\nsub-images, and spawning a thread pool\n  Pieter, OpenCV switches between spatial and frequency domain\ncorrelation depending on the size of the image and kernel\n  the switch is automatic for whichever will be most efficient\n5:34 PM me: Ah indeed\n5:35 PM when I browsed the source it looked straightforward dft use. I\nguess for typical images that conversion would be most efficient.\n5:36 PM Thouis: The threading question can probably be answered using\na simple test with multiple threads trying to process images.\n5:37 PM me: Yup will check.\n5:38 PM I'm also going to start looking at the Hough transform, it is\none of the items on my list.\n  I would like to see some code getting into the lib :)\n5:39 PM Sooner rather than later of course\n  Just a question: I see that rgb2grey function returns non-contiguous arrays\n5:40 PM That threw quite a wrench into my gpu and cv work\n5:41 PM It is non-contiguous of course, because there is code similar\nto \"return image[:, :, 0]\"\n Chris: we have a hough transform\n me: I'm just wondering what would be policy\n Chris: which I recently improved\n me: Probablistic Hough\n Chris: only numpy version though\n  ah!\n  nvm :)\n me: I'll try to backendize as I go along :)\n  http://simplecv.org/\n5:42 PM That is interesting site\n  That library aims to wrap rather than implement, but I like its user\nfriendly approach.\n5:44 PM Another thought:\n  I see for the color conversion routines we assume float input between 0 and 1\n5:45 PM Is that a good general rule of thumb for the scikit?\n Thouis: not always\n Chris: yeah, that's a tricky topic\n Thouis: If an algorithm doesn't require 0-1 (e.g., FFT), it shouldn't\nbe limited to it.\n5:46 PM Chris: I think the other libs have it (more or less) right,\nwith different funcs for different input types\n Thouis: unless there are performance consideration\n me: Yeah performance can be an issue with float calcs.\n5:47 PM Opencv color converts handles each type differently\n  Compressing hue into 180 for example for bytes\n Thouis: probably best to concentrate on the most general versions\nthat can benefit from a backend, then think about type and encoding\nspecific versions\n5:48 PM or approximate ones\n me: If you load a png image, that will give you 255 per pixel.\n  The fact that the convert functions required floats was a bit\nnon-intuitive for me.\n5:49 PM Normalized floats\n5:52 PM Thouis: I'm happier working with floats. But if there are\nalgorithms that only make sense in 0-255 or 0-2**16-1, and you can\nconvert in a reasonable way, the approximation is probably fine\n5:56 PM me: Perhaps what would work is imread returning floats by\ndefault except as otherwise specified. Or a flag to enable norm\nfloats. Or perhaps just a normalize function that is used often.\n  Anyway, thanks for joining. I'll send transcript over to Stefan. Cheers\n5:57 PM Thouis: talk to you later\n  thanks\n Thouis has left", "group_id": 9959, "id": 1565248}]