convore.json/groups/arc-runtime-project/caching-compiles/messages.json

[{"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307728257.396436, "message": "This also doesn't let us cache compiles across runs of Racket, since the Arc compiler emits procedures and ar-tunnel's that aren't serializable.  This might be fixable... but again, another project.", "group_id": 9739, "id": 1363706}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307728071.127048, "message": "It might or might not also turn out to be possible to cache Racket's compile as well... but that would be another investigation.", "group_id": 9739, "id": 1363675}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307727967.352998, "message": "It loads the pre-compiled arc.arc in about 1 second vs. 4 seconds... which isn't exactly *fast*, but at least it's *faster*.  Doesn't pass all tests yet, but that's not surprising... I was actually surprised that it got as far along as it did on the first try.", "group_id": 9739, "id": 1363658}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307727848.394913, "message": "Initial stab at caching the compile of Arc into Ail: https://gist.github.com/1019335", "group_id": 9739, "id": 1363633}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1307756782.8838301, "message": "Argh, cached compilations. With Penknife, I was going to pursue hygiene in a way very similar to your embed-it-in-the-sexpr idea for ar, and then I wanted to do cached compilations for performance, and those were were totally incompatible with that. Complexity blossomed from there a bit as I added \"lexids,\" serializable paths for looking up indirectly-referenced variables from the runtime instance currently being generated from the compiled format. Lexid-laden variables (\"hypervars\") are looked up in \"hyperenvironments,\" even local-scope hyperenvironments, and to accomplish hygiene a new lexid is generated and passed to each macro application. A lot of complexity just for performance is unsettling, right? I actually kinda like those semantics (especially the idea that my program is normalized to something purer as part of the process), but I'm not sure how much it's worth it, and I was very inspired when you posted your hygiene article and I felt ar could manage to avoid that kind of rigamarole. ^_^", "group_id": 9739, "id": 1368098}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1307756979.8562801, "message": "Hmm, it might help to consider each sentence to be a separate paragraph there. ^^", "group_id": 9739, "id": 1368108}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307757213.79105, "message": "Hmm, well here caching the compile is simply saving the output of ac, so if ac produces the same output for the same input I imagine it will probably work.", "group_id": 9739, "id": 1368119}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1307757906.858005, "message": "That's what it is in Penknife too, basically. (The difference is that Penknife's \"fully\" compiled representation is Arc code rather than Racket code/bytecode.)", "group_id": 9739, "id": 1368157}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1307767111.861232, "message": "In less Penknifey terms: If macros expand into procedure literals, serializing the compiled form to disk is probably going to be a pain. If we take your approach to hygiene, our macros will expand into procedure literals. My point is to express that there's a significant challenge if we want to have both separate compilation *and* hygiene, and that while I have my own approach to this challenge, I don't know how to get separate compilation *and* hygiene *and* simplicity.", "group_id": 9739, "id": 1368560}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1307767294.2744629, "message": "(BTW, the two-hour delay between those posts was simply because I chose to split up my post into smaller pieces and I was interrupted before the second part. ^_^ )", "group_id": 9739, "id": 1368565}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307767817.848897, "message": "Ah yes, now that you mention it, I realize that caching compiles across runs of Racket would need serializing closures... so it won't be something I'm worrying about now ^_^", "group_id": 9739, "id": 1368583}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307812465.593013, "message": "aha, the reason why the \"dlet infile\" test fails is because when arc.arc is compiled, it creates a parameter for the \"infile\" dynamic variable.  This parameter is used in places where \"infile\" appears.  Then when the compiled code is loaded, it blindly makes another \"infile\" parameter.  The \"dlet infile\" in the test refers to the second parameter, but the compiled code is referring to the first parameter.", "group_id": 9739, "id": 1370533}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1307812862.832109, "message": "A general fix would be to make ac functional: to have it return the same result for the same input.  (currently it returns different parameters for a dynamic variable on input).  If that's not possible, a specific fix would be to make the non-functional parts aware of when it's loading previously compiled code.", "group_id": 9739, "id": 1370562}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308070586.6258669, "message": "#lang racket\n\n(define n1 (make-base-namespace))\n(define n2 (make-base-namespace))\n\n(namespace-set-variable-value! 'x 123 #t n1)\n(namespace-set-variable-value! 'x 456 #t n2)\n\n(define k\n  (parameterize ((current-namespace n1))\n    (compile 'x)))\n\n(parameterize ((current-namespace n2))\n  (eval k n2))\n\n\n\nprints: 123\n", "group_id": 9739, "id": 1392392}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308070333.054291, "message": "I took a look at whether Racket compiles could be cached.  I believe the answer is no: the Racket \"compile\" procedure (http://bit.ly/kGdBJ4) compiles code relative to the current namespace.  Thus if I say \"(compile 'x)\" with the current namespace parameterized to n1; and then eval the compiled result in n2 with (eval k n2), \"x\" still refers to the x in n1, not the x in n2.", "group_id": 9739, "id": 1392369}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308070682.4240191, "message": "(parameterizing current-namespace to n2 is redundant when passing n2 to eval, but I was just trying it out to make sure it didn't make a difference)", "group_id": 9739, "id": 1392399}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308074628.149775, "message": "I was afraid that caching compiles (even at the Arc level) might break having macros expand into functions or macros literals for hygiene (http://awwx.posterous.com/a-solution-for-using-macros-with-namespace-mo), but it seems to work:", "group_id": 9739, "id": 1392766}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308074630.293468, "message": "(use compile)\n\n(= r1 (runtime '(arc)))\n(= c1 (compile1 r1 '(def bar () \"version 1\")))\n(= c2 (compile1 r1 '(mac foo () `(,bar))))\n\n(= r2 (runtime '(arc)))\n(ar-racket-eval r2 c1)\n(ar-racket-eval r2 c2)\n(eval '(def bar () \"version 2\") r2)\n(= myfoo r2!foo)\n(erp (myfoo))\n\n\nprints:\n*** redefining bar\n(myfoo): \"version 2\"\n", "group_id": 9739, "id": 1392767}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308081326.1770451, "message": "\"Racket compiles can be loaded from disk\": oh, I'm not even trying to cache compiles to disk, either at the Arc level or at the Racket level.  We inject function values into the compiled output even with something as simple as (mac foo () `(,bar)), so we're not going to be able to save ac output to disk without serializing functions.", "group_id": 9739, "id": 1393546}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308081513.581578, "message": "\"...seems to work...\"\n\nThat example doesn't quite get to the kind of trouble I expect. I think compiled code that has *used* the first version of 'bar will keep referring to the first 'foo even if we evaluate it in the second runtime. Even if we use my `(,latemac.foo) spin on your hygiene technique to achieve late binding.", "group_id": 9739, "id": 1393571}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308081638.4026711, "message": "Isn't that true even without caching compiles?  Once a macro foo has been expanded, it's been expanded, and changing bar that point doesn't do anything.  The question is does caching compiles introduce any new problems...", "group_id": 9739, "id": 1393593}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308080783.7930551, "message": "\"...test fails is because...\"\n\nAn approach to that I'm considering for Penknife is to encourage the library programmer to store the parameter (or whatever else) in a global variable so that the application programmer can load the code in a spoof environment where that variable has a value already and does nothing when the library tries to overwrite it. Not that that's a good idea, necessarily. Common Lisp has 'defvar for a similar purpose, and an Arc programmer can say (unless bound!foo (= foo ...)) for a 'defvar -like behavior.", "group_id": 9739, "id": 1393457}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308081018.3834131, "message": "\"...whether Racket compiles could be cached...\"\n\nRacket compiles can be loaded from disk, I'm pretty sure, but that design's probably built around Racket modules.", "group_id": 9739, "id": 1393501}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308081208.645467, "message": "\"...test fails is because...\":  I fixed that in https://github.com/awwx/ar/commit/d3dc5ee4e26f7265e32b4c8cbbcec6e5f9462953", "group_id": 9739, "id": 1393532}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308085596.210737, "message": "@rocketnia \"figure out what you did there\": the problem was that \"foo\" expand into \"(#<parameter>)\" but the parameter wasn't tied to the current namespace.  So the trick was to have \"foo\" expand into \"(aparameter)\" where aparameter is a plain global variable containing the parameter (and thus is naturally tied to the namespace).  I also gave the global variable the same name as the implicit variable... which is probably confusing, but avoided introducing a global variable named \"__parameter_foo\" or something.", "group_id": 9739, "id": 1394210}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308083827.2786641, "message": "\"I fixed that...\": Hmm, it'll take me a bit more time to figure out what you did there.", "group_id": 9739, "id": 1393876}, {"user_id": 32737, "stars": [], "topic_id": 38950, "date_created": 1308084379.749177, "message": "Hey, awwx, why don't you just use (make-implicit cwd ...) rather than assigning it to a variable? I *especially* dislike hardcoding cwd into ac-global-assign... make-implicit should work better.", "group_id": 9739, "id": 1393972}, {"user_id": 32737, "stars": [], "topic_id": 38950, "date_created": 1308086313.965961, "message": "Personally, I would expect assignment to dynamic/implicit variables to work, so I'd suggest adding that back in. :P", "group_id": 9739, "id": 1394340}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308086204.1336679, "message": "@Pauan hmm, a couple things happened.  cwd didn't use make-implicit before, because it needed to do the string/path conversion, and I wasn't using make-derived-parameter.  Then I stopped having assigning to dynamic/implicit variables set the parameter value, because I didn't seem to be using it anywhere.  But now cwd *is* using make-derived-parameter, so it probably *could* use make-implicit, if I added back in the functionality of assigning to a dynamic/implicit variable.", "group_id": 9739, "id": 1394328}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308086477.610105, "message": "\"Isn't that true even without caching compiles?\": Well, it's a nonsensical issue until we draw some kind of useful separation between compile and execute phases so that we even care about executing the same compiled code more than once. In Arc application code, not only do we not care about what happens when we separate those phases, we don't separate them in the first place--at least not in any implementation-independent way. (We only have one phase: 'eval.) Ar may provide us with direct access to 'ac, but it isn't until we try something like cached compiles that we encounter this scenario in practice and get to decide whether it's good this way or not. So I do think cached compiles introduce this as a new issue.", "group_id": 9739, "id": 1394365}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308088273.024065, "message": "@rocketnia I guess I don't understand what you meant by \"code that has *used* the first version of 'bar will keep referring to the first 'foo even if we evaluate it in the second runtime\". What breaks if I cache compiles that works if I don't?\n\n\"nonsensical issue until we draw some kind of useful separation between compile and execute phases\" I'm not following you here either.  Right now if I say (runtime '(foo)) it's slow because it compiles all of Arc, and then if I say (runtime '(foo)) a second time it's slow *again* because it does the same work over again.  I'd like to cache the work of the compile if I can so that saying (runtime '(foo)) isn't as slow after the first time.  Do you some idea of where this might not work, or do you have some other goal (that caching compiles isn't a solution for)?", "group_id": 9739, "id": 1394642}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308093043.9308541, "message": "@Pauan ok", "group_id": 9739, "id": 1395467}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308119857.77775, "message": "This is what I fear will happen:", "group_id": 9739, "id": 1398360}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308119887.4766541, "message": "(use compile)\n\n(= r1 (runtime '(arc)))\n(= c1 (compile1 r1 '(def bar () \"version 1\")))\n(= c2 (compile1 r1 '(mac foo () `(,bar))))\n(= c3 (compile1 r1 '(def baz () (foo))))\n\n(= r2 (runtime '(arc)))\n(ar-racket-eval r2 c1)\n(ar-racket-eval r2 c2)\n(ar-racket-eval r2 c3)\n(eval '(def bar () \"version 2\") r2)\n(= mybaz r2!baz)\n(erp (mybaz))\n\n\nprints:\n*** redefining bar\n(mybaz): \"version 1\"", "group_id": 9739, "id": 1398362}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308121411.1005111, "message": "\"I'd like to cache the work of the compile if I can so that saying (runtime '(foo)) isn't as slow after the first time.\"\n\nYeah, exactly. In Lathe I do this by caching the resulting namespace, but you're going for (and I support) an approach where each importer gets a fresh instance of the runtime it can customize to its heart's content. Yet you'd rather not compile the code multiple times (and I sympathize), so you're talking about compiling the commands once and reusing the compiled versions. This means there's now a distinction between procuring something at compile time and embedding it as a literal, versus embedding a variable name and procuring its value at run time. The former uses the compile time namespace, while the latter uses the run time namespace and can happen more than once.\n\n\"Do you some idea of where this might not work, or do you have some other goal (that caching compiles isn't a solution for)?\"\n\nWhat doesn't work here is your approach to hygiene. It relies on the assumption that the compile time namespace and the run time namespace are the same.", "group_id": 9739, "id": 1398461}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308122392.1362879, "message": "Also for clarity's sake, yeah, that example I just gave would still break if we did it all in one namespace, 'cause we're embedding the value eagerly rather than using 'latemac (http://arclanguage.org/item?id=14521). If it helps, pretend I said `(,latemac.bar).", "group_id": 9739, "id": 1398537}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308122145.5933521, "message": "For clarity's sake, your approach to hygiene can work just fine if this phase separation doesn't exist.", "group_id": 9739, "id": 1398529}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308158243.945349, "message": "@rocketnia hmm, in plain Arc, (mac foo () `(bar)) works with redefining bar because foo expands into a symbol bar, which is interpreted as a variable reference in the (only) namespace.  To get this example to work with importing across runtimes, what we want is to have foo effectively expand into a reference to a variable in r1. (I interpret your latemac as one mechanism for doing this).\n\nThe output of ac can cached (memoized) if it's functional: if it produces the same output for the same input.  Implicit variables broke because ac was returning a reference to the actual parameter, so when the output of ac was loaded in a different runtime it was still referring to the parameter in the first runtime.  I said, \"how can I get the output of ac be functional while referring to a parameter that changes depending on which runtime we're in?\"  The answer there was to put the parameter into a global variable in the runtime.  (I happened to call it \"foo\", but it could have been e.g. \"__parameter_foo\").  Now ac can generate a literal \"(__parameter_foo)\" which is functional, and it automatically refers to the parameter in the right namespace.\n\nSo having a macro expand into what is effectively a variable reference to \"this\" namespace (instead of a variable reference in the namespace it is being imported into) doesn't strike me as being intrinsically incompatible with cached compiles: but we need to implement things like \"latemac\" in a functional way so that the \"this\" namespace ends up being the namespace that the compiled code is loaded into.", "group_id": 9739, "id": 1402493}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308160203.385402, "message": "For example, suppose we used one Racket namespace, and runtimes were implemented by giving symbols in different runtimes a different prefix.  (This would make garbage collecting runtimes hard so it probably isn't a very good way of implementing runtimes, but it's useful as a thought experiment).  So in this thought experiment, a global variable \"bar\" in runtime r13 would be the Racket global variable \"r13_foo\" in the single Racket namespace.\n\nThen we could imagine that there could be some mechanism in macro expansions to say that I wanted to refer to a variable in my runtime instead of the runtime the macro is expanding in.\n\n(mac foo () `(,this.bar))\n\nWhen I expand (foo) in runtime r14, I'd end up with a reference to the Racket global variable \"r13_bar\" instead of \"r14_bar\", which is what I would have gotten if I had simply had foo expand into the plain symbol \"bar\".\n\nOf course, the step of converting \"x\" into \"r13_x\" or \"r14_x\" clearly can't be cached, because it generates a different result depending on whether we're loading into runtime r13 or r14.  But that could be done as a final, non-cachable step after ac generated functional output indicating whether x refers to the target runtime that the code is being loaded into or some source runtime...", "group_id": 9739, "id": 1402680}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308158933.003973, "message": "One option that could be used if needed is to take the output of ac, and as a final step do a tree walk and convert global variables references into something specific to the runtime.  This final step wouldn't be functional and so couldn't be cached, but by taking the non-functional parts out of ac, that would make ac functional and so the functional parts could be cached.", "group_id": 9739, "id": 1402543}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308160630.8478251, "message": "I wonder how much slower Arc would run if I implemented global variables using boxes?  (http://bit.ly/m8Gg6B)  It would make it easy to say \"I want to refer to a global variable in this runtime or that runtime\", and also neatly solve the garbage collection problem.", "group_id": 9739, "id": 1402750}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308173026.1561179, "message": "But maybe it won't be as bad as I expect. For consistency in Penknife, I'm using lexid-tagged variables even for local scopes, but ar would achieve that kind of hygiene using 'w/uniq instead. There could be other simplifying details like that one.", "group_id": 9739, "id": 1404682}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308172979.6475191, "message": "\"One option that could be used if needed is to take the output of ac, and as a final step do a tree walk and convert global variables references into something specific to the runtime.\"\n\nThat's more or less my plan for Penknife. ^_^; There'll be the parse phase, which does the \"functional\" things, the resolve phase, which does that tree walk, and then the execute phase. Rather than using a simple global variable, I essentially use a path (\"lexid\") from variable to table key to table key, and I capture the environment in the macro itself so it can act as one of those tables. This way people can import a single macro without importing its dependencies. This is the kind of complexity I'm now skeptical about.", "group_id": 9739, "id": 1404677}, {"user_id": 32943, "stars": [], "topic_id": 38950, "date_created": 1308173301.744287, "message": "Also, lexids aren't all *that* bad, but they were frustrating to design in the first place, and I've never had high hopes that they'd be easy to explain. ^_^;", "group_id": 9739, "id": 1404742}, {"user_id": 25438, "stars": [], "topic_id": 38950, "date_created": 1308239458.7127321, "message": "ok, I think I understand the issues pretty well now :)  I'm going to put off working on caching compiles for the time being, and get some more basic issues done first.", "group_id": 9739, "id": 1411463}]