Skip to end of metadata
Go to start of metadata

Rationale

Much useful information is contained in Clojure's analysis compilation phase. There should be an interface to collect this information a la carte, in a nice form (ie. a map).

Steps

  • build an interface to the current Compiler
    • takes Form, results in Map
  • test the things we can do with this information, is it useful?
  • request to add some information to the compiler to provide missing information
  • iterate until useful

Progress

Building interface: https://github.com/frenchy64/analyze/blob/master/src/analyze/core.clj

Issues

Redefinition of Classes at Compile time

Running `clojure.core` through the analyzer attempts to redefine functions at (what seems to be) macroexpand time.

Offending line in `clojure.core`:

I don't have a good idea how to fix this. Maybe a flag that indicates to catch a redefinition error, and then lookup the class in the cache?

Labels:
  1. Nov 28, 2011

    (Take everything with a pinch of salt: half baked ideas below.)

    I'm unsure that substituting analyze with your own is desirable. Obviously you can do everything this way but the drawback is that you are on your own.

    For analysis purposes it often seems easier to work from a fully expanded tree (what the AST provides) – driving macroexpansion yourself seems brittle. In my opinion, if you need to record extra information, the analyzer should call your code and not the other way round.

    On the syntax front, metadata could be the answer: either on the ns or on individual forms (forms can be grouped with 'do).

    1. Nov 28, 2011

      Agreed, my approach seems brittle.

      I'm trying to connect the dots with the metadata idea. So when the analyzer comes across a particular metadata entry, it would call a function you produce? That sounds very nice.

      Here's a vague interpretation of that idea as a sketch.

      One thing I don't understand is how to keep these compiler hooks namespaced. I guess I haven't delved into that subject before.

      Is this close to what you were saying Christophe Grand?

      1. Nov 28, 2011

        Sort of. On metadata, I was thinking of using ^ and not with-meta so that metadata would be available on the form straight from the reader.

        ^::tc/type-check
        (defn-typed add [(a : Number) (b : Number)] : Number
        (+ a b))

        I don't understand your question  about keeping the compiler hooks namespaced: are you refrering to several hooks compteing for the same keyword or something else?

        The important part is that to me a hook should provide functions to be called by the analyzer but the analyzer should stay in control – to avoid duplicating the analyzer behavior in hooks.

        1. Nov 28, 2011

          Right, I was thinking something along those lines (competing hooks). I'm not sure if it's a problem, I was just putting it out there.

          How could we provide the actual functions? Via metadata again?

          1. Nov 28, 2011

            Before asking how at the syntax level, it could be good to sketch out what is our goal. There are three things I'd like to hook into the analyzer to do:

            • whole AST transform
            • defining new special forms
            • macroexpansion control (to enrich the AST with information that may be lost otherwise or to add sugar like the interop sugar)

            Plus I'd like to be able to feed the AST back into the analyzer but that's out of scope and just an AST -> sexpr converter away.

            Yours?

            1. Nov 28, 2011

              Wrt Typed Clojure, I want to:

              • add metadata to forms during compilation (such as preconditions)
              • add type information to a database during compilation, sourced from existing metadata
              1. Nov 28, 2011

                Are you only interested in top-level forms? (I don't think so)

                So basically you want to inser a function which sees every form and that may tweak them. You are not interested in the AST itself. Do you?

                1. Nov 29, 2011

                  Top level forms?

                  Not just top-level forms, no. The reason I brought it up was to mirror Racket's behavior with the #%top-interaction macro. But the metadata based approach is nicer and more general.

                  Typed Clojure

                  I'll flesh out some ideas.

                  Macroexpansion

                  I want to fully macroexpand forms to perform some basic type inference at compile time.

                  The idea is that we tell the type system how to reason about the special forms, then we reduce all code to those special forms, and perform the reasoning.

                  I am interested in the macroexpanded AST (if I understand you correctly).

                  EDIT: I misunderstood what the AST was referring to. No, I don't need to access the AST, just the Clojure form.

                  EDIT2: This should be a hook that gets passed the AST

                  Tweaking existing code

                  When importing functions from the untyped world, we want to add runtime pre/postconditions on the arguments at compile time.

                  Say I want the + function from the untyped world.

                  I want to (somehow) add run-time pre/postconditions on the argument types.

                  1. Nov 29, 2011

                    As far as I understand, require-untyped, reminds me of Fogus' Trammel (additionally you may add the sig to your types database). I don't see why you need compiler hooks for require-untyped.

                    1. Nov 29, 2011

                      You're right, I've got it backwards.

                      If you use a typed function from the untyped world, contracts must be added to protect the invariants defined by the type checker.

                  2. Nov 29, 2011

                    There's no such thing as a fully macroexpanded form (but you can build it from the AST) in CLJ or CLJS compiler

                  3. Nov 29, 2011

                    Why do you want the form and not the AST? If you don't have the AST you'll have to construct it yourself right?

                    1. Nov 29, 2011

                      Heh, I've spent today understanding what the AST gives you, you're correct of course. It's pretty awesome!

                      So I'd want a "post-analysis phase" hook that gets passed the resulting AST.

  2. Nov 28, 2011

    What does :compiler-hooks do that a require wouldn't?

    In what I envision a namespace register (through a aptly named register-compiler-hook function) a compielr extension associated with a keyword and then any ns which requires this ns can use this extension simply by adding metadata on form or by adding metadata at the ns-form level (won't work as is). However it's just the surface.

    1. Nov 28, 2011

      Yes, you're right. Your idea well and truly puts the "namespacing" issue I had previously to rest. I've updated the wiki.

      I'd imagine metadata at the ns-form level would mean wrapping the body of a file in an extra form? This behavior is more useful to me than adding metadata to individual forms. 

      1. Nov 29, 2011

        No whole-file wrapping because it goes against Clojure compilation model (one form at a time, the ns is not a file or a compilation unit). So it means additional compiler state.

        (Btw don't take my ideas or opinions as definite: I'm exploring this design too and can rapidly change my mind in face of compelling arguments.)

        1. Nov 29, 2011

          Also whole-file wrapping is just less necessary in Clojure. Definitions must appear in the proper order.

  3. Nov 29, 2011

    I still like

    (ns foo.bar
    (:analyzer clojure.typed/analyzer))

     

    1. Nov 29, 2011

      What would be the semantics?

      1. Nov 29, 2011

        the named analyzer (possibly analyzer post processor?) would be used for the given namespace

        1. Nov 29, 2011

          What if I don't want to use the analyzer (eg aggressivelly-deforesting-analyzer) for the whole namespace but only for a handful of defs? What if I want to use several analyzer?

          If it's about being obvious from the namespace that you are using a custom analyzer, wouldn't

          (ns my-user-ns
            {::tc/type-checked true}
            (:require [typed-clojure.core :as tc]))
           
          be enough to content you?
          If not, please explain what's lacking.
          1. Nov 29, 2011

            sure, a per form analyzer is strictly more powerful, but do you really want to allow mixing and matching of analyzers inside a form? inside a function? where does it end?

            namespaces seem like a nice granularity for declaring these options.

            this extension would allow you do to very interesting and cool things, but it would complicate the model of the clojure language.

            so how can we limit the scope of the complications while still enabling the power you get from being able to reach into the guts and make it do what you want.

            again scoping analyzers to a namespace seems like a good fit.

             

            EDIT:

            and the simplicity of the model (reading/macroexpansion/compilation/running) is a real strength of clojure.

            1. Nov 29, 2011

              If we eventually have multiple compiler extension points, we could take inspiration from Racket's #lang form to group extensions as a separate "language".

            2. Nov 30, 2011

              The reading/macroexpansion/compilation/running model is slightly a lie in Clojure: macroexpansion and compilation are intertwined, at no time you get a fully macro-expanded form that gets passed to the compiler (I'd like that but that's not the CLJ or CLJS compiler design).

              The true model is reading/analyze/compilation/running and macroexpansion is part of analyze.

              Strictlty speaking post-analyze hooks/AST transformers are as powerful as macros (you could do the same thing with macros but it would more brittle: you have to replicate all the macroexpansion/code-walking of the compiler) – it's just a saner way to implement such macros.

              If we have an analyze function and a reform function (which outputs forms from an AST) then

              would be functionally equivalent to a post-analyzer hook.

              Metadata tricks or ns declarations are just sugar on top of this functionality. This functionality being in the same class as macros there is no reason not to offer them at the form level. 

              For example I envision post-analyzers to be used to perform specific aggressive optimizations and I definitely don't want to apply them on a whole namespace or to reorganize all my namespaces to isolate the functions to optimize.

              To me the ns-level is too coarse but I'm more than willing to make sure the resulting model is understandable and its behavior well scoped.

              1. Nov 30, 2011

                what is your use case that makes ns-level post analyzers too coarse?

                1. Nov 30, 2011

                  For example to  "transientize" an expression: most expressions can't be converted and, really, it's not the kind of optimization I want to unleash on a whole ns. (Transients are going to be phased out but a t least transientization is an actual aggressive (possibly breaking) optimization.)

  4. Nov 29, 2011

    <<redacted>>

  5. Dec 04, 2011

    So, this seems to me to be immediately combining things. Should we not first:

    • make analyze independently callable a la carte
      • ensure it provides sufficient interesting information, in a useful form
    • experiment with things we can do with it
      • you can do type checking independent of compilation, there's not much reason to do it as you go, and doing so will only invite the problems of mandatory typing
    • then, and only then, consider extending the compilation process to interact with such things

    That is, the rationale is broken. What are the problems you are trying to solve?

    The first problems might be like this:

    • I want to write a type checker, IDE etc and need access to the knowledge of the compiler analysis phase, but can't get at it. Access to the analyzer helps solve this.

    Later, the problem might be:

    • I've written a type checker and I'd like to run it during compilation because (some good reason here).

    Making it pluggable because we can isn't going to yield good results.

    1. Dec 04, 2011

      Thanks for the feedback Rich.

      I'm taking a step back and exploring your first two points.