Rationale
Much useful information is contained in Clojure's analysis compilation phase. There should be an interface to collect this information a la carte, in a nice form (ie. a map).
Steps
- build an interface to the current Compiler
- takes Form, results in Map
- test the things we can do with this information, is it useful?
- request to add some information to the compiler to provide missing information
- iterate until useful
Progress
Building interface: https://github.com/frenchy64/analyze/blob/master/src/analyze/core.clj
Issues
Redefinition of Classes at Compile time
Running `clojure.core` through the analyzer attempts to redefine functions at (what seems to be) macroexpand time.
Offending line in `clojure.core`:
I don't have a good idea how to fix this. Maybe a flag that indicates to catch a redefinition error, and then lookup the class in the cache?
Labels:
29 Comments
Hide/Show CommentsNov 28, 2011
Christophe Grand
(Take everything with a pinch of salt: half baked ideas below.)
I'm unsure that substituting analyze with your own is desirable. Obviously you can do everything this way but the drawback is that you are on your own.
For analysis purposes it often seems easier to work from a fully expanded tree (what the AST provides) – driving macroexpansion yourself seems brittle. In my opinion, if you need to record extra information, the analyzer should call your code and not the other way round.
On the syntax front, metadata could be the answer: either on the ns or on individual forms (forms can be grouped with 'do).
Nov 28, 2011
Ambrose Bonnaire-Sergeant
Agreed, my approach seems brittle.
I'm trying to connect the dots with the metadata idea. So when the analyzer comes across a particular metadata entry, it would call a function you produce? That sounds very nice.
Here's a vague interpretation of that idea as a sketch.
One thing I don't understand is how to keep these compiler hooks namespaced. I guess I haven't delved into that subject before.
Is this close to what you were saying Christophe Grand?
Nov 28, 2011
Christophe Grand
Sort of. On metadata, I was thinking of using ^ and not with-meta so that metadata would be available on the form straight from the reader.
I don't understand your question about keeping the compiler hooks namespaced: are you refrering to several hooks compteing for the same keyword or something else?
The important part is that to me a hook should provide functions to be called by the analyzer but the analyzer should stay in control – to avoid duplicating the analyzer behavior in hooks.
Nov 28, 2011
Ambrose Bonnaire-Sergeant
Right, I was thinking something along those lines (competing hooks). I'm not sure if it's a problem, I was just putting it out there.
How could we provide the actual functions? Via metadata again?
Nov 28, 2011
Christophe Grand
Before asking how at the syntax level, it could be good to sketch out what is our goal. There are three things I'd like to hook into the analyzer to do:
Plus I'd like to be able to feed the AST back into the analyzer but that's out of scope and just an AST -> sexpr converter away.
Yours?
Nov 28, 2011
Ambrose Bonnaire-Sergeant
Wrt Typed Clojure, I want to:
Nov 28, 2011
Christophe Grand
Are you only interested in top-level forms? (I don't think so)
So basically you want to inser a function which sees every form and that may tweak them. You are not interested in the AST itself. Do you?
Nov 29, 2011
Ambrose Bonnaire-Sergeant
Top level forms?
Not just top-level forms, no. The reason I brought it up was to mirror Racket's behavior with the #%top-interaction macro. But the metadata based approach is nicer and more general.
Typed Clojure
I'll flesh out some ideas.
Macroexpansion
I want to fully macroexpand forms to perform some basic type inference at compile time.
The idea is that we tell the type system how to reason about the special forms, then we reduce all code to those special forms, and perform the reasoning.
I am interested in the macroexpanded AST (if I understand you correctly).EDIT: I misunderstood what the AST was referring to. No, I don't need to access the AST, just the Clojure form.EDIT2: This should be a hook that gets passed the AST
Tweaking existing code
When importing functions from the untyped world, we want to add runtime pre/postconditions on the arguments at compile time.
Say I want the + function from the untyped world.
I want to (somehow) add run-time pre/postconditions on the argument types.
Nov 29, 2011
Christophe Grand
As far as I understand, require-untyped, reminds me of Fogus' Trammel (additionally you may add the sig to your types database). I don't see why you need compiler hooks for require-untyped.
Nov 29, 2011
Ambrose Bonnaire-Sergeant
You're right, I've got it backwards.
If you use a typed function from the untyped world, contracts must be added to protect the invariants defined by the type checker.
Nov 29, 2011
Christophe Grand
There's no such thing as a fully macroexpanded form (but you can build it from the AST) in CLJ or CLJS compiler
Nov 29, 2011
David Nolen
Why do you want the form and not the AST? If you don't have the AST you'll have to construct it yourself right?
Nov 29, 2011
Ambrose Bonnaire-Sergeant
Heh, I've spent today understanding what the AST gives you, you're correct of course. It's pretty awesome!
So I'd want a "post-analysis phase" hook that gets passed the resulting AST.
Nov 28, 2011
Christophe Grand
What does :compiler-hooks do that a require wouldn't?
In what I envision a namespace register (through a aptly named register-compiler-hook function) a compielr extension associated with a keyword and then any ns which requires this ns can use this extension simply by adding metadata on form or by adding metadata at the ns-form level (won't work as is). However it's just the surface.
Nov 28, 2011
Ambrose Bonnaire-Sergeant
Yes, you're right. Your idea well and truly puts the "namespacing" issue I had previously to rest. I've updated the wiki.
I'd imagine metadata at the ns-form level would mean wrapping the body of a file in an extra form? This behavior is more useful to me than adding metadata to individual forms.
Nov 29, 2011
Christophe Grand
No whole-file wrapping because it goes against Clojure compilation model (one form at a time, the ns is not a file or a compilation unit). So it means additional compiler state.
(Btw don't take my ideas or opinions as definite: I'm exploring this design too and can rapidly change my mind in face of compelling arguments.)
Nov 29, 2011
David Nolen
Also whole-file wrapping is just less necessary in Clojure. Definitions must appear in the proper order.
Nov 29, 2011
Kevin Downey
I still like
Nov 29, 2011
Christophe Grand
What would be the semantics?
Nov 29, 2011
Kevin Downey
the named analyzer (possibly analyzer post processor?) would be used for the given namespace
Nov 29, 2011
Christophe Grand
What if I don't want to use the analyzer (eg aggressivelly-deforesting-analyzer) for the whole namespace but only for a handful of defs? What if I want to use several analyzer?
If it's about being obvious from the namespace that you are using a custom analyzer, wouldn't
(ns my-user-ns
{::tc/type-checked true}
(:require [typed-clojure.core :as tc]))
Nov 29, 2011
Kevin Downey
sure, a per form analyzer is strictly more powerful, but do you really want to allow mixing and matching of analyzers inside a form? inside a function? where does it end?
namespaces seem like a nice granularity for declaring these options.
this extension would allow you do to very interesting and cool things, but it would complicate the model of the clojure language.
so how can we limit the scope of the complications while still enabling the power you get from being able to reach into the guts and make it do what you want.
again scoping analyzers to a namespace seems like a good fit.
EDIT:
and the simplicity of the model (reading/macroexpansion/compilation/running) is a real strength of clojure.
Nov 29, 2011
Ambrose Bonnaire-Sergeant
If we eventually have multiple compiler extension points, we could take inspiration from Racket's #lang form to group extensions as a separate "language".
Nov 30, 2011
Christophe Grand
The reading/macroexpansion/compilation/running model is slightly a lie in Clojure: macroexpansion and compilation are intertwined, at no time you get a fully macro-expanded form that gets passed to the compiler (I'd like that but that's not the CLJ or CLJS compiler design).
The true model is reading/analyze/compilation/running and macroexpansion is part of analyze.
Strictlty speaking post-analyze hooks/AST transformers are as powerful as macros (you could do the same thing with macros but it would more brittle: you have to replicate all the macroexpansion/code-walking of the compiler) – it's just a saner way to implement such macros.
If we have an analyze function and a reform function (which outputs forms from an AST) then
would be functionally equivalent to a post-analyzer hook.
Metadata tricks or ns declarations are just sugar on top of this functionality. This functionality being in the same class as macros there is no reason not to offer them at the form level.
For example I envision post-analyzers to be used to perform specific aggressive optimizations and I definitely don't want to apply them on a whole namespace or to reorganize all my namespaces to isolate the functions to optimize.
To me the ns-level is too coarse but I'm more than willing to make sure the resulting model is understandable and its behavior well scoped.
Nov 30, 2011
Kevin Downey
what is your use case that makes ns-level post analyzers too coarse?
Nov 30, 2011
Christophe Grand
For example to "transientize" an expression: most expressions can't be converted and, really, it's not the kind of optimization I want to unleash on a whole ns. (Transients are going to be phased out but a t least transientization is an actual aggressive (possibly breaking) optimization.)
Nov 29, 2011
Kevin Downey
<<redacted>>
Dec 04, 2011
Rich Hickey
So, this seems to me to be immediately combining things. Should we not first:
That is, the rationale is broken. What are the problems you are trying to solve?
The first problems might be like this:
Later, the problem might be:
Making it pluggable because we can isn't going to yield good results.
Dec 04, 2011
Ambrose Bonnaire-Sergeant
Thanks for the feedback Rich.
I'm taking a step back and exploring your first two points.