Feature Expressions

I am not wild about the conditional stuff happening at read time, it means the conditionals have no data representation and cannot be manipulated as such

I received an email from Kevin Lynagh. He could not write to the wiki, so I post it here:

Hi Roman,

• I like the keywords :clojure and :clojurescript, that is how the dialects are known.
• I don't think it makes sense to add e.g. :rhino until/unless there is something that can be read only there.
• AFAIAC namespace support means just "allow namespaced names", which the attached patch handles just fine.
• I think compilation process and naming questions can be considered separately.

> Are those keywords ok? Is :jvm for Clojure and :js for ClojureScript better? 

> Should ClojureScript add something like :rhino, :v8 or :browser as well?

:jvm isn't a good idea for Clojure, since :rhino is also on the JVM. If you were writing some ClojureScript code, and wanted to test for a Java package, you'd want to check for :jvm. Similarly, you'd want a :gclosure member as well, if there was ever an implementation that targeted JS without it.

I'd imagine the Clojure set could look like #{:clojure :java :jvm}

And the ClojureScript one could be #{:clojurescript :js :gclosure} potentially conj :jvm for Rhino

Because it will soon be possible to have #{:clojurescript :lua}

This raises an interesting point: Why a set? Why not a map? Here's an example Clojure map:

{:dialect :clojure, :clojure {:version "1.2.3"}, :target :java, :runtime :jvm, :jvm {}}

And one for ClojureScript:

{:dialect :clojurescript, :clojurescript {:version "2.3.4"}, :target :js, :js {}, :runtime :browser} ; maybe something for :gclosure version

This way the #+clojurescript and #-js things will still work like set memberships. We could come up with some way to include an arbitrary predicate against map values, or we could leave that to be feature not available to the reader, only to macros against *features*.

A few issues: Since this happens at read time, it may be too late to decide :v8 vs :rhino, etc. Although, some JavaScript shops/frameworks/teams/whatever do multi-compile sources for different browsers. ie. Do conditional logic by browser version, server side, rather than client side, serve up the browser-specific javascript. In that case, you even could support a :browser key with the various information there.

I feel like this needs some kind of cond macro. I should be able to check for "do I have a runtime that optimizes foo? if so, do this, otherwise, do it the slow way like this" I realize that I could use #+foo, then #-foo, but what if I had two possible optimizations? I'd need #+foo(x) #+bar(y) #-foo(#-bar(y)) ;; seems like I need some kind of "else" expression.

RE: read-time vs macro-expansion-time. Seems like the right thing for this is read-time because of symbol resolution, but we should also consider some compile-time macro support against the same dynamic var. In theory, it should be workable with all the standard macros and stuff, since it's just a set or map, but we should still explore it a bit, so we know we're not missing something.

I asked the Common Lisper's for some input over here:

https://groups.google.com/forum/?hl=en&fromgroups#!topic/comp.lang.lisp/x2k3XbW3LmA

Elias Mårtenson responded to one of my questions on
comp.alt.lisp. Here's what he wrote:

I can address one of the quest, whether they are better
implemented as a macro. The answer to this is clearly no. In
fact, it couldn't be implemented as a macro, because at the time
the macro is evaluated, the symbols making up the expression has
already been interned by the reader. Thus, if the form references
any packages that does not exist in your environment, you would
get an error in the reader, before your conditional expression
has had the chance to run.

A somehow related thread I found on this topic is here:

https://groups.google.com/forum/?hl=en&fromgroups#!topic/comp.lang.lisp/CBB4hzqRCS8

What are the next steps?

I would like to see the feature system being open to use by libraries. In pallet, we recently introduced features to be able to write providers that work with multiple versions of pallet, and I think it would make sense to replace this with whatever feature system ends up in clojure.

The current proposal seems to support this, but it would be good to clarify this as a requirement of the feature system.

What problems, if any, could this change cause for existing code? Obviously pre-feature-expression code will be unable to read the forms. In particular:

• 1.4 Clojure and ClojureScript fail to read feature expressions, thinking they are tagged literals with no handler
• pre-1.4 Clojure fails with a ClassNotFoundException

What if we provided read-time variants of unquote and unquote-splicing?

Here's a splicing example:

The code inside the #~ and #~@ forms would be run at read-time and subject to *read-eval*. This gives you the full power of Clojure for conditional compilation at read time. We could provide some standard function and macro utilities for conditional compilation that are available both at read and macro expansion time. Allowing for something like:

This proposal gives you the full power of Clojure for controlling conditional reading and compilation without adding dramatically new ideas (it's basically just syntax-quote at read-time).

pushing conditionals in to the reader does not save us from requiring the code to be readable on all platforms. it has to be readable for the reader to even recognize it as a form to throw away

if any recursive call to read can (some how?) returning multiple values I think the entire reader would be much more complex.

I guess recursive invokations of read could be replaced with read-seq-of-forms + concat et al.

That would buy us read-splice, but: Is it worth the performance cost? If yes, should read-seq-of-forms be made public?

I guess at the top level, a splice would just assume an implicit "do".

I'm kind of meh on (= (read-str "#~@(range 5)") '(do 1 2 3 4 5))

If I want a do, I'll write (a regular form generating) it. `~@ sets a nice precedent.

Those complications make me wonder if #~@ serves any conceivable purpose, that #~ together `(~@) can't (or shouldn't) serve.

pushing conditionals in to the reader does not save us from requiring the code to be readable on all platforms. it has to be readable for the reader to even recognize it as a form to throw away

True, this should be an intended design constraint. We want to have the same basic clojure syntax on every platform, i.e. a very small superset of edn, right?

The reason we even need conditionals in the reader is that the reader already does some possibly platform-dependent side-effects, i.e. reader tags, defrecord instances and friends.

So let's just KISS the syntactic considerations goodbye for a moment:

That gets us to the real meat of the problem:

Say we have:

Under current reader rules, would the #platform tag have a chance to remove the form before the reader tried to apply the #asm tag?

I suspect that's not the case, which makes me wonder if it's worth changing the reader to allow reader tags to omit reader tags from their child form. This would imply the ability to generate new reader tags (and other reader side effects) in code a reader tag emits.

I also suspect that if we try to just allocate a couple of new reader table characters  and work downwards from there, we will end up with pretty much the same code it would take to apply reader tags in a second pass, except it would be less general purpose.

We still might not want to expose it through reader tags, since it would alter their semantics on the input side. Opinions on this one?

This proposal additionally adds a lot of complexity to tools which work on unexpanded source, such as Cursive. In particular, with complicated conditional feature expressions it may no longer be possible to accurately identify which language a particular form is written in. Admittedly this is already a problem with CLJS macros, the only way to tell if the body of a syntax-quote block in a macro is Clojure or CLJS is to look at where the macro is actually used - I'd definitely welcome any steps to make that easier to test.

This isn't just applicable to Cursive, any tool which interacts with a debugger and allows breakpoints to be set will need to know the language of a particular line, since Javascript breakpoints will almost certainly be managed differently to JVM ones.

The lack of an explicit extension for mixed-mode files means that all tooling will have to assume that all Clojure files might potentially be treated as CLJS or code for any other platform - this hugely complicates things like completion and documentation lookup since docstrings may differ for a particular form across platforms. Currently things like numeric operators are functions in Clojure, and both functions and macros in CLJS. Identifying which the user is referring to is already complicated but with this change will be considerably more difficult. I much prefer the patch style system proposed by Timothy Baldridge where code is contained in platform specific files and combined somehow - this does sound like it would create a lot more code duplication, though.

I'd also like to ensure that the implementation will still maintain line/column information accurately, as CLJX currently does. I assume that's fairly easy, at least with the read time proposal.