Scala Macros: “Oh God Why?”

The blog post and tweets seem to say the Scala community only cares about fancy language stuff and not about the developer experience. Nothing could be further from the truth. Take Typesafe for example. We have three full time engineers on staff to improve the Scala IDE for Eclipse. We made a lot of progress and are continuing to do so. Nobody pays us for any of that; we do it because we know that IDE experience is crucial for Scala developers. Typesafe is just one company and it cannot pour unlimited resources into all aspects of Scala development, so we need to rely on the community for that. And the community does step up to the task. One of the criticism was on documentation. I wonder whether people have recently looked at <a href="http://docs.scala-lang.org/" rel="nofollow">http://docs.scala-lang.org/</a> ? In my mind, that's community-driven development at its best!<p>Why macros? I was a long-time sceptic. I now tend to think about them differently because I believe we hit on a brilliantly simple scheme that can express a lot of different use-cases. In particular we'll be able to do the analogue of Microsoft's LINQ with macros. And we can express optimizations such as turning foreach applications into while loops. And we can remove a lot of compiler plugins. And finally it looks like we can remove down the road several special cases in the language and the compiler. So macros might pay for their own complexity handily. But I should also say that at present this is a SIP, a Scala Improvement _Proposal_. It's not yet accepted. And part of the current proposal is also that macros will be enabled only under a special compiler flag.

The bug in this post is thinking that if you have someone willing to contribute feature X to an open source project, they were also available to contribute feature Y.<p>Open source patches are all itch-scratchings. For Scala macros, it's a researcher at EPFL, I believe. I don't think you could write a thesis about fixing bugs in the IDE. So this contributor was not available to submit IDE bugfixes; they _were_ available to submit macros.<p>It's the same for other contributors. Someone building an app on Scala may be available to submit a fix for some scalability issue they are hitting in production; they will not be available to work on macros, or on IDE bugfixes.<p>Open source projects get the fixes they get. You can't act like the priorities are set by some central product manager.<p>So I think it's just not correct to argue that "the priorities are wrong, why are they doing this instead of that, etc." - everyone is doing what matters to them. Some people _are_ working on IDE bugfixes, other people are working on macros. People can do what they want.

Where I work (i.e. write code for money, to satisfy Ted Nyman's requirement that my opinion matter), our reaction was exactly the opposite--we're very much looking forward to having macros in Scala.<p>If this was a more esoteric language feature, I'd be inclined to agree, but macros (if done well), I think will be a huge boon to the language.<p>I do all sorts of hackery in Java now (external code generation, heavily using annotation processing tools (APTs) which are a very, very limited form for language-based code generation) to do semi-metaprogramming stuff and can't wait to do the real thing.<p>(And do it well, e.g. not to make things "even more cryptic" just for the fun of it, but to solve real problems.)

Macros make language design experimentation easier. This issue is not orthogonal to tackling the other issues the OP cares about.<p>There are good reasons to doubt the overall value of macros in production code - I won't argue this either way - but OCaml has had success by separating the macro expansion mechanism (camlp4) from the main language. People who want macros can have them, people who want a production language without macros can have that too.

"I understand macros to provide two things (1) forced code in-lining and (2) a kind of non-strict semantics or lazy evaluation of arguments"<p>False. There isn't much else to say, but if this is all he thinks macros are about, he doesn't know anything about them, their uses, or their potentials.<p>Macros would basically take a large amount of nasty extra-lingual stuff (code-generation, byte-code generation, compiler plugins) and codify it into a systematic, easier to use and easier to understand framework.<p>F#-style Type Providers? Compile-time typechecked regex literals? Compile-time typechecked html? Compile-time typechecked CSS? This sort of thing is already being done (see Play!) and is pure awesome as a user (i &#60;3 the type-checked HTML templates) but the implementation involves a huge amount of mysterious extra compilation steps and magic that macros could help simplify and codify.

I seems to be a rather common tension in open source projects developed by research institutions.<p>The goal of users of those projects is immediate usability which clashes with goals of developers, research.<p>I have some doubts where one can get PHD (publish an article with high citation index) by improving a compiler error messages. As opposed to adding a new cool feature to the language/compiler, cryptic error messages be damned.

The listed points are valid, but they are by far not as easy as the author thinks they are. I don't say they are not improvable, but they are not 'little things' that need to get fixed before new features come in. For example, the jars are large because every closure needs its own class (thank you jvm!). Compile time is high because of type inference. Error messages and stack traces are cryptic, because you do high-level programming but get low-level diagnostics. etc.

The author doesn't even know what macros are: "I understand macros to provide two things (1) forced code in-lining and (2) a kind of non-strict semantics or lazy evaluation of arguments"

The post raises a few good points -- those are real problems in using Scala -- but the problems are mostly a result of the compiler targeting the JVM as its primary backend. Obviously, a JVM-based language can leverage a rich infrastructure of libraries, but it also will have to deal with the fact that the JVM is primarily designed for Java, not as a more general backend (such as the LLVM or the CLR).<p>(That the compiler is slow has other reasons, but is also something that they've been working on simultaneously for 2.10, so it's not as though that has been pushed aside by macros.)

I'd like to single out this line from the linked article:<p>"Unlike Java, type signatures in Scala don’t really explain idiomatic usage to mortals very well."<p>This is a great diagnosis of one of the big issues with Scala's style of static typing. Everyone knows that the type descriptions take a bit to grok compared to a C-style languages. Why? Well, in C-style languages, a function definition is itself a representation of idiomatic usage. In Scala and its brethren, you have to translate from one to the other.<p>I'm sure that's a commonplace observation, but it had not jumped out at me like that before.

For those curious about what Scala macros actually are: "This facility allows programmers to write macro defs: functions that are transparently loaded by the compiler and executed during compilation. This realizes the notion of compile-time metaprogramming for Scala." <a href="http://bit.ly/zeMjOb" rel="nofollow">http://bit.ly/zeMjOb</a><p><a href="http://scalamacros.org/usecases/index.html" rel="nofollow">http://scalamacros.org/usecases/index.html</a>

Macros are useful in Scala. Mostly not for application developers, but for library or framework developers. It enables you to do things while keeping the typesafety nature that makes Scala so great.<p>In Play, we have generated code for forms validation and json serialization. This enables the compiler to see code that can't work, while a more dynamic approach would lead to an exception at runtime.

For those interested the next release of Rakudo will be the first to (partially) implement Perl6 macros.<p>ref: <a href="http://strangelyconsistent.org/blog/macros-progress-report-d1-merged" rel="nofollow">http://strangelyconsistent.org/blog/macros-progress-report-d...</a>

I think many people share some concerns about macros, especially the need for good error messages. A far as I know Scala's language designers have been against including macros for a long time. So the proposal for macros has to be damn good if they change their opinion.<p>Macros are supposed to be a replacement for writing compiler plugins in many cases (compare with Java annotation processing vs. Java compiler plugins). The API is easier and more straightforward, semantics are easier to understand and don't require separate setup step.<p>In the meantime work has of course not stopped:<p>- Syntax has been simplified and several confusing bits have been removed/deprecated (Octal literals, FP literals without digit after dot, ...).<p>- The jar size of the library has been reduced, while adding fixes and functionality.<p>- Value classes have been integrated, which provide the performance of primitives with user-defined classes.<p>- Inlining has been improved a lot.<p>- Compilation speed has been improved a lot.<p>- Some collection classes have been performance optimized and are now a lot faster (and sometimes faster than the Java implementation).<p>- Play 2.0 and Akka 2.0 have been released, both with huge amounts of documentation.<p>- Documentation has been improved.<p>- ...<p>My personal impression is that while Scala has roots in academia, solving real-world issues is the focus of the team.