Exploring Fantom DSLs

{toc}

For built-in DSLs, the expression type matches to the anchor type, so that

regex := Regex<|\d+|> // regex has type Regex

However, it is not necessary - basically the DSL expression can be compiled to expression of any type - you can write a DSL plugin with return type depending on DSL str, like this:
{noformat}
foo := MyDSL<|1|> //resolves to Int at compile time
bar := MyDsl<|str|> //resolves to Str at compile time
{noformat}
I don't know whether this is a bug or feature :)

field := FieldDef.make(loc, parent, fieldName,

h3. Static imports
Not so long ago there's a [Static imports|http://fantom.org/sidewalk/topic/1223] discussion on Fantom's forum. Without arguing whether static imports are good or bad, let's see what we can do using DSLs, so the code like this would be possible:
{noformat}
class Sample
{
StaticImports imports :=
StaticImports<|
sys::Str.spaces
sys::File.os
|>
Void main()
{
file := os("/")
prefix := spaces(4)
}
}
{noformat}

So, what we want here - the DSL source just defines a list of fully qualified static methods, and DSL plugin just creates local static methods which route to corresponding 'foreign' static methods:
{noformat}
class Sample
{
private static File os(Str str) { File.os(str) }
...
}
{noformat}

The creation of method routers and their bodies look fairly simple, but there's a problem - again, DSL plugins are called a bit too late and compiler have already generated a lot of compiler errors. Fail? No! We can do another black magic trick - go through all complier [errs|http://fantom.org/doc/compiler/Compiler#errs] and remove those which caused by (yet) undefined methods:
{noformat}
//build qnames of new instance methods
newQnames := qnames.map { "${type.qname}.${it[it.indexr(".")..-1]}" }
//build compiler error message for further filtering
errMsgs := newQnames.map { "Unknown method '$it'" }
compiler.errs = compiler.errs.exclude { errMsgs.contains(it.msg) }
{noformat}

However we also need to manually go through all expressions and _adjust_ [CallExpr|http://fantom.org/doc/compiler/CallExpr] instances to insert correct references to our newly created methods. After then everything compiles and runs as expected, though console output will still display compilation errors during building. The dirty code of {{StaticImportDsl}} can be found [here|^StaticImportDsl.fan].

In fact, the example above have nothing common with initial purpose of DSLs, it is complete compiler plugin which significantly changes the source code meaning.

h3. Inheritance
So, using DSL plugins it is possible:
* Create objects
* Modify types by adding new methods
* Modify method bodies
* Remove compilation errors

But also, we can extend existing types! So, inside DSL plugin we can create new type extending anchor type, and then the instance of our new type will be just upcasted to the base type. On of the examples where it can be useful, is the following:
{noformat}
parser := Parser<| Value = [0-9]+ | '(' Expr ')'
Product = Value (('*' / '/') Value)*
Sum = Product (('+' / '-') Product)*
Expr = Sum |>
{noformat}
Here the parser grammar defines a set of non-terminal symbols which can refer to other symbols. We can generate the class extending 'Parser' with a set of methods corresponding to symbols right at compile time, something like that:
{noformat}
class MyParser : Parser
{
override Bool expr(InStream in, Ast ast) { sum(in, ast) }

Bool sum(InStream in, Ast ast)
{
if(!product(in, ast)) return false
while(peek == '*' || peek = '/')
{
consume
if(!product(in, ast)) return true
}
return true
}

Bool product(InStream in, Ast ast)
...
}
{noformat}
And therefore, since our parser extends {{Parser}}, the code like this will work correctly and give us great performance:
I'm not a parser guy, and don't have a complete working example on my hands, but I verified that extending types is possible on a simple synthetic example:
{noformat}
class Foo
{
virtual Str str() { return "a" }
}

class Sample
{
Void main()
{
echo(Foo<|bar|>.str) //prints 'bar'
}
}
{noformat}
The [Foo DSL plugin|^FooInheritor.fan] creates class like this:
{noformat}
class Foo1290429812650070000 : Foo
{
override Str str() { synth }
protected Str synth() { "bar" }
}
{noformat}

h3. Magic facets
DSL plugins is the backdoor which gives a full access to compiler API and allows to do almost everything. However, in such use cases, we don't need the return value of DSL expression. One of the ways of 'organizing chaos' can be something like below.
Let's define facet {{Magic}}:
{noformat}
facet class Magic
{
const Obj? kind
}
{noformat}

It's has a single purpose - we can put it on type or slot. The field {{kind}} has a single purpose too - we can assign DSL expression to it.
Also, we can define the base class for our 'magic' plugins, which could simplify the creation of plugins. Here's the basic idea:
{noformat}
abstract class MagicPlugin : DslPlugin
{
new make(Compiler c) : super(c) {}

override Expr compile(DslExpr expr)
{
loc := expr.loc
doMagic(type(loc, expr), slot(loc, expr), expr.src)
return LiteralExpr.makeNull(expr.loc, ns)
}

**
** type - type definition of enclosing type
** method - method definition. If method is null,
** then magic applied to type, otherwise - to method
**
abstract Void doMagic(TypeDef type, SlotDef? slot, Str src := "")

... ton of helper methods for simplifying AST manipulation ...
}
{noformat}
The implementation of {{type}} and {{slot}} methods searching for appropriate definitions can be found [here|^MagicPlugin.fan].

Later, we can write plugins as simple like this:
{noformat}
class EchoMagic : MagicPlugin
{
new make(Compiler c) : super(c) {}

override Void doMagic(TypeDef t, SlotDef? s, Str src := "")
{
echo("type - $t")
echo("slot - $s")
echo("src - $src")
}
}
{noformat}

And use them like this:
{noformat}
@Magic { kind = Echo<||> }
class Sample
{
@Magic { kind = Echo<||> }
Void method() {}

@Magic { kind = Echo<||> }
Int slot := 1

Void main() { echo("hello, world!") }
}
{noformat}

Though, it is not really interesting, let's do something cool, how about...

h3. Actor locals
I've noticed that almost all the time when I use [Actors|http://fantom.org/doc/concurrent/Actor], I write the code like this:
{noformat}
private Str mutableStr
{
get { Actor.locals.getOrAdd("mutableStr") |->Str| { "default value" } }
set { Actor.locals["mutableStr"] = it }
}
{noformat}

It would be great to have a keyword {{actorlocal}}, so I could just write
{noformat}
private actorlocal Str mutableStr := "default value"
{noformat}
But the same effect can be achieved via {{@Magic}}! We can write an {{ActorLocal}} DSL plugin and then use it like this:
{noformat}
@Magic { kind = ActorLocal<||> }
private Str mutableStr := "default value"
{noformat}
Probably a bit more typing than with a keyword, but definitely better than current variant.
So, [ActorLocal|^ActorLocal.fan] DSL does the following:
# If magic facet is applied on type or method, returns compile error
# Removes [Storage|http://fantom.org/doc/compiler/FConst#Storage] flag from the field (to indicate that we provide getter and setter explicitly)
# Removes [Synthetic|http://fantom.org/doc/compiler/FConst#Storage] flag from getter and setter
# If field has the default value, locates initializer (static or instance depending on field flag) and removes field assignment from it (since filed has no storage and we will use initializer in our getter)
# Changes getter and setter to AST equivalents of this code:
{noformat}
get
{
Actor.locals.containsKey("fieldName") ?
Actor.locals["fieldName"] :
(initExpr ?: fieldType.defVal)
}
set { Actor.locals["fieldName"] = it }
{noformat}

Voila!

There's one problem with this code, imagine the case like this:
{noformat}
const class Foo
{
@Magic { kind = ActorLocal<||> }
Str s
}
const class MyActor
{
const Foo foo1
const Foo foo2
...
override Obj? receive(Obj? msg)
{
foo1.s = "str" //now foo2.s equals to "str" too
}
}
{noformat}
And I don't really know if this can be fixed correctly. In theory, we could store map {{\[Foo:FieldType\]}} in {{Actor.locals}}, but there are two obstacles:
# We won't know when to remove values from this map, probably weak keys could help
# Foo must be const - only const objects can be used as map keys

However, I don't think it is a big deal, since typically such fields can be declared in actors itself and used only from {{receive}} method, so they'll act as normal instance fields.

h3. Conclusion

The DSL plugins seem to be powerful and dangerous, though quite complex tool, allowing to completely change the meaning of the source code. They strongly rely on Compiler API, which is quite big and probably subject to change. Also it is not clear at which stage the DSL plugins are invoked, so sometimes it may be hard to predict all consequences of AST modifications, therefore it might be too risky to write something heavy using DSLs.

On the other hand, in some cases it can provide pretty elegant and robust solutions which can replace a ton of boilerplate code with full support of compile-time validation.