Meta attribute

[asterite]

While trying to think how to have JSON.mapping work with class inheritance, being able to add new properties to the mapping of a subclass, I concluded that the easiest way to implement this would be to attach metadata to instance variables. That way one would need to traverse these instance variables to generate the mapping. This will work through deep hierarchies, even with included modules and so on. This is also similar to how most other languages do JSON mapping, like Java, C# and Go. We actually discussed this in the past with @waj but thought that we would need to define these attributes somewhere, like in Java and C#... but maybe it can be done without all of that complexity (more similar to Go).

So I thought of something like this:

class Point
  @[Meta(:json)]
  @x : Int32

  @[Meta(:json)]
  @y : Int32

  @[Meta(:json, {converter: Time::EpochConverter}]
  @time : Time
end

So a Meta attribute has a namespace (a symbol) and optional data.

To traverse the instance vars one can do:

{% for ivar in @type.instance_vars %}
  {% if attr = ivar.attribute(:json) %}
    # Do something
    # attr would be nil for @x and @y, and {converter: Time::EpochConverter} for @z
  {% end %}
{% end %}

Maybe we can even provide a method to only get instance vars with a present attribute:

{% for ivar in @type.instance_vars_with_attribute(:json) %}
  # ...
{% end %}

What's missing in the above snippet is the code that defines new(pull : JSON::PullParser) and to_json(io). Maybe one can include a JSON::Mapping module or similar:

module JSON::Mapping
  macro included
    JSON::Mapping.from_instance_vars
  end

  macro from_instance_vars
    def self.new(pull : JSON::PullParser)
      {% for ivar in @type.instance_vars_with_attribute(:json) %}
        # ...
      {% end %}
    end 
  end 
end

class Point
  include JSON::Mapping
  # or simply invoke JSON::Mapping.from_instance_vars, but the above is shorter and nicer
end

Now, the topmost snippet doesn't define getters and setters, one has to do that manually:

class Point
  @[Meta(:json)]
  @x : Int32

  property x
end

However, since property expands to an instance variable declaration, the Meta attribute would apply to the macro expansion, so to the instance variable, and we can do:

class Point
  @[Meta(:json)]
  property x : Int32
end

Maybe writing all that @[Meta(...)] is a bit tedious, so we can have a JSON.attribute macro that would expand to @[Meta(:json)] (this is just a convenience syntax):

class Point
  JSON.attribute
  property x : Int32
end

We could still keep JSON.mapping, which will expand to a series of properties with the @[Meta(:json)] attribute, and also include the JSON::Mapping module (I think the current JSON.mapping syntax is very short and convenient when mapping to an existing API you want to consume)

Now a subclass of Point needs only to add more properties and map them:

class Point3 < Point
  JSON.attribute
  property z : Int32
end

Because the new(pull : JSON::PullParser) method that was generated uses @type inside it, it's a macro method, so this method will be different for Point3 than from Point, taking into account this new instance variable.

Maybe in the future we'd like to apply @[Meta] tags to methods and types, so we should probably have an extra parameter that specifies the valid targets. For example:

# Only applies to instance variables
@[Meta(:json, nil, :instance_var)]
@[Meta(:json, {converter: ...}, :instance_var)]

# Only applies to methods
@[Meta(:json, nil, :method)]

# Only applies to types
@[Meta(:json, nil, :type)]

Basically, @[Meta(key, optional_data, optional_target)].

And of course one could query this metadata from types and methods as well. One valid examples of method metadata is a test framework, where you would annotate methods with @[Meta(:test)] to later know which methods to run.

Another good example of instance var metadata is an ORM. Right now one has to keep that metadata separated from the instance vars, but keeping it in the instance vars is probably the best/easiest way to do it.

Also, this reduces duplication when defining both JSON and YAML mapping:

class Point
  JSON.attribute; YAML.attribute
  property x : Int32

  JSON.attribute; YAML.attribute
  property y : Int32
end

# vs.
class Point
  JSON.mapping x : Int32, y : Int32
  YAML.mapping x : Int32, y : Int32
end

The second form duplicates the types, and the macro also generates properties for each of those variables, so a lot of duplicated code under the scene (it's not a big deal, but the first form is more DRY)

I believe this approach will reduce code complexity a lot. In fact, with this we can implement extensible JSON mapping without using the newly finished macro hook, but that doesn't mean finished is useless, it can have other uses (like, when you need to store stuff that isn't necessary attached to instance variables, methods or types, and process that at the end).

This is just a preliminary design, I haven't discussed this with anyone else, so comments are welcome!

/cc @bcardiff @waj

[cjgajard]

Nice feature. Though, JSON.attribute seems useless to me. It's just one more thing to learn, while @[Meta(:json)] is short enough and explains what is happening.

[bcardiff]

I'm all in to be able to add attributes/metadata to ivars.
I have many issues to address regarding the proposal

1. Meta sounds like one more well known attribute by the compiler.
   But every attribute is actually metadata.
   I would prefer to just allow the user create it's own attributes. Like in C#.

class JsonAttribute < Attribute
end

# will allow to use @[Json(*options)]

Maybe Meta can be defined this way to basic "tagging" ivars.
Having some sort of attribute declaration can allow autocomplete in the future.
And we can also get compilation error for extensions based on metadata.

1.1. Maybe the allowed *options can be declared somehow, so more safety on this side.

2. Multiplicity. I would not assume there is only a single instance of the attribute to an ivar.
   So the traversal should have that in mind.

2+1. Multiplicity can also be restricted in the declaration.

3. Target. Definitily we want more that just ivars. Let's go for types & methods!

3+1. Again, could be restricted in the declaration if wanted and generate compile time errors.

4. Traversal. For sure there can be many ways to traverse the attributes, the basic should have the Multiplicity into account.

5. Macros and tagging. I don't think that using an Attribute before a macro call should apply that attribute to the expanded code. That seems meesy. You might want to apply something to the getter and something to the setter. This leads to the next point.

6. Meta tagging. Since code can be metaprogrammed I would search a solution of applying attributes to ivar/methods/types. Maybe would be hardcoded in the compiler.

class Customer
  @[Apply(@name, Json, *options)]
  # ...
end

So basically a Json.auto macro/module can be created that will automatically tag every ivar before compiled and all are serializable. That would be one use case. But it will be able to reach all the generated code.

7. JSON.mapping should be rewritten in terms of this IMO. For the extensible story, similar to .net, attributes can be defined to be inherited or not.

[RX14]

I too would prefer a strongly typed, more powerful implementation, such as in C# or java. We should at the very least be able to implement flags enums, with the current syntax, with just standard library code.

How about this for a syntax?

@[Targets(:class)]
@[AllowMultiple]
attribute JSON(name : String, allow_nil)
  # this is now macro code which receives the class and expands inside it
end

I think that attributes being themselves macros is a powerful concept which is quite simple, and may work well. Obviously they should be used in moderation, but that's true for macros in general.

Adding a Replaces attribute to the attribute definition could allow a macro to expand instead of what it was annotating, instead of alongside it.

This is a rough first idea I thought up in 20 minutes so please do criticize it.

[RX14]

In terms of tagging getter/setter/property, the simplest way forward would just to have the property macro place the attributes it has on the instance variable. This would mean attributes expand after normal macros (if we make attributes functions), and that macros could receive the attributes placed on their call.

[RX14]

Note that with my proposal, data only attributes would simply have an empty body. They would still be traversable meta data on instance variables. For example, the JSON attribute would have a body which just included a JSON::Mapping module (duplicated includes are ignored right?) which would iterate instance variables in its included macro and check the attribute values.

[ysbaddaden]

I like the feature a lot!

I'm not fond of the Type.attribute syntax. It looks like a method call, not a decorator associated with the following declaration. It's confusing.

I'd prefer to be able to specify many metas (or tags?) at once:

@[Meta(:json, :yaml)]
property x : Int32

[asterite]

@bcardiff @RX14 In my proposal the implementation is very easy: when you ask for an attribute of a given name (:json) at compile-time, the compiler returns true if it's present (without additional data) or the full AST node that was specified as data if any. Then you manipulate that AST node like you do now. So doing:

@[Meta(:json, {converter: Time::EpochConverter})]
@x : Time

is very similar to doing:

time: {type: Time, converter: Time::EpochConverter}

We just changed the way this information is exposed and stored.

However, adding an attribute declaration is a lot more complex: a new keyword, a new type definition... Also, what does this attribute declaration hold? Is it a class or a struct? Does it have methods? How do I specify what arguments can it receive? Are the arguments typed? If so, what's the type of a JSON converter? Right now I can pretty much specify anything that responds to from_json/to_json as a converter, there's no "module" or "interface" for it.

That's why I was aiming for something simple. In fact, this is very similar to how Go implements metadata: https://golang.org/pkg/reflect/#StructTag

package main

import (
	"fmt"
	"reflect"
)

func main() {
	type S struct {
		F string `species:"gopher" color:"blue"`
	}

	s := S{}
	st := reflect.TypeOf(s)
	field := st.Field(0)
	fmt.Println(field.Tag.Get("color"), field.Tag.Get("species"))
}

By convention, tag strings are a concatenation of optionally space-separated key:"value" pairs. Each key is a non-empty string consisting of non-control characters other than space (U+0020 ' '), quote (U+0022 '"'), and colon (U+003A ':'). Each value is quoted using U+0022 '"' characters and Go string literal syntax.

That is, in Go a tag is just a string with some convention on it. No need to declare attributes, their type, etc. At runtime you just parse it and do whatever you need with it.

In Crystal this would be similar (according to my proposal): you can attach any metadata (any AST node), and then access it only at compile-time and do whatever you need with it.

This feels more lightweight and in fact much more powerful, because if we bring types in then we'll be more restricted (like in the converter example above). Also, it works in Go, and I like its simplicity, so I think it can work too for Crystal.

But, having typed attributes is definitely an option. But if we want to consider it, someone (@bcardiff, @RX14) has to write a full proposal about all the details (class/struct declaration, constructor? methods? types?)

Also... what is a use case where you would want multiple instances of the same attribute applied to an entity?

[asterite]

Actually, we could have annotation definitions, but I would make them untyped. So maybe something like this:

# This declares an attribute together with the required arguments (here empty),
# the optional arguments (key, nilable, etc.),
# also the target, maybe the multiplicity
@[Attribute(JSON,
  required: [],
  optional: [:key, :nilable, :default, :emit_null, :converter, :root]),
  target: :instance_var]

To use it:

@[JSON]
@x : Int32

@[JSON(converter: Time::EpochConverter)]
@time : Time

We can probably store attributes under a different namespace, so the JSON attribute doesn't conflict with the JSON module, or simply assume it will be something like JSON::Attribute or JSONAttribute or JSON::__ATTR__ and so on (this is a compiler detail)

I admit that this does look better than the original proposal, and one could also do now:

@[JSON, YAML]
@x : Int32

@[JSON(key: "foo"), YAML(key: "foo")]
@y : Int32

This is just a bit less lightweight than the original proposal, but it looks a bit better (shorter to use annotations), we get a bit of type safety (the annotation name), and it's easy to apply multiple attributes to a variable. It still keeps the "metadata is an AST node" thing, though. But I think this is better, because this metadata will be processed by macros at compile time, and macros are closer to a dynamic language than to a typed language (in fact, macros are interpreted)

Anyway, I'm just throwing more ideas to the table :-)

[bcardiff]

@RX14 is far more flexible to allow the attribute declaration (if any) to leave the semantic aside. There are many use cases and some might not be expressed as "if I apply this attribute I need to do this to the target" so you might not know what to do. For example: you might need to access the class attributes when applying something based on ivar attributes.

[bcardiff]

@asterite I agree to be as lightweight as possible. Before jumping in a c#-ish style I tried to imagine what would be a ruby-like for this.

I think that been able to discover possible attributes/options would open the possibility for better editor/ide support in the future.

Again, we could have a Meta implemented on top of some flexible attributes. I wanted to push forward to stop adding ad-hoc/fixed attributes: Link, Flags, etc. since we are looking to allow users define semantic on attributes.

I agree that is neither a struct nor a class and using them is a hack. Maybe @[Attribute...] can be the declaration.

Regarding type safety for properties, I would say that not typing the properties is fine. Everything will need to be interpreted in macro mode as you said. I did think of having required/allowed properties. I see them as if when you call a method without type annotations they have have required and allowed named arguments.

So, I am not fully convinced of the @[Attribute..] syntax, but I do agree on the semantic.

[asterite]

@bcardiff Yes, I used the @[Attribute...] syntax because it doesn't require a change to the language, but a specialized syntax for it could also work :-)

Regarding automatically doing stuff at compile-time if an attribute is present, maybe we can have another property of an annotation, and that is to trigger some macro call inside the type. So something like this:

@[JSON::Serializable]
class Point
  @[JSON::Attribute]
  property x : Int32

  @[JSON::Attribute]
  property y : Int32
end

I used JSON::Attribute instead of JSON because I need two attributes now: one for the class, one for the instance variable. Then:

@[Attribute(JSON::Serializable, finished: JSON.define_mapping)]

Here finished is the same as the new finished macro hook, so at the end of the program one can imagine JSON.define_mapping being inserted inside the type declaration, and that macro would define the constructor from a pull parser and the to_json method by iterating instance var attributes.

Maybe an attribute that is applied to an instance variable can also have a finished property, and it would apply to the enclosing type (only once). So we could go back to:

class Point
  @[JSON]
  property x : Int32

  @[JSON]
  property y : Int32
end

which is a bit more DRY: if we use a JSON attribute on an instance var, it's obvious that we want the type to be serializable, so we want to generate the necessary methods.

[drhuffman12]

What about something like:

class Point
  property x : Int32 ser [JSON,YAML]
  property y : Int32 ser [JSON,YAML]
end

[RX14]

I think that the syntax for custom attributes should be exactly the same as the current attribute syntax. Whether attributes are implemented in the compiler or using macros, they are simply metadata which add info to what they are annotating, which may cause a behavioural change. The implementation details (compiler or not) should not be obvious.

I also prefer repeating annotations to having them seperated with commas. I think that having them on the same line means multiple (possibly unrelated) concerns per line.

@[JSON(key: "foo")]
@[YAML(key: "bar")]
property foo : String

vs

@[JSON(key: "foo"), YAML(key: "bar")]
property foo : String

I also think that attributes should be explicitly declared, with exactly the semantics and syntax for parameters in macro declarations (just what's in the brackets). That is, define the possible names, and their default values, but not their types. Splats are also obviously allowed, and adding a default value would make the parameter optional. Sometime like attribute JSON(key = nil, nillable = nil, ...)

I strongly dislike @asterite's syntax where declaring an attribute was done inside the attribute syntax itself. (@[Attribute(JSON, ...]). Attributes should be attaching metadata to existing constructs, not being a construct of their own, which creates an entirely new type of their own. I think attributes should be used to annotate the attribute definition with what it can target and whether the attribute can be repeated on the target multiple times. (repeatable annotations were useful enough to make it into java so I think we should have it)

I do think we should be able to define a macro which runs when an attribute is applied, because it removes the need to have both the annotation to describe the data, and to manually apply the macro to use the data. Whether this is done by naming the macro which should be run when the attribute is applied in the attribute definition, or by making the attribute definition a macro in itself, I don't mind.

[trans]

It strikes me as just imitative -- copying Go or Java/C# -- the approach seems quick and dirty and is clearly ugly. Embedding this information into classes leads to metadata bloat and/or an explosion of names (a subclass for each possible serialization). My gut feeling is that an improved macro system is where an elegant solution lies, but I am not sure. Ideally there would be a clean SOC between Crystal classes and the transformation code that converts serializations to/from them.

[kostya]

I solve this by magic finished macro, and shards: https://github.com/kostya/auto_constructor, https://github.com/kostya/auto_msgpack, https://github.com/kostya/auto_json

require "auto_json"
require "auto_msgpack"

struct Person
  include AutoJson
  include AutoMsgpack

  field :name, String
  field :age, Int32
  field :email, String?, json_key: "mail"
  field :balance, Float64, default: 0.0, json: false
  field :data, String?, msgpack: false
end

person = Person.new(name: "Vasya", age: 20, balance: 10.0, email: "bla@ru")
p person # => Person(@age=20, @balance=10.0, @data=nil, @email="bla@ru", @name="Vasya")

json = person.to_json
puts json # => {"name":"Vasya","age":20,"mail":"bla@ru"}

person2 = Person.from_json(json)
p person2 # => Person(@age=20, @balance=0.0, @data=nil, @email="bla@ru", @name="Vasya")

msgpack = person2.to_msgpack
puts msgpack # => Bytes[132, 164, 110, ...]

person3 = Person.from_msgpack(msgpack)
p person3 # Person(@age=20, @balance=0.0, @data=nil, @email="bla@ru", @name="Vasya")

[paulcsmith]

I've tried to do something similar to @kostya, but my problem was the finished macro was that using it with a server seemed to never call the macro since the program never actually "finished." Maybe the instance meta attributes could solve this. Here's an example of the finished problem with sleep

class User
  macro finished
    puts "1, 2, 3"
  end
end

sleep(5)
# Won't print "1, 2, 3" until *after* the sleep is done. Is there a way to "force" the finished hooks to run?"
# When it's a server the "finished" macro *never* gets called

[luislavena]

@paulcsmith the finished macro is processed a compile time. If the code that is inside the macro produces output (either on the main fiber or another one) it might be affected in similar way to other fiber-related issues.

You can see that finished macro works with the following example:

class Foo
  macro finished
    BAR = 10
  end
end

p typeof(Foo::BAR) # => Int32
p Foo::BAR         # => 10

If you replace the sleep with Fiber.yield it will produce output:

class Foo
  macro finished
    puts "1, 2, 3"
  end
end

puts "main"
Fiber.yield

main
1, 2, 3

[paulcsmith]

@luislavena Ah ok I'll try this out. Thanks for the explanation and example code!

[emilebosch]

Hi all, hope i can jump in. I've been programing c# for over 8 years, eventually moved away from Ruby. I'd love it if it is possible to have typed attributes exactly like c#.

And if possible also drop the [] it looks like weird clutter to me, totally unruby, more like machines then for humans to be honest.

[GraphqlType(name: "Person")]
class UserType < GraphqType
   [Deprecate(reason: "Comments are now on the user model")]
   def comments
   end
end

Or + ? Hmm not sure about this one.

+GraphqlType(name: "Person")
+Schema(visability: :internal)
class UserType < GraphqType
   +Deprecate(reason: "Comments are now on the user model")
   def comments
   end
end

I'm writing a graphql crystal and annotations/attributes would really be handy, but please make them for humans <3. (even if it means making language changes)

[refi64]

What about @ like Python/Java/Dart?

[watzon]

I love this proposal! This is one of my favorite parts about Go’s marshaling/unmarshaling.

[zatherz]

Why not associate custom attributes with macros?

@[Attribute(name: "TestAttr")]
macro test(node)
  # do something with node...
  node
end

@[TestAttr]
def abc
end
# test gets called with this Def node, and its result is pasted in place of it

[bew]

@zatherz this can already by done with macros:

macro test(node)
  # do something with node...
  node
end

test def abc
end
# test gets called with this Def node, and its result is pasted in place of it

https://carc.in/#/r/2i3u
or did I missed something in what you were proposing?

[zatherz]

yes, you missed the attribute syntax

@[Attribute(name: "TestAttr")]
macro test(node, test)
  # do something with node...
  node
end

@[TestAttr(test: 1)] # node would always be passed through the `node` argument?
def abc
end
# test gets called with this Def node and 1, and its result is pasted in place of it

[straight-shoota]

Can be done like this:

test test: 1, def abc
end

But that's something entirely different anyway than what this is about. I don't think it would make sense to implement arbitrary annotations as macros.

[zatherz]

Bump.

[kostya]

right now there is one type of meta information exists, have variable default value or not, in case:

class A
  @a : Int32 = 1
end

can i access to such information from macro (have variable default value or not), to construct macro like mappings and others? or how hard to add such info to @type.instance_vars

[asterite]

So, I've been thinking a bit more about this, and here's a proposal.

Declaring an attribute

You do that with an attribute declaration (similar to a class declaration, but with a different keyword):

module JSON
  attribute Field
  end
end

Right now the atrribute definition will be empty, in the future maybe we can specify there to what it applies (types, ivars, etc.), maybe what keys it can hold, etc. One can document an attribute as usual, and it will appear in the docs.

We can even go one step further and make the current attributes defined like that, but be "primitives" that affect compilation (like they currently do). The difference is that they will show up in the docs.

(in that way, I'd also like to make include and extend "primitive" macros, so we can document them in docs, but also to be able to define a method include or extend in a nested scope and call them, without having include and extend be keywords... but that's another topic)

Applying attributes

You do it like you do it now:

class Foo
  include JSON::Serializable

  @[JSON::Field(key: "ex")]
  property x : Int32
end

Note that property x : Int32 will expand to @x : Int32 and some other stuff, and the attribute will "go through" the macro expansion, effectively affecting the instance variable declaration.

Except that right now JSON::Field in an attribute doesn't parse. It's an error if the attribute wasn't previously declared. One can pass any arguments to the attribute (not enforced so far).

Querying attributes

You use the .attribute(type) macro method. It will return the attribute's value, if present, or nil if not. From the attribute you can access its values with #[], passing an index for positional arguments or a string/symbol for named arguments.

With that, we could implement JSON.mapping by simply traversing all instance vars and querying their attributes. The attribute will hold optional key, converter, etc. The default value can be directly taken from the instance var (much simpler). And instance vars without an explicit attribute will be serialized. To avoid serialization of one instance var, you could do (for example):

class Foo
  @[JSON::Field(ignore: true)]
  @x : Int32
end

I think it's better to choose attributes to drop in serialization, as usually we'd like to serialize all attributes. It also makes it easier to serialize to JSON, YAML and other formats: you don't need to specify that attribute again and again for all the fields and the different formats (you do have to do that if the key is different, or some other value is different, but those are exceptions).

Advantages over the current JSON.mapping:

• works well for subclasses
• less typing
• less repetition

For now you could attach attributes to types and instance vars. Maybe in the future we could also attach them to methods.

Serialization is just one example for attributes. I'm pretty sure people will come up with great examples using them.

Thoughts?

[RX14]

Right now the atrribute definition will be empty, in the future maybe we can specify there to what it applies (types, ivars, etc.), maybe what keys it can hold, etc. One can document an attribute as usual, and it will appear in the docs.

I don't think it makes sense to have an attribute definition without at least containing the possible arguments to the attribute. Having an attribute Foo; end without even being able to put anything in it is weird.

Apart from that, looks fine to me!

[kostya]

how to acces to attributes from different modules?

class Foo
  @[JSON::Field(key: "y")]
  @[YAML::Field(key: "z")]
  @x : Int32
end

here double .attribute("key")

[asterite]

@kostya for example:

{% for ivar in @type.instance_vars %}
  {% attr = ivar.attribute(JSON::Field) %}
  {% key = attr ? attr[:key] : ivar.name %}
{% end %}

[asterite]

Closed by #6063