Static Metaprogramming

[jakemac53]

Metaprogramming refers to code that operates on other code as if it were data. It can take code in as parameters, reflect over it, inspect it, create it, modify it, and return it. Static metaprogramming means doing that work at compile-time, and typically modifying or adding to the program based on that work.

Today it is possible to do static metaprogramming completely outside of the language - using packages such as build_runner to generate code and using the analyzer apis for introspection. These separate tools however are not well integrated into the compilers or tools, and it adds a lot of complexity where this is done. It also tends to be slower than an integrated solution because it can't share any work with the compiler.

Sample Use Case - Data Classes

The most requested open language issue is to add data classes. A data class is essentially a regular Dart class that comes with an automatically provided constructor and implementations of ==, hashCode, and copyWith() (called copy() in Kotlin) methods based on the fields the user declares in the class.

The reason this is a language feature request is because there’s no way for a Dart library or framework to add data classes as a reusable mechanism. Again, this is because there isn’t any easily available abstraction that lets a Dart user express “given this set of fields, add these methods to the class”. The copyWith() method is particularly challenging because it’s not just the body
of that method that depends on the surrounding class’s fields. The parameter list itself does too.

We could add data classes to the language, but that only satisfies users who want a nice syntax for that specific set of policies. What happens when users instead want a nice notation for classes that are deeply immutable, dependency-injected, observable, or differentiable? Sufficiently powerful static metaprogramming could let users define these policies in reusable abstractions and keep the slower-moving Dart language out of the fast-moving methodology business.

Design

See this intro doc for the general design direction we are exploring right now.

Update January 2025

We have an unfortunate update on macros/metaprogramming. We have invested significant time and resources to prototype macros over the past couple years. Unfortunately, each time we solved a major technical hurdle, we saw new ones pop up. At this point, we are not seeing macros converging anytime soon toward a feature we are comfortable shipping, with the quality and developer-time performance we want.

After considering the opportunity cost — in particular, the features we could be shipping to the community instead — we’ve made the difficult decision to stop our work on macros.

For additional details, please see the blog post:
https://medium.com/dartlang/an-update-on-dart-macros-data-serialization-06d3037d4f12

[rrousselGit]

Would static function composition be in the scope of this feature?

At the moment, higher-order functions in Dart are fairly limited since they require knowing the full prototype of the decorated function.

An example would be a debounce utility:

void Function() debouce(Duration duration, void Function() decorated) {
  Timer? timer;
  return () {
    timer?.cancel();
    timer = Timer(duration, () => decorated());
  };
}

which allows us to, instead of:

class Example {
  void doSomething() {

  }
}

write:

class Example {
  final doSomething = debounce(Duration(seconds: 1), () {
  
   });
}

but that comes with a few drawbacks:

• obvious readability decrease
• our debounce utility is not very reusable. It works only on void Function(), but we'd like it to work for all functions.

With static meta-programming, our debounce could inject code in the class at compilation, such that we could write:

class Example {
  @Debounce(Duration(seconds: 1))
  void doSomething() {
    print('doSomething');
  }
  
  @Debounce(Duration(seconds: 1))
  void doSomethingElse(int value, {String named}) {
    print('doSomethingElse $value named: $named');
  }
}

[jakemac53]

There is a delicate balance re: static function composition, but there are certainly many useful things that could be done with it. I think ultimately it is something we would like to support as long as we can make it obvious enough that this wrapping is happening.

The specific balance would be around user confusion - we have a guiding principle that we don't want to allow overwriting of code in order to ensure that programs keep their original meaning. There are a lot of useful things you could do by simply wrapping a function in some other function (some additional ones might include uniform exception handling, analytics reporting, argument validation, etc). Most of these things would not change the meaning really of the original function, but the code is being "changed" in some sense by being wrapped.

Ultimately my sense is this is something we should try to support though. I think the usefulness probably outweighs the potential for doing weird things.

[mateusfccp]

I like Lisp approach (in my opinion, the utmost language when it comes to meta-programming). Instead of defining a @Debounce or something alike, we would define new syntax that would simply expand to a regular method at compile-time. I don't know, however, how much complex is to make something like this considering Dart syntax.

[lrhn]

For something like debounce, a more aspect-like approach seems preferable. Say, if you could declaratively wrap a function body with some template code:

class Example {
  void doSomething() with debounce(Duration(seconds: 1)) {
    print('doSomething');
  }
  
  void doSomethingElse(int value, {String named}) with debounce(Duration(seconds: 1)) {
    print('doSomethingElse $value named: $named');
  }
}

template debounce<R>(Duration duration) on R Function {
  template final Stopwatch? sw;
  template late R result;
  if (sw != null && sw.elapsed < duration) {
    return result;
  } else {
    (sw ??= Stopwatch()..start()).reset();
    return result = super;
  }
}

This defines a "function template" (really, a kind of function mixin) which can be applied to other functions.
It cannot change the signature of the function, but it can access arguments (by forwarding them as with templateName(arg)), and it can do things before and after the original body is run.
The template variables are per-template instantiation variables (just as we could declare static variables inside normal functions).

(Maybe we just need AspectD for Dart.)

[rrousselGit]

It cannot change the signature of the function, but it can access arguments

But an important part of function composition is also the ability to inject parameters and ask for more parameters.

For example, a good candidate is functional stateless-widgets, to add a key parameter to the prototype and inject a context parameter.
This means the user would define:

@statelessWidget
Widget example(BuildContext context, {required String name}) {
  return Text(name);
}

and the resulting prototype after composition would be:

Widget Function({Key? key, required String name})

where the final code would be:

class _Example extends StatelessWidget {
  Example({Key? key, required String name}): super(key: key);

  final String name;

  @override
  Widget build(BuildContext) => originalExampleFunction(context, name: name);
}

Widget example({Key? key, required String name}) {
  return _Example(key: key, name: name);
}

[jakemac53]

I definitely agree we don't want to allow for changing the signature of the function from what was written. I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want. The original function might be private in that case.

[rrousselGit]

I don't think that is prohibitive though as long as you are allowed to generate a new function/method next to the existing one with the signature you want

That's what functional_widget does, but the consequence is that the developer experience is pretty bad.

A major issue is that it breaks the "go to definition" functionality because instead of being redirected to their function, users are redirected to the generated code

It also causes a lot of confusion around naming. Because it's common to want to have control on whether the generated class/function is public or private, but the original function to always be private.

By modifying the prototype instead, this gives more control to users over the name of the generated functions.

[jakemac53]

Allowing the signature to be modified has a lot of disadvantages as well. I think its probably worse to see a function which is written to have a totally different signature than it actually has, than to be navigated to a generated function (which you can then follow through to the real one). You can potentially blackbox those functions in the debugger as well so it skips right to the real one if you are stepping through.

[bouraine]

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

[mateusfccp]

@bouraine

I suppose this will allow generating fromJson and toJson methods at compile time for Json serialization ?

Yes.

[jakemac53]

@tatumizer This issue is just for the general problem of static metaprogramming. What you describe would be one possible solution to it, although we are trying to avoid exposing a full AST api because that can make it hard to evolve the language in the future. See https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md for an intro into the general design direction we are thinking of here which I think is not necessarily so far off from what you describe (although the mechanics are different).

[idkq]

Great intro & docs.

Hopefully we'll stay (far far) away from annotations to develop/work with static meta programming?!

[jakemac53]

so it looks like copyWith is seen as a crown jewel of the upcoming facility

The main reason we use this as an example is its well understood by many people, and it is also actually particularly demanding in terms of features to actually implement due to the public api itself needing to be generated :).

The language needs some mechanism of dealing with default values, which has been a showstopper in dart from day one.

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

[rrousselGit]

Can you elaborate? Default values for parameters are getting some important upgrades in null safe dart (at least the major loophole of being able to override them accidentally by passing null explicitly is closed).

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

freezed supports this, but only because it relies on factory constructors and interface to hide the internals of copyWith (that is in fact a copyWith({Object? name = _internalDefault}))

[jakemac53]

I believe the issue is that we cannot easily differentiate between copyWith(name: null) and copyWith() where the former should assign null to name and the latter just do nothing

Right, this is what I was describing which null safety actually does fix at least partially. You can make the parameter non-nullable (with a default), and then null can no longer be passed at all. Wrapping functions are required to copy the default value, basically it forces you to explicitly handle this does cause some extra boilerplate but is safe.

For nullable parameters you still can't differentiate (at least in the function wrapping case, if they don't provide a default as well)

[idkq]

Metaprogramming is a broad topic. How to rationalize? We should start with what gives the best bang for buck (based on use cases).

Draft topics for meta programming 'output' code:

1. Methods
2. Classes (shell)
3. Class members
4. Types
5. Enums
6. Statements (?)
7. Mixins (?)
8. Generics (?)

Also on output code:

Be able to visualize in some way the code generated into your program, at development time
(https://github.com/dart-lang/language/blob/master/working/static%20metaprogramming/intro.md#usability)

Would be great if this could work without saving the file, a IDE-like syntax (hidden) code running continuously if syntax is valid. I refuse to use build_runner's watch

[porfirioribeiro]

Metaprograming opens doors to many nice features
For the data class thing, this is something i miss from Kotlin.
When i used Java we used @Data / @Value from Lombok that was some sort of generator, i guess having something like this would be enough for the data class's

Other language that does a great job at implementing macros is Haxe you can use Haxe language to define macros

I guess there are many challenges to implement this.

[ykmnkmi]

can we extend classes with analyzer plugin?
can we use external and patch like patches in sdk libraries for extending classes?
plugins for CFE?

[escamoteur]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

[jodinathan]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

I agree with this.
I like the idea of meta programming as long as it doesn't remove how readable and maintainable a Dart code is.

[idkq]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

[leafpetersen]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity. The fact that it isn't as concise as other languages it in reality an advantage because it makes Dart code really easy to read and to reason about.
I fear meta programming will kill this. How will a goto-definition in an IDE work with it? How discoverable and maintainable is such code?

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

[jodinathan]

@escamoteur Writing less code does not make it more complicated necessarily. It can, I agree, if someone does not fully understand the new syntax. But the trade-off is obvious: time & the number of lines saved vs the need for someone to learn a few capabilities.

Generated code is normal simple code. I just suggested real-time code generation instead of running the builder every time or watching it to save. That way you get real time goto. But if you are using notepad then of course you need to run a process.

In my honest opinion: things must be obvious, not magical.
Every time I have to read or develop in PHP, JS or C with preprocessors etc... I just hate it.
Too many magical stuff that you just can't read or debug easily.
Dart is the opposite of that without being boring as hell as Java.
In fact, there was a time that some Dart packages used to implement the noSuchMethod to create magical methods. Gee, what a pain.
Meta programming could be the next Dart transformers if it takes the glittering magical road.

Just to be 100% clear, we are intensely focused on these exact questions. We will not ship something which does not integrate well with all of our tools and workflows. You should be able to read code and understand it, go to definition, step through the code in the debugger, get good error messages, get clear and comprehensible stack traces, etc.

^ this

[esDotDev]

I'm not sure if I like the idea having this added to Dart because the beauty of Dart is its simplicity.

But there is nothing beautiful about writing data classes or running complicated and and slow code-generation tools.

I'm hoping this can lead to more simplicity not less. Vast mounds of code will be removed from our visible classes. StatefulWidget can maybe just go away? (compiler can run the split macro before it builds?). Things can be auto-disposed. Seems like this could hit a lot of pain points, not just data classes and serialization..

[safasofuoglu]

Since dart currently offers code generation for similar jobs-to-be-done, I'd suggest evaluating potential concerns with that consideration:

• Metaprogramming is not simple/obvious - can it be made at least as simple/obvious as codegen through tooling?
• Metaprogramming will be abused - Is there a reason to think it will be abused more than codegen? (potentially, if it provides better ergonomics)

On the other hand, besides being an upgrade from codegen for developers, metaprogramming could provide healthier means for language evolution beyond getting data classes done. Quoting Bryan Cantrill:

Another advantage of macros: they are so flexible and powerful that they allow for effective experimentation. For example, the propagation operator that I love so much actually started life as a try! macro; that this macro was being used ubiquitously (and successfully) allowed a language-based solution to be considered. Languages can be (and have been!) ruined by too much experimentation happening in the language rather than in how it’s used; through its rich macros, it seems that Rust can enable the core of the language to remain smaller — and to make sure that when it expands, it is for the right reasons and in the right way.
http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/

PS @jakemac53 the observable link leads to a private google doc.

[insinfo]

this would be fantastic if it allowed, the longed-for serialization for JSON natively without the need for manual code generation or reflection in time of execution

Today practically all applications depend on serialization for JSON, a modern language like dart should already have a form of native serialization in the language, being obliged to use manual codegen or typing serialization manually is something very unpleasant

[felixblaschke]

My approach on a macro mechanism. Basically tagging a certain scope with a macro annotation, that refers to one or multiple classes to 1:1 replace the code virtually... like a projection. It's very easy to understand and QOL can be extended by providing utility classes.

#ToStringMaker() // '#' indicates macro and will rewrite all code in next scope
class Person {
    String name;
    int age;
}

// [REWRITTEN CODE] => displayed readonly in IDE
// class Person {
//    String name;
//    int age;
//
//    toString() => 'Person(vorname:$name, age:$age)'
// }

class ToStringMaker extends Macro {

    // fields and constructor can optionally obtain parameters

    @override
    String generate(String code, MacroContext context) { // MacroContext provides access to other Dart files in project and other introspection features
        var writer = DartClassWriter(code); // DartClassWriter knows the structure of Dart code

        writer.add('String toString() => \'${writer.className}(${writer.fields.map(field => '${field.name}:\${field.name}').join(', ')})\'');

        return writer.code; // substitute code for referenced scope
    }

}

[navaronbracke]

Good to see that, despite all this, augmentations will be unaffected. Does this mean that - in the very far away future - a different avenue could be considered to give macro's a second try?

[Mikkelet]

We have an unfortunate update on macros/metaprogramming. We have invested significant time and resources to prototype macros over the past couple years. Unfortunately, each time we solved a major technical hurdle, we saw new ones pop up. At this point, we are not seeing macros converging anytime soon toward a feature we are comfortable shipping, with the quality and developer-time performance we want.

After considering the opportunity cost — in particular, the features we could be shipping to the community instead — we’ve made the difficult decision to stop our work on macros.

For additional details, please see the blog post:
https://medium.com/dartlang/an-update-on-dart-macros-data-serialization-06d3037d4f12

You might as well include build runner and json_serializable in stdlib, then. Disappointing.

[rrousselGit]

Good to see that, despite all this, augmentations will be unaffected. Does this mean that - in the very far away future - a different avenue could be considered to give macro's a second try?

I wouldn't bet on that. Otherwise I doubt the team would've cancelled macros.

But we can have alternatives!

[MohiuddinM]

I wouldn't bet on that. Otherwise I doubt the team would've cancelled macros.

But we can have alternatives!

Will augmentation be enough to have the new freezed version (that was going to use macros previously)? Or maybe a lite version of freezed?

[TekExplorer]

In that case, can we put effort into making using build_runner easier to work with?
or rather, the analyzer.

most of the time I have no idea how to even do anything!

[tatumizer]

From the article:

Runtime introspection (e.g., reflection) makes it difficult to perform the tree-shaking optimizations that allow us to generate smaller binaries.

I don't understand that. If reflection is supported only during build runner run, this doesn't affect any further optimizations at all.
But reflection provides a much nicer interface than AST or anything.

[bivens-dev]

In that case, can we put effort into making using build_runner easier to work with? or rather, the analyzer.

most of the time I have no idea how to even do anything!

I think one of the bigger potential improvements that could happen to the build system side of things would be from a documentation and tutorial point of view. What already exists today is incredibly powerful and has some really nice features but getting started is very complicated and not a clear path for many.

[jodinathan]

Augmentations and a faster build_runner are more than enough.
Please, focus on that.

[rrousselGit]

I've mentioned it in many other places, but I think another critical feature for better codegen is the changing "go-to-definition" to avoid having to step through the generated code.

Cf:
dart-lang/sdk#56518
dart-lang/sdk#56764
dart-lang/sdk#48827

[Wdestroier]

It would be great to avoid out-of-sync build_runner generated files on disk and eliminate the need to manually run build_runner.

[lucavenir]

[we need] documentation and tutorial point of view

Augmentations and a faster build_runner are more than enough.

No.

I wish this would be true. build_runner, way too often, just doesn't work out for the average developer.

Some examples include, but are not limited to:

• gargantuous wait times on big projects, even a 10x improvement might not be enough on a big project with a medium machine
• analysis times are impacted even more by code generation, as analysis is being re-run every time a file is written by the runner, causing a weird delay-loop of "generation x analysis" that can go on for minutes (the developer can't really type code within these frames)
• suddenly, a subset of files (or a single file) stop being tracked, or stop being generated by the runner, with no errors nor warnings in the logs
• most of the time, to solve the above, you have to stop the runner and then run flutter pub clean && flutter pub get && dart run build_runner watch -d again; this will likely waste us even more minutes (yes, not just seconds!), and this can happen again... (and again...)
• crazy hard to deal with and to setup in a monorepo; one would think that splitting a "big" project into smaller packages would help analysis and the runner, but in reality this disrupts the builder even more (e.g. you now have to be extra careful with your inner dependencies!)
• dart_runner watch stopping because I randomly / casually / involountarly saved a pubspec.yaml file. Did you forget to re-run your builder? Yeah, after a pub get it has to re-generate everything from scratch
• randomly receiving "you've hit a bug with build runner, file a report", with a stacktrace I can't really deal with: it's often hard if not impossible to track nor to report these behaviours; this is also worrying because it shows we can't really help you narrow down problems with the runner, and these problems are likely there to stay a long time (and btw they're here since forever)
• go-to-definition on generated elements uselessly leads the developer to the generated code, causing indirection and noise
• weird syntax (e.g. @freezed, or class Name with _$Name) that often overlaps with other non-builder annotations (e.g. @protected) that for sure confuses the hell out of the language beginners
• What do you do with generated files, analysis wise? Do we include them in analysis_options.yaml? Both "yes" and "no" answers lead to a different trade off that adds even more useless complexity on top of everything else
• What do you do with generated files, git wise? Do we include them into .gitignore? Maybe the answer here is a little more straightforward, but is that documented, anywhere?
• Because of build_runner, hot reload might be negatively affected (best case scenario: double hot reload to make it work; worst case: "try performing a hot restart instead")

As you can see most of these problems are intertwined with other aspects of the language: analysis, packages, hot reload, dependencies, mono repo scripts, etc.
No wonder macros tried to solve these in one fell swoop.
I fear that if you're giving up on macros, you'll likely give up on solving some of the above, and that's scary.

[tatumizer]

My feeling is that multi-staged augmentations will only aggravate and perpetuate the problem.
Why do you need to run a compiler just to be able to extract the names/types of the fields?
And then run it again to provide an input to yet another layer of generators. And then run it again...
This architecture is invented in the name of composability, which in this context is a utopian idea anyway.
The top macro (e.g. at class level) should be able to declare: I can work with such-and-such inner (e.g. field-level) annotations, and then generate everything in a single pass. At least, the user can be sure that the whole suite was designed to work in concert and was tested for this.
Also, shoehorning the input into dart syntax is too restrictive. The prototype of a class can be written in the form of some DSL or even .md file - its processing will take milliseconds. This will solve the problem without creating N bigger problems.

The alternative is to spend another 2 years in a futile pursuit of composability, with the same result.

[insinfo]

@mit-mit
This is one of the most disappointing news about Dart, if only Dart had static virtual members in interfaces, and a serialization interface that was part of the Dart SDK that could be manually implemented by the developer in custom objects so that it would be possible to pass custom objects directly to JSONEncode it would be great.

https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/tutorials/static-virtual-interface-members

[leafpetersen]

Hi folks - We're all of course quite disappointed about this. @jakemac53 and company did a lot of amazing work to try to make this happen. Making the call to cut our losses and move on was terribly difficult. I do very much believe it was the right call though, and I'm very excited about the opportunities this opens up. I do want to be very clear - we have no plans to pick this up again. That's not to say that we never will. I still love a lot of the ideas we explored, and I do think that some form of macro system (even if more limited) would be a great addition to Dart. But the decision here is to move the effort onto more focused language features to solve the core user problems, and that simply doesn't leave room on our roadmap to pick this work up in the foreseeable future.

I do hope that with augmentations and some work on the build infrastructure we can make codegen in Dart substantially better. I think we learned a lot from the macros effort that can potentially be applied here. I don't think this will be enough in and of itself though, and we will be taking a fresh look at what features we should add to the language to make working with data and serialization great. Many of the ideas mentioned in this thread are on the table, but this discussion is still very open ended. We also of course will be continuing to ship features to keep making Dart more usable, fresh and modern. Static access shorthands are in the implementation pipeline now and we're actively working on getting a few more things spec'ed out and ready for the implementation teams this year. I'm very much looking forward to the seeing what we can build out in the next few years. Thanks for all of the interest and discussion on the issue tracker, and thanks for being part of the Dart community!

[davidmorgan]

Thanks @lucavenir ... I really feel all those points, having worked on macros for a year.

I'd like to reassure that we are not giving up on solving those problems--rather we are now going to tackle them piece by piece instead of, as you say, all in one fell swoop :)

For my part, I'll be starting on that by working on the performance question here.

[TheHudek]

I’m sad that this is now discontinued and don’t get me wrong, build_runner could be way faster, but just out of curiosity what are these

gargantuous wait times

you’re talking about?

I work on medium-sized projects on a medium machine and my build times are about 3-5 minutes.

Again, don’t get me wrong, I’m just genuinely interested since I haven’t seen anyone say anything specific.

[davidmorgan]

I’m sad that this is now discontinued and don’t get me wrong, build_runner could be way faster, but just out of curiosity what are these

gargantuous wait times

you’re talking about?

I work on medium-sized projects on a medium machine and my build times are about 3-5 minutes.

Again, don’t get me wrong, I’m just genuinely interested since I haven’t seen anyone say anything specific.

I consider 3-5 minutes to be gargantuan for anything that is part of your dev workflow :)

It's supposed to be an interactive experience, not "grab a cup of coffee" :)

[amrgetment]

@TheHudek For my team after using generate_for: for build_runner it improved the building time

@davidmorgan I think generate_for needs to be as an annotation like @build or @generate
I wonder could the macros feature split it into smaller features but not canceled
Like the following:

1. Augementations feature
2. data classes feature
3. etc

So instead of a big Macros feature, we would have a Macros Goal, not a feature
WDYT?

[mateusfccp]

I consider 3-5 minutes to be gargantuan for anything that is part of your dev workflow :)

It's supposed to be an interactive experience, not "grab a cup of coffee" :)

This varies a lot depending on the machine. In my old laptop, build times that used to take 5 minutes are now taking 1 minute after switched to a M3 Pro.

However, I always think that people overexaggerate the cost of build_runner. I won't say that it's the most ergonomic of the tools, and all improvements are welcome, but 90% of the workflows don't required you to run it every 5 minutes. In practice, considering that most people use it for json_serializable or freezed, you use it only when you introduce new models, and if you have many models to introduce as part of a feature, you can do it in a single cycle. Using watch is even better, as it is way quicker than building everything again.

I'm more worried about the stability of the tool than the performance. Again, performance improvements are always welcome, but most of my frustration on using build_runner is the fact that it's often crash with no apparent reason or gives not so helpful error messages.

[lrampazzo]

In my opinion, by adding native data classes (possibly with primary constructors) and improving serialization, which are pain points for developers, will greatly reduce the amount of work required by build_runner and thus the execution time.

Combined with your suggested improvements and augmentations (which is btw is a very valuable result of the static metaprogramming topic) will greatly improve dart code generation in general, and macros won't be missed so much.

[davidmorgan]

I consider 3-5 minutes to be gargantuan for anything that is part of your dev workflow :)
It's supposed to be an interactive experience, not "grab a cup of coffee" :)

This varies a lot depending on the machine. In my old laptop, build times that used to take 5 minutes are now taking 1 minute after switched to a M3 Pro.

However, I always think that people overexaggerate the cost of build_runner. I won't say that it's the most ergonomic of the tools, and all improvements are welcome, but 90% of the workflows don't required you to run it every 5 minutes. In practice, considering that most people use it for json_serializable or freezed, you use it only when you introduce new models, and if you have many models to introduce as part of a feature, you can do it in a single cycle. Using watch is even better, as it is way quicker than building everything again.

Indeed that can work pretty well. When it becomes a problem is if your data model depends on your app code--then it wants to run whenever there is a change.

However, build_runner currently has a scalability problem: when you double the codebase size, the build time increases x4. When you get to a codebase size of ~thousands of files this starts to be a real problem. This is the specific issue I plan to fix first.

I'm more worried about the stability of the tool than the performance. Again, performance improvements are always welcome, but most of my frustration on using build_runner is the fact that it's often crash with no apparent reason or gives not so helpful error messages.

Improving stability sounds good to me, filed dart-lang/build#3801 to track. I do want to look at performance first, but there's no harm in starting to collect examples of stability problems. Thanks!

[crefter]

Very sad. Is this dart`s death?

[lucavenir]

I work on medium-sized projects on a medium machine and my build times are about 3-5 minutes.

Wait... that's acceptable? Even on a first start?

As mentioned by others, the problem is the development cycle.

We're speaking about one minute of build times between saves, plus analysis times. And this is happening in a monorepo (i.e. split code), with "optimizations" such as generate_for mentioned above (!)

This, added to the fact that most times build_runner has to re-start from the beginning, and you've essentially blocked the development of a project because of the runner.

@davidmorgan thank you. Addressing runner times is a good starter but, as shown above, that's just the tip of the iceberg, believe me.

[davidmorgan]

@davidmorgan thank you. Addressing runner times is a good starter but, as shown above, that's just the tip of the iceberg, believe me.

I think we are in agreement :)

[davidmorgan]

@davidmorgan I think generate_for needs to be as an annotation like @build or @generate I wonder could the macros feature split it into smaller features but not canceled Like the following:

1. Augementations feature
2. data classes feature
3. etc

So instead of a big Macros feature, we would have a Macros Goal, not a feature WDYT?

That's pretty much how we're thinking of it, yes. Not macros as a feature but all the different things needed to address the main use cases that macros were intended to address.

Augmentations is still proceeding as planned.

[lucavenir]

When it becomes a problem is if your data model depends on your app code--then it wants to run whenever there is a change

Can we ship a lint that suggests not to put logic onto these files? e.g.

@freezed
class DataClass ... {}

extension ApiDataClassExt on AnotherApiDataClassOnAnotherFile {
  // adapter
  DataClass toEntity() => ...
}

This is very, very handy. But because of the runner, I can't simply do this at scale.

[amrgetment]

@lucavenir freezed_lint may help

[lucavenir]

Cool, but that should be part of the whole ecosystem. Not just freezed.

Remember: this issue addresses every static metaprogramming idea we've had.

Examples include Riverpod, GoRouter, etc.

[davidmorgan]

Can we ship a lint that suggests not to put logic onto these files? e.g.

...

Cool, but that should be part of the whole ecosystem. Not just freezed.

No, we won't ask people to keep logic out of their source files to make codegen fast.

We will make it fast for that case :)

That said, it is a good idea to structure your code in a modular way. So for example if you think about how you want your data classes to work, and decide that they shouldn't depend on your UI code, then make that a clear separation: your code will be better and it will probably help the tools work faster.

But we don't want you to have to make your code worse to help the tools. That's a problem with the tools.

[jodinathan]

One of the issues with the current code generation approach is that the client developer needs to manually add part 'file.g.dart'; to the original code. It would be interesting if this wasn’t mandatory while still providing a way for the developer to see the included part directives.

I’ve created an issue for this: #4241