Would current approach to _ex methods lower QOL?

[mhgolkar]

Problem?

I'm not sure if this is a legitimate concern, so there is a question mark on top of everything,
but there are things about #332 that feel capable of lowering Quality-of-Life:

1. Devil's (perhaps) in the defaults

First thing that occurs to me, is the amount of decisions made on behalf of developers via default methods (without _ex).
Some of these decisions are normal, but others can really make debugging a relatively large project a little problematic.

Here is an example:
Previously if a developer wanted to create a SceneTreeTimer, they had to decide for every single parameter.
In a quick-and-dirty way it would look like this:

let timer = scene_tree.create_timer(secs, true, false, false);

This is pretty much deliberate and educated, yet of-course a little longer.
Now with the ease provided by #332, they can go with:

let timer = scene_tree.create_timer(secs);

The method makes few decisions for us (i.e. defaults):

ScneneTree.create_timer(
    time_sec: float,
    process_always: bool = true,
    process_in_physics: bool = false,
    ignore_time_scale: bool = false
) -> SceneTreeTimer

Here is the catch/concern:
The parameters process_always, process_in_physics and ignore_time_scale which can easily affect your game (e.g. in case of pausing the scene-tree) are only available in the time of initialization. The SceneTreeTimer does not expose them as properties or set/get methods. Forgetting this simple fact, a developer may easily get confused, why their timer does not respect pausing the scene-tree and gives them no way to change it!
They should have done this:

// Now:
let timer = scene_tree.create_timer_ex(secs).process_always(false).done();
// Previously:
// let timer = scene_tree.create_timer(secs, false, false, false);

Godot reminds us of these decisions being made (almost constantly, via popups in its editor or language server docs/hints), but in case of Rust-GDExt, we (at least for now) don't have access to that.

2. Syntax can get Weird

Specially in case of short methods, we are going to have some weird syntax.
Before #322 we had:

tween.set_loops(1);

Now we have:

tween.set_loops_ex().loops(1).done();

Suggestion

If this is actually something of a legitimate concern, and not just my taste,
I would suggest following approach(es) to address it:

1. Let's put the the genie back in the bottle and keep the gold!

I'm not opposed to #332 totally. The extender is a nice idea actually when you want it;
but does it need to be forced?

If it is possible regarding internals of GDExt, one way could be to leave the main (_ex less) methods to behave as they used to.
In that case developers could choose if they want to go with the defaults or set everything deliberately.
That would hint educated decisions, may actually seem cleaner in some cases, and solves the above-mentioned syntax weirdness.

// Normal method, with a default changed:
let to = scene_tree.create_timer(secs, false, false, false);
// ... instead of current syntax:
// let tn = scene_tree.create_timer_ex(secs).process_always(false).done();
// 
// When defaults are preferred:
let to = scene_tree.create_timer_ex(secs).do();
// .. instead of current syntax:
// let tn = scene_tree.create_timer(secs);

And yes, I suggest .do() instead of .done() as well, to protect our hands from typing fatigue.

2. Reminding developers constantly

We can follow the Godot's approach and make sure all the defaulted methods (without _ex) remind the developer of the decisions made on behalf of them. We can have method level documentations pointing to the defaults. Albeit it does not solve the issue with weird and needlessly long syntax mentioned above.

[lilizoey]

one issue that has been discussed before with solution 1, is that godot adding a new default argument to a function becomes a breaking change for us. whereas it isn't a breaking change for godot.

[mhgolkar]

... Godot adding a new default argument to a function becomes a breaking change for us.

That is the case for sure, but I don't think it would happen frequently.
As far as I'm aware, Godot developers are really careful about breaking changes.
On the other hand, QOL decisions can stay with us for ever, and definitely affect more ground and larger amount of work.

And by the way, if there is a breaking change, meaning a new decision made in the background by the new API for the developer, I would prefer to face it as a broken piece of code, rather than an unexpected behavior in runtime.
Another developer can only use the _ex versions which means they wouldn't get the compile time error at all, even if there is a new parameter added to a method.

[Bromeon]

Thanks a lot for bringing this up! 🙂
Always interesting to hear feedback.

Now with the ease provided by #332, they can go with:

let timer = scene_tree.create_timer(secs);

The method makes few decisions for us (i.e. defaults):

ScneneTree.create_timer(
    time_sec: float,
    process_always: bool = true,
    process_in_physics: bool = false,
    ignore_time_scale: bool = false
) -> SceneTreeTimer

Note that this is precisely the behavior that GDScript and C++ offers, and the whole point of default parameters: you don't have to provide values for them. But let's look into details more closely 😉

---

Godot reminds us of these decisions being made (almost constantly, via popups in its editor or language server docs/hints), but in case of Rust-GDExt, we (at least for now) don't have access to that.

Is the problem missing documentation? If yes, that's something that will be addressed sooner or later. gdnative already has documentation for engine classes and methods.

Popups happen in any modern IDE as well, the _ex variant is appearing in auto-completion next to the base variant:

---

Specially in case of short methods, we are going to have some weird syntax. Before #322 we had:

tween.set_loops(1);

Now we have:

tween.set_loops_ex().loops(1).done();

Unfortunately, we have to live with the fact that Rust has neither named nor default parameters. I don't like the builder pattern either, in Kotlin we could write much nicer code. But it is what it is, and from all the possibilities, builders seemed the most sane and idiomatic (there was extensive discussion in godot-rust/gdnative#814).

Commit 01ce0bb is also a good example how code in general gets more readable with #322. Have you seen the diff? I agree that in some cases the _ex makes things a bit more verbose, but in many cases, it has the opposite effect. Let me show you some examples:

1. If we go with your suggestion, people would need to write

   // either:
   parent.add_child(
       child,
       false,
       node::InternalMode::INTERNAL_MODE_DISABLED,
   );
   
   // or:
   parent.add_child_ex(child).do();

   However, with the change, the most common usage stays short:

   parent.add_child(child);

2. Regarding your example above. Compare your suggestion:

   let to = scene_tree.create_timer(secs, false, false, false);

   with the current gdext API:

   let to = scene_tree.create_timer_ex(secs)
       .process_always(false)
       .process_in_physics(false)
       .ignore_time_scale(false)
       .done();

   Which one is easier to understand, two months after you wrote the code? The one with named arguments or 3 boolean literals in a row?

3. Some lower-level APIs have an excessive amount of parameters. For example, RenderingServer::canvas_item_add_triangle_array comes with 4 required and 5 optional parameters. If the user is not interested in default values, requiring them means not only more boilerplate, but also that they have to look up what the default value even is in the first place, to make sure semantics are unchanged.

---

And yes, I suggest .do() instead of .done() as well, to protect our hands from typing fatigue.

That's something we could have a look at. I've never seen "do" in builder APIs though, "build" or "done" seems more frequent. But maybe that's related to do being a keyword in most languages.

Have you seen such Rust APIs anywhere?

---

That is the case for sure, but I don't think it would happen frequently.
As far as I'm aware, Godot developers are really careful about breaking changes.

The thing is, adding a default parameter is not a breaking change in C++ and GDScript, but it is in Rust -- at least if we go with the "full" method syntax. We can definitely not expect this to happen rarely just because our language is annoying in this regard.

It also has very practical implications: once we are on crates.io, any such change (added default parameter) means we have to release a semver-breaking version. If we don't detect it, we violate versioning guarantees.

---

And by the way, if there is a breaking change, meaning a new decision made in the background by the new API for the developer, I would prefer to face it as a broken piece of code, rather than an unexpected behavior in runtime.

New default parameters should of course be added in a way where the default preserves the old behavior. But that's API design on Godot's part -- I'm confident that they regard this, but I cannot uphold it. The same is true for any breaking behavior in existing functionality, you don't need to add parameters if you want to break user code relying on certain assumptions. As such, I don't see this as an argument against the current system.

[mhgolkar]

Hi @Bromeon
Thanks for the attention

Note that this is precisely the behavior that GDScript and C++ offers, and the whole point of default parameters ...

Yes, sure. But shall we consider Rust as just another GDScript or C++? Developers can use GDScript with much less hassle, so what is the point of GDExtension? Performance is not the answer because in many cases, people use another language mainly because of its different ergonomics. Rust is different, and the way we use it may be as well. Rust does not have null, it has Option, and there is a reason for it. There are reasons and concerns for other features as well. I think you know this even better than me. Leaning towards the Rust way of doing things may prove healthier in the long run. Yet it's a matter of taste. I may be wrong.

Is the problem missing documentation?

Not really. Documentation is just the last resort. The real issue is ergonomics.

Unfortunately, we have to live with the fact that Rust has neither named nor default parameters...

Many features of Rust that may seem like quirks, are actually things some people love about it.

Commit 01ce0bb is also a good example how code in general gets more readable with #322. Have you seen the diff?

Not only I have seen that, I've already change 29 modules of my own project and continued my work.
(Yes I've taken risks of developing a large-scale work based on your WIP project. This is how much I trust GDExt's future!)
Believe me I would never brought it up, if I doubted that everyone would benefit from it in the long run.

... Which one is easier to understand, two months after you wrote the code? The one with named arguments or 3 boolean literals in a row?

As you mentioned above correctly, "[Nice stuff] happen in any modern IDE as well".
I see it (through rust-analyzer) and find things clear with arguments in a row even after month.
Yet again it is mostly a matter of taste; but it is also about developer's freedom to choose

Please note that I'm not against extender approach, I think it is better not to be forced, and can be a helper.
One may even put it behind a feature/flag. Why not?
The question is: shall we tolerate syntax like tween.set_loops_ex().loops(1).done(); and sweep all the parameters we know exist under the carpet, or should we allow optional use of parent.add_child_ex(child).done(); that seems (at least to me) much less troublesome, much more deliberate.
Isn't it more hygienic to be open, clear and aware of details in the system (~ Rust ideal and focus on compile-time awareness).

Nothing stops us from choosing to use the _ex method deliberately, where the "APIs have an excessive amount of parameters".

... Some lower-level APIs have an excessive amount of parameters.

This is when we may need _ex and we can use it freely and knowingly.
My suggestion would not mean throwing it away, it is just about having the option to be deliberate about decisions.

I've never seen "do" in builder APIs though,... maybe that's related to do being a keyword

You're right. Double checked and it is a reserved keyword, so that's the part we can forget about.

The thing is, adding a default parameter is not a breaking change in C++ and GDScript, but it is in Rust ...

The only true question here is: do we benefit in terms of ergonomics?
If the answer is yes, I think ergonomics are always in higher priority and the versioning hassles are worth it.
At the end of the day, you know this system better than anyone else, and shall decide which approach really helps everyone.

---

I appreciate the great job you and the GDExt contributors have done.
It is shining even in its current work-in-progress state.
Thank you.

[Bromeon]

As you mentioned above correctly, "[Nice stuff] happen in any modern IDE as well".
I see it (through rust-analyzer) and find things clear with arguments in a row even after month.
[...]

Hm, good point. But in this code example, the difference is mostly a matter of typing, no? Visually both variants occupy the same space, and provide the same information. And is typing a real issue, given IDE completions?

Or is the main concern that you might miss parameters that are introduced in newer versions?

---

Yet again it is mostly a matter of taste; but it is also about developer's freedom to choose

One may even put it behind a feature/flag. Why not?

There are two problems with that:

1. If there are redundant ways to address the problem, none of which offers objectively obvious benefits, which one should a user choose? There is a lot of value in having fewer options when using APIs: consistency, less bikeshedding, easier recognizability of code written by others, no need to document "when to use which".
2. A feature flag adds quite a bit of technical complexity -- it multiplies the design space in which the library can be used. This means there are now much more combinations that need to be maintained, tested in CI, that can cause bugs, etc. This increases exponentially with the number of "choices" a user can make, which is why features should be added rather carefully, preferably when they're orthogonal to other parts in the library.

But more on that in the next section...

---

The only true question here is: do we benefit in terms of ergonomics?
If the answer is yes, I think ergonomics are always in higher priority and the versioning hassles are worth it.

That being said, we already have a very similar problem elsewhere: enums.
Godot can add new enumerator values in a future version, which is a breaking change. So we have multiple options to represent an enum in Rust:

• Use enum -- forces the user to handle all cases, which can be a nice thing. But it means that code may be broken in the future.
• Use enum with #[non_exhaustive] -- forces the user to always write a _ branch. This is forward-compatible, but means the user may forget to handle new enumerators if they're introduced.
• Use const instead of enum -- also possible, but even less type support (no IDE completion in match, for example).

So what I was thinking about was a feature forward-compat (or no-format-compat, depending on how we model it), which would change how some symbols are mapped to Rust. But it would be all-or-nothing, not selective choice per symbol.

What if under that feature, default parameters would have such an API? Or would that still be a problem regarding "possible to forget some in the base case"? Basically I'm looking for a way to enable requiring all arguments while also keeping common things like add_child(node) simple.

// base case
let to = scene_tree.create_timer(secs);

// extended case -- user must provide all
let to = scene_tree.create_timer_ex(secs, false, false, false);

---

Not only I have seen that, I've already change 29 modules of my own project and continued my work.
(Yes I've taken risks of developing a large-scale work based on your WIP project. This is how much I trust GDExt's future!)
Believe me I would never brought it up, if I doubted that everyone would benefit from it in the long run.

I appreciate the great job you and the GDExt contributors have done.
It is shining even in its current work-in-progress state.
Thank you.

Thanks a lot for the confidence! Very happy to hear that gdext has helped you so far 🙂

[mhgolkar]

Hm, good point. But in this code example, the difference is mostly a matter of typing, no? Visually both variants occupy the same space, and provide the same information. And is typing a real issue, given IDE completions?

Putting them in one line, the first one is by far shorter, and still has the hints.
Anyway the thing that matters there is that although they both look the same, one encourages/forces developer to stay in control and micro-manage behavior(s) of the call.

That being said, we already have a very similar problem elsewhere: enums.
... So what I was thinking about was a feature forward-compat (or no-format-compat, depending on how we model it), ... But it would be all-or-nothing ...

I'm always in favor of deliberate decision-making, awareness and control of details.

In case of enums I think there are two kinds of them:

1. Input enums
   These are enums that are chiefly/only used as arguments, including flags such as object::ConnectFlags.
   Developers rarely need to handle these enums, they are just passed. I think it is safe here to use enum with #[non_exhaustive].

2. Output enums
   They are mostly/only returned by methods. The most common one is global::Error.
   A new variant in these enums means either a new opportunity or a new threat. In either case, the developers should or are better be aware of new situation and handle it. Even if it is not new, matching to variants of an output is always a good idea when you want to make sure all the possible outcomes are known and properly handled and it's easier with enums.
   Here I firmly suggest going with normal enums.

I sure agree that using const instead of enum cannot be the perfect design in terms of hygiene or ergonomics, mainly for the concerns you mentioned.

One other point that comes to mind is that there should be a level of coherency between enums and their receptors or producers. In other words, if we are going with standard Rust enums, the methods that accept or return it, are better as well to work with the enum directly and not the integer behind it, and if we are going to implement them as consts the type should be the same (both sides i32 or both sides u32 to avoid need for conversion). This may seem obvious, but is easily missed.

What if under that feature, default parameters would have such an API? Or would that still be a problem regarding "possible to forget some in the base case"? Basically I'm looking for a way to enable requiring all arguments while also keeping common things like add_child(node) simple...

Your solution does it well, where priority is ease of use:

// base case -- with defaults that we are expected to be aware of
let normal = scene_tree.create_timer(secs);
// extended case -- user must provide all
let deliberate = scene_tree.create_timer_ex(secs, false, false, false);

Few things are about it to be addressed though:

• It is all-or-nothing. We shall either use the short form or provide all the arguments, while the other approach allows setting only one or two parameters when needed.
• It does not really make an ergonomic change: If the raw method is the second choice (which is telegraphed by the _ex method name in the context above), there is little difference whether I input all the arguments (this suggestion) or only those that I need (current state). I'm less exposed to the internals anyway. It solves the set_loops_ex().loops(1).done() syntax issue, but is a regression in case of much longer methods when the developer still wants control, the problem with very long and argument-demanding methods you were trying to solve remains if we are still concerned about control.

The other solution is still towards the same basic goal you mentioned, but tries to address those objectives as well, by turning the question upside-down. The priority here is awareness & control while in a level it also keeps the freedom of setting parameters that exist in the _ex methods (with #322) when they are really preferred (e.g. the case of long methods). I believe this is optimal solution addressing maximum number of issues.

// base case -- user must provide all (so must think consequences ahead)
let normal = scene_tree.create_timer(secs, false, false, false);
// optional -- user knowingly opts in to use defaults
// (and have the power to add one or two params when desired)
let deliberate = scene_tree.create_timer_ex(secs).done();

Thanks a lot for the confidence!

Thanks again for your efforts.

p.s.
@Bromeon and @lilizoey;
I edited this comment with few additional thoughts.

[lilizoey]

In case of enums I think there are two kinds of them:

1. Input enums
   These are enums that are chiefly/only used as arguments, including flags such as object::ConnectFlags.
   Developers rarely need to handle these enums, they are just passed. I think it is safe here to use enum with #[non_exhaustive].

I would like to note that ConnectFlags is a bitflag enum, so modelling it as a rust-enum would not really be the best solution. Since you can pass for instance CONNECT_DEFERRED | CONNECT_ONE_SHOT.

---

One possible solution to the api-compatibility and semver issue is that we could add an edition-like system.

So each gdext library will declare an "edition", probably matching a minor version of godot, like say 4.1. And if they declare that then gdext will always generate an api that matches what the api looked like in 4.1.

If godot now adds a default argument to a function in 4.2, then we just wont let the user pass that default argument in through the normal function call, and it will always be the default. instead they'll need to either use the extender, or change their edition to 4.2.

This should prevent any semver breaking changes that aren't semver breaking in godot, since the api will always match what a library was made for. Though we should probably require people to specify the edition, maybe something like (actually this exact syntax probably wont work):

struct MyExtension;

#[gdextension(edition = 4.1)]
unsafe impl ExtensionLibrary for MyExtension {}

So now we can have what @mhgolkar is suggesting with the default function having all parameters, required and default. In addition to the extender. Without fear of breakage.

[Bromeon]

Isn't it more hygienic to be open, clear and aware of details in the system (~ Rust ideal and focus on compile-time awareness).

Anyway the thing that matters there is that although they both look the same, one encourages/forces developer to stay in control and micro-manage behavior(s) of the call.

I'm still not sure if I fully understand this desire of control. Previously you also mentioned safety (which I understood as robustness against errors, not unsafe), but I think you edited and deleted that.

Yes, it's good to be in control -- but default parameters are an explicit "you don't have to care" feature.
They are like adding a new method in a class, or a new type in a module -- or any other forward-compatible change.

None of the above "notifies" you during library updates that the API has changed, because your existing code has no reason to stop working. You may or may not use the additional features, but there is no harm if you don't. We don't break user code if Godot adds new APIs, so I don't really see why we should do it when optional parameters are added?

Adding an enum variant on the other hand is not necessarily forward-compatible -- a new case is introduced, which existing code doesn't handle. In cases where this is acceptable (aka by design), #[non_exhaustive] is appropriate.

---

In case of enums I think there are two kinds of them:

This is a great observation. Somehow the in/out principle reminds me of variance and covertibility (Kotlin has a good explanation).

It will be hard to classify enums in an automated way though -- and if Godot starts using enums in return positions, we would again be facing breakage.

[lilizoey]

In case of enums I think there are two kinds of them:

This is a great observation. Somehow the in/out principle reminds me of variance ...

huh, you're right. that is exactly what this is.

That is, say we have two enums A and B, such that every variant of A is also a variant of B. then we can say that A is a subtype of B. (of course rust doesn't really have subtyping but anyway)

Then we know that a function that takes an input of type B can be treated as a function that takes an input of type A (contravariance, since fn(B) is a subtype of fn(A)).

and if a function returns a value of type A, we can treat it as a function that returns a value of type B (covariance, since fn() -> A is a subtype of fn() -> B).

So now if we have an enum which has an extra variant added to it, this means the old enum is a subtype of the new one. So we can safely update any function that takes this enum as an input to the new enum, since those functions are subtypes of the old functions. but we can't safely update any function that returns it as an output since the new function is a supertype of the old one.

(or equivalently, if an enum is only ever used as an input, it's not breaking to update. but if it's used as a return type, then an update is breaking. this is obviously ignoring all the other aspects to what makes something a breaking change.)

[mhgolkar]

I find everything above reasonable and agree.

Just a few thoughts:

I'm still not sure if I fully understand this desire of control

I don't understand it sometimes too! Maybe it's just a philosophical obsession.
The decision is yours, which level of it works for GDExt and when we should forget about it.

which I understood as robustness against errors, not unsafe ...

Yes. You're correct. I edited that to "awareness and control", trying to dodge technical baggage of safety.
"Robustness against errors" is a valid reading.

None of the above "notifies" you during library updates that the API has changed, because your existing code has no reason to stop working.

The whole point is to make sure there will be some kind of (Rust) barrier (even artificial) notifying or forcing reaction, when changes happen. In case of all-provided arguments with optional extender, when the API (and probably my GDExtension edition) changes, then my code won't compile because I haven't provided a newly added required argument. It is all unless I've deliberately chosen to use extender and accept the defaults, which means my code feels nothing.

You may or may not use the additional features, but there is no harm if you don't. We don't break user code if Godot adds new APIs, so I don't really see why we should do it when optional parameters are added?

New API (method) is something you may use or not. New parameter/argument is a potential new behavior for the method you may have already used. Although in an ideal world (and thanks to all the cautious opensource developers out there in our not-so-ideal world) it is most-likely harmless, but there is no guarantee, specially in micro level.

... this means the old enum is a subtype of the new one ... if an enum is only ever used as an input, it's not breaking to update. but if it's used as a return type, then an update is breaking.

I couldn't explain it any better!

[Bromeon]

The whole point is to make sure there will be some kind of (Rust) barrier (even artificial) notifying or forcing reaction, when changes happen.

This may be possible in the specific case of default arguments, but is generally doomed to fail.

Instead of add_child(child, optional_arg), Godot could just have a 2nd method:

add_child(child)
add_child_with(child, optional_arg)

In C++ or C#, they could even use an overload:

add_child(child)
add_child(child, optional_arg)

The 2nd method is functionally identical to a default parameter. Making one artificially breaking while the other is forward-compatible looks arbitrary to me.

---

Don't get me wrong -- I understand the desire to be notified about forward-compatible changes. But forcing it into the API is the wrong approach in my opinion. It's also technically impossible in most cases apart from default parameters.

You should consider alternatives that are more promising, e.g.:

• Read changelogs (usually they're categorized, so you can skip irrelevant aspects)
• Write a tool that compares two extension_api.json files and points out differences
• Test functionality whose stability is important to you
• Delay updates to make sure early bugs are ironed out

---

New API (method) is something you may use or not. New parameter/argument is a potential new behavior for the method you may have already used. Although in an ideal world [...] it is most-likely harmless, but there is no guarantee, specially in micro level.

As mentioned earlier, Godot developers can also change semantics without adding a default parameter. This is no reason to revisit every single function you use.

[mhgolkar]

Well... I don't have any other argument to offer (and everything was under a question mark from the beginning), so you can close this issue if we've reached the conclusion.

Thanks a lot for your time.

[Bromeon]

Thanks a lot for the interesting discussion, I really enjoyed it! 👍

Even if the API is probably going to stay for the time being, there were some really good inputs regarding compatibility, ergonomics, enum variance and generally different trade-offs, some of which may be reconsidered in the future. Thanks a lot!