feat(flags): Create an experiment from a feature flag

[danielbachhuber]

Changes

Allows an experiment to be created when creating a feature flag:

• Adds a new "Create draft experiment" field to the feature flag form.
• When "Create draft experiment" is checked, experiment_set is set to 'new'. This is somewhat magical reuse of the experiment_set property, which is read-only and includes an array of experiment ids attached to the feature flag.
• When "Create draft experiment" is checked, "Multiple variants with rollout percentages" is selected and "control" and "test" variants are assigned.

CleanShot.2025-01-27.at.13.17.10.mp4

How did you test this code?

Manual review. Tests should pass.

[danielbachhuber]

If you have an opinion, a couple bits I'd love feedback on:

• I don't love the Create draft experiment checkbox UI. Is there something better we should use?
• Should it be possible to create an experiment from an existing feature flag? I'm mixed on it.

[github-actions]

Size Change: +31 B (0%)

Total Size: 1.16 MB

ℹ️ View Unchanged

Filename	Size	Change
frontend/dist/toolbar.js	1.16 MB	+31 B (0%)

_{compressed-size-action}

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[dmarticus]

• Should it be possible to create an experiment from an existing feature flag? I'm mixed on it.

After thinking about this for a day, I don't really see the appeal – I feel like adding the big checkbox at the top of the flags creation UI experience that gives folks the option of making a relatively incomplete experiment (compared to the amount of options they have when creating an experiment from scratch) doesn't really add much. Is this in response to customers asking to have the ability to create experiments from the flag UI? Or was this more of a "we can do this easily, so we should"? If it's the latter, my perspective is that I don't think the value add is worth the cost of adding UI clutter to the flags page.

[danielbachhuber]

• Should it be possible to create an experiment from an existing feature flag? I'm mixed on it.

After thinking about this for a day, I don't really see the appeal – I feel like adding the big checkbox at the top of the flags creation UI experience that gives folks the option of making a relatively incomplete experiment (compared to the amount of options they have when creating an experiment from scratch)

@dmarticus To clarify, are you reacting to:

1. The ability to create an experiment while creating a new feature flag?
2. The ability to create an experiment from an existing feature flag?

My original question was about the latter, but it seems like your comment is in response to the former (the current PR).

Is this in response to customers asking to have the ability to create experiments from the flag UI?

This PR is in response to at least a few instances I've seen of "how do I use my existing feature flag with an experiment?" Ostensibly, they go about setting up the feature flag in their codebase first, and then realize it's not possible to create an experiment with that feature flag.

Although, now that I'm saying that, maybe the more appropriate implementation is the ability to associate an experiment with an existing feature flag.

If it's the latter, my perspective is that I don't think the value add is worth the cost of adding UI clutter to the flags page.

It seems like a little bit of a problem that there's zero screen real estate for experiments on the flags page. What do you think would be more appropriate?

[danielbachhuber]

Although, now that I'm saying that, maybe the more appropriate implementation is the ability to associate an experiment with an existing feature flag.

First pass: #28004

CleanShot.2025-01-28.at.16.58.39.mp4

@jurajmajerik @andehen What do y'all think?

[danielbachhuber]

@joshsny ^ If you have some opinions, given you've been working in the area recently.

[joshsny]

@danielbachhuber thanks for picking this up!

#28004 feels like a much more useful implementation - I've often created a flag for testing, switch it on and off etc. and only later decided to run an experiment, so being able to do that feels really helpful.

I think as @dmarticus mentioned for this PR, the additional real estate used in the flags page probably isn't worth it, as if I knew I wanted to create an experiment when creating the flag I'd have probably gone to the experiments page from the get go

[jurajmajerik]

I don’t have strong feelings about this. The visual clutter from this extra field IMO doesn’t seem too bad, but if a user wants to create an experiment, it indeed makes more sense for them to do it from the Experiment form. That said, adding this does improve visibility and cross-sell for experiments, which is not an insignificant benefit.

UI experience that gives folks the option of making a relatively incomplete experiment

The only issue I see is that this always creates a Product Experiment. It would need an option to choose between product and no-code experiments in the future. Any other option is easily configurable in the draft mode.

If it does come to merging this, it would be good to add a link to the newly created experiment in the success toast, opening it in a new tab.

Linking an existing flag is definitely useful and has been requested quite a bit 👍

[dmarticus]

shoot, I had a writeup from yesterday that I forgot to save, and both Juraj and Josh covered the points I was trying to make. So I'll just say:

If it does come to merging this, it would be good to add a link to the newly created experiment in the success toast, opening it in a new tab.

This is what I'd recommend for the creation experience

Linking an existing flag is definitely useful and has been requested quite a bit 👍

Agreed this is most useful

Looking at the UI again I think this all makes sense. I'm fine with it.

[danielbachhuber]

I think I like my other PR better 😄

Ready for initial review: #28004

In particular, I'd like a second pair of eyes on the validation logic for determining whether a feature flag is eligible to be assigned to an experiment.