Add tests with preconfigured policies

[jalvz]

See elastic/kibana#94509

[mdelapenya]

I wonder what do we want to verify here:

• is it possible to add a preconfigured policy? I think this is covered by the unit tests in the kibana repo (see https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/preconfiguration.test.ts).
• what happens after a preconfigured policy is added? I'd say that there are multiple combinations here, i.e.: if we preconfigured apache integration in the policy, what is the expected outcome in an agent using that policy?

@ruflin, as Juan is not our point of contact anymore, who should we contact from now on?

[ruflin]

I assume the tests in Kibana mostly cover that the API works as expected. What I hope to see in the e2e is that the kibana.yml file is used for the configuration, checked that the policies are created and most importantly, that then the Elastic Agents enroll and ship data as expected.

Recently an additional change has been merged that some of the configs can be changed across restarts and will be applied. It would be great to confirm this with e2e. @jen-huang I wonder if Kibana tests have any way of testing this today?

Moving forward, best directly coordinate with @jen-huang

[jen-huang]

Recently an additional change has been merged that some of the configs can be changed across restarts and will be applied. It would be great to confirm this with e2e. @jen-huang I wonder if Kibana tests have any way of testing this today?

I just check our unit tests and that case, introduced in elastic/kibana#103722, is not covered, but it should be possible to cover it. Btw, this only applies to managed preconfigured policies (ones with is_managed: true). I opened a tech debt issue to add unit tests for this in elastic/kibana#106606.

It's still good to have e2e test for this feature since the unit tests don't mock kibana.yml directly.

[mdelapenya]

It's still good to have e2e test for this feature since the unit tests don't mock kibana.yml directly.

@jen-huang let's cooperate in creating the right test scenario for the feature.

It'd be great if we are able to express the different expected behaviours following this approach:

Given a precondition (optional)
When something happens
Then and outcome will be verified

where each block is written in plain english hiding implementation details, such us clicking some UI element. Instead we prefer using high-level, business language.

[jen-huang]

@mdelapenya Do you need my help in starting the the scenarios? The PR elastic/kibana#94509 and docs have a good amount of information.

[mdelapenya]

Yes, I've read the PR and it contains super helpful information although I think it's more related to unit testing the feature. And the docs will be the source of truth in terms of configuration.

What I'm more interested in is in defining business-oriented use cases, with a high abstraction about the internals of the preconfigured policies, and there I think I cannot help as I do not know much about the feature. I can for sure help out modelling the scenarios to reach great and maintainable ones.

I'm able to zoom for further online discussions if you need it :)

[EricDavisX]

@jen-huang hi - I'm scanning through the e2e-testing tickets and came across this. One of the Engg Productivity resources can probably make some progress, but I wanted to note that doing feature-work tests isn't the primary focus of either of the Engg Productivity teams (just honestly speaking). They focus first on infrastructure/framework efforts (and training us on using them) and adding 'initial tests' as examples. I can discourse more off line, it may be a lack of communication about the team's intentions, and we can do better for sure.

About progress on this, it is a fairly long read through of that PR to infer all of the spec decisions to lay out test cases. And even then, the question is of what coverage do we have in other tests (API or unit tests etc) is extremely relevant, hence the suggestion / request for someone close to the feature (me in some cases) or the engineer themselves to pair up instead of having 'qa' resources pushing through without team support.

Finally, I acknowledge, sometimes a single e2e-test is warranted even with other coverage. So, if we want to add in a first test, we can work that and discuss more details later, which is a passive form of prioritization for sure, which is ok by me for now. I appreciate you and the team, thanks for the discussion.

[EricDavisX]

@IgorGuz2000 depending on what other issues are assigned, would you be interested to read through the PR and put in a simple test? I can try to flesh it out with you when you get to it.

[mdelapenya]

@nchaulet, after our conversation I created #1644 providing support for running scenarios with different kibana profiles. That will enable your work here. Please let me know if you have any question or doubt

[nchaulet]

Resolve by #1680