[Self-Managed]: Error: [config validation of [xpack.lists].enabled] on running kibana.bat file while environment setup.

[amolnater-qasource]

Kibana Version: 8.0 Snapshot self-managed environment.

Artifact Link: https://snapshots.elastic.co/8.0.0-89da64e1/downloads/kibana/kibana-8.0.0-SNAPSHOT-windows-x86_64.zip

Pre-requirement:

1. Elasticsearch should be running.

Issue:
On running kibana.bat we are getting below errors:

Kibana.yml:
kibana.zip

Impact:
We are not able to setup self-managed 8.0 Snapshot environment.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[EricDavisX]

thanks for logging. I'm unaware of the change, but will ask if anyone knows. I don't see any PRs or Issues that make mention of xpack.lists (or just 'lists') that may relate. A quick look in slack I do see mention of all 'enabled' fields being deprecated (I think). Let me find the right Kibana folks to see what they think.

[EricDavisX]

@amolnater-qasource is the xpack.lists.enabled uncommented by default in the .yml you are using (was it unmodified out of the box?) - I see one reference to an .enabled file, in kibana.yml:
https://github.com/elastic/kibana/blob/main/config/kibana.yml#L37
and it is security related and commented - were you indeed testing with certs / security on ?

perhaps it is one more item to be fixed wrt the security-on by default effort that is underway. just making guesses.

[mshustov]

lists plugin cannot be disabled starting from #113495
xpack.lists.enabled should be removed from kibana.yml

[amolnater-qasource]

Hi @mshustov & @EricDavisX
Thanks for sharing feedback.

xpack.lists.enabled uncommented by default in the .yml you are using (was it unmodified out of the box?)

We were using it as enabled xpack till 7.16.0 and it was working fine.

Further We have updated our xpack to:

elasticsearch.requestTimeout: 180000
xpack.encryptedSavedObjects.encryptionKey: 'fhjskhsdhd678ehkdfdlliverpoolfcr'
xpack.fleet.agents.tlsCheckDisabled: true

We are able to setup the 8.0 Self-managed environment.

Query:
cc: @jkakavas
As setup passwords interactive command is also deprecated.
Could you please confirm the exact parameter in the CLI where the new password is visible on running elasticsearch.bat.

Sharing the CLI below:
elasticsearch.bat-CLI.txt

Currently we reset the password everytime to get elasticsearch password with: elasticsearch-reset-password -u elastic command.
Thanks

[jkakavas]

Query:
cc: @jkakavas
As setup passwords interactive command is also deprecated.
Could you please confirm the exact parameter in the CLI where the new password is visible on running elasticsearch.bat.

If I understand this correctly, you are asking why don't you get a password printed on startup ? The most possible reason is that we don't run auto-configuration because the elasticsearch.yml with which you start elasticsearch contains security related configuration already. We assume that this is an indication that you want to do your security configuration on your own and don't auto-configure things for you. I will be able to tell you with certainty if you share your elasticsearch.yml

As setup passwords interactive command is also deprecated.

It is deprecated but as you have figured out, you can use elasticsearch-reset-password instead, so this (in itself) shouldn't be an issue right ?

[elasticmachine]

Pinging @elastic/kibana-qa (Team:QA)

[EricDavisX]

Through a quick slack chat this morning, we found the source of the mis-configuration, it is in an old word doc we keep that cites specific steps for setup and which was out of date with this new expectation. we can update it.

I will let Amol confirm back on final question/answer so we are confident in testing / assessment. @amolnater-qasource please do close it out when done, presuming.

[amolnater-qasource]

Hi @jkakavas
Thanks for the details, yes please find elasticsearch.yml file below:
elasticsearch.zip

as you have figured out, you can use elasticsearch-reset-password instead, so this (in itself) shouldn't be an issue right ?

Yes that's not an issue, we just asked as we thought it is expected to be available on startup.

We are closing this as we are successfully able to setup self-managed 8.0 environment.
cc: @EricDavisX
Thanks!

[jkakavas]

The reason that security is not auto-confiugured is that you explicitly set:

xpack.security.enabled: true
xpack.security.authc.api_key.enabled: true

in elasticsearch.yml.

You don't need to set these explicitly any more because the default value for both of these settings in true in 8.0.0 + .If you remove these, we will autoconfigure TLS for elasticsearch ( which might or might not be what you want/need ) and we will auto-generate a password for the elastic user and print it on startup.

[amolnater-qasource]

Hi @jkakavas
Thanks for sharing detailed information.
We will update our self-managed setup steps from 8.0 onwards.

Thanks