Allow relocation of secret.yaml

[sshaikh]

Is your feature request related to a problem? Please describe

Although it's appreciated that secrets are no longer held in configuration (#2201) the secret.yaml file can only be found in data/. This is a tiny bit irritating as data/ is part of Z2M's git tree and so secret.yml will always appear as an untracked file there - this is also especially inconsistent if you've relocated configuration.yaml using the ZIGBEE2MQTT_CONFIG environment variable.

Describe the solution you'd like

To be able to relocate secret.yml, either via a similar environment variable (ZIGBEE2MQTT_SECRETS_DIR?) or by automatically looking in the same directory as configuration.yaml.

Describe alternatives you've considered

I could add data/secret.yaml to the ignore file and commit that, but then I'll always be one commit ahead of upstream master.

[tricoos]

I just thought I should open a bug ticket because the information on how to specify the path to the "secret.yaml" file is missing in the documentation. As it specifically says "e.g secret.yaml" - "e.g." (!) on https://www.zigbee2mqtt.io/information/configuration.html I assumed that just like "devices.json" I could just specify the path to the file and the info on where to specify the name it is just missing here.

Also, I find it misleading to put both "mqtt" and "advanced" options into a file named "secret.yaml".

[Koenkk]

@sshaikh I've added de data/*.yaml to the ignore files.

Instead of ZIGBEE2MQTT_CONFIG you can use ZIGBEE2MQTT_DATA. Since ZIGBEE2MQTT_CONFIG provides strange behaviour I marked it as deprecated, users should use ZIGBEE2MQTT_DATA instead. Docs: https://www.zigbee2mqtt.io/information/configuration.html#different-location

[tricoos]

@Koenkk So I assume that the "e.g." on https://www.zigbee2mqtt.io/information/configuration.html is wrong? ;)

[Koenkk]

@tricoos what "e.g." do you mean (there are many of them on this page)

[tricoos]

@Koenkk I quoted above "e.g secret.yaml". That confused me because I searched for a config param to specify the path.
I actually don't like the way it is with a "secret.yaml" file name that you cannot specify but you can specify the file names for devices and groups. It works but it is not consistent. ;)

[Koenkk]

@tricoos can you make a proposal on how to refactor this? Note that secret is a little bit different then devices/groups because it also has to point to a property.

[tricoos]

@Koenkk Do you mean a logical proposal or a code proposal? I could only do the first.

[Koenkk]

@tricoos a logical proposal (what you want to put in configuration.yaml.

[sshaikh]

I've moved to using ZIGBEE2MQTT_DATA which fixes the issue as described if you'd like to close it.

I would suggest that CONFIG and DATA are different things though, as I'd like to source control config but not necessarily things like the database. But for now I can git ignore the files I feel are superfluous.

[tricoos]

@Koenkk Here is my proposal. I don't like if there is a single file that picks several attributes from various consistent blocks (like mqtt credentials but not the rest) and the network_key from the advanced section. However, splitting the security-related data is definitely useful.
Thus, what we actually do here - and I consider this very useful - is specifying a part of the values somwehere else. However, this might also confuse a bit so I suggest the following:

There is a new includes section that you can use to override any aspect of the configuration file as long as you use the same hierarchy as in the configuration.yaml file.

Sample configuration.yaml (just as before, but with the new includes list)

mqtt:
  base_topic: zigbee2mqtt
  server: 'mqtt://localhost:1883'
  ca: '/etc/ssl/mqtt-ca.crt'
  key: '/etc/ssl/mqtt-client.key'
  cert: '/etc/ssl/mqtt-client.crt'
  user: my_user
  password: my_password
...
includes:
  - secret.yaml
  - homeassistant.yaml
  - ..

secret.yaml

mqtt:
  user: my_real_user
  password: my_real_password
advanced:
   network_key: [1, 3, 5, 7, 9, 11, 13, 15, 0, 2, 4, 6, 8, 10, 12, 14]

homeassistant.yaml

homeassistant: true
advanced:
  homeassistant_discovery_topic: 'homeassistant'
  homeassistant_status_topic: 'homeassistant/status'
  homeassistant_legacy_triggers: true

This allows you to even group related configuration values that reside in different parts of the config (like with homeassistant).

Notes:

• Regardless of the position of the includes part in the original config file it is replaced after parsing all of the data in the file
• With multiple include files specified they are replaced in the order the files are listed (i.e. if I override or define one value in the first includes file I can still replace it in the third one in the list). The last one always wins.
• This also means that the configuration file can now be split in as many parts as the user wants - a configuration value might be missing in configuration.yaml, but specified in one of the include files later.

Actually, as I have just split my Zigbee network into two I would now be able to use the same configuration.yaml but add a custom.yaml include file where I only list the differences (PAN id, network key, channel, ...)

[Koenkk]

@tricoos thanks, I like the proposal a lot, since this is not very easy to implement (because Zigbee2MQTT will e.g. add devices to devices:, and this needs to be saved back to the correct file) and because I want to get some feedback from other users I've created a new feature request for this: #4806

Since the OP problem has been solved I will close this one.

[JeanMertz]

I'm sorry for commenting on an old + closed issue, but I wanted to +1 the comment made by @sshaikh:

I would suggest that CONFIG and DATA are different things though, as I'd like to source control config but not necessarily things like the database. But for now I can git ignore the files I feel are superfluous.

I'm deploying zigbee2mqtt on Nixos, and would like to have the configuration live (symlinked from the nix store) in XDG_CONFIG_HOME, but have the database/state files live (non-symlinked) in XDG_DATA_HOME.

[qm3ster]

It would be ideal to be able to pass the root config file via env var or argument, independently from the working directory/include resolution directory.
(commenting here because this is not actually about .yaml includes but more about the other files like coordinator backup)
(should probably be a new issue (I nominate @JeanMertz))
(it came to me in a dream)