[CT-651] Custom merge behavior for configs

[jtcohen6]

See: #3326, #3816, #4105

Most configs today, when defined, will clobber less-specific inherited values. Some configs have different merge behavior, however:

• tags are appended
• meta is merged/updated
• pre-hook + post-hook are additive, and functional... I appreciate that these are quite similar to indexes; I'm not sure if I'd make the same decision today!

Sometimes, it's desirable to clobber tags or pre/post hooks. Sometimes, it's desirable to append/merge custom configs that are clobber-only.

We need to figure out:

• a workable syntax for specifying whether a config should be appended, updated, or clobbered
• how to expose the MergeBehavior for arbitrary configs

Possible syntax

Just some ideas we've played with:

{{ config(tags += ['add', 'to', 'existing'] }}
{{ config(tags -= ['clobber_existing', 'only_these_two'] }}

config:
  tags:
    value:
      - clobber
      - existing
    merge_behavior: clobber

[Fraser-Isbester]

{{ config(tags += ['add', 'to', 'existing'] }}
{{ config(tags -= ['clobber_existing', 'only_these_two'] }}

What about:

{{ config(tags += ['add', 'to', 'existing'] }}
{{ config(tags = ['clobber_existing', 'only_these_two'] }}
{{ config(tags -= ['deletes_only_this_tag_if_exsits_from_less_specifc_config'] }}

Using this syntax generally would allow some pretty powerful patterns and seems like it would cover most use cases.

[jtcohen6]

We've made an initial experimental attempt at this in #5230 for the grants config. One thing that's different from the proposal above: rather than sticking the semantic operators on the config key or in the "set" (=) operator, we'd stick them as prefixes on the config values.

What would it look like to extend that for other configs?

The default behavior needs to remain, for backwards compatibility. We should do our best to give a reasonable default per config type. In the case of tags, unlike some other configs, there's very little "harm" in adding a new one (though this request has certainly come up before). So this would still have the effect of "adding" default_behavior as a tag to the existing list of tags for this node:

{{ config(tags = 'default_behavior' }}

But then we might also support:

{{ config(tags = '+add_this_one' }}
{{ config(tags = '-remove_this_one' }}
{{ config(tags = ['+add_this_one', '-remove_this_one'] }}

version: 2
models:
  - name: my_model
    config:
      tags: default_behavior
      # or maybe this
      tags: '+add_this_one'
      # or this
      tags: '-remove_this_one'
      # or this
      tags:
        - '+add_this_one'
        - '-remove_this_one'

I'm not sure of the right way to enable "clobber" for a config that's additive by default. Definitely open to ideas here. Some kind of "remove all tags via wildcard operator?":

{{ config(tags = '-*', '+just_this_one' }}

Given this capability going forward, I think we'd be more inclined to support clobbering by default for new configs.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.