[Lens] Datasource dimensions are not cleared properly

[mbondyra]

Kibana version:
main, only 8.9 caused by [2c491b5] [CM] - lens (#155700)

When updated, the Lens object is being deep-merged so all the properties ever set stay in the saved object even if they were removed. What is odd is that the save call gets the proper response from the backend and Lens app works on correct saved object (removed properties are cleared) but after refreshing we can see that saved object from the DB keeps everything unless it was overwritten.

Here's the flow to reproduce for datasource columns:

1. Create a visualization with some dimensions and save

2. Remove two bucket dimensions and save

3. Go to /app/management/kibana/objects and inspect your saved object.
   The dimensions are still there:

I've put impact medium, but it's imo potentially high as it can create incorrect saved objects that will be difficult to clean up and cause errors we cannot predict. @stratoula feel free to change.

The side effects include for example:

1. if remove the runtime field and then fix the visualization by removing the dimension using the field, we will get error message impossible to get rid of.
2. If we remove the dimension that shows an error coming from datasource, we will never be able to fix visualization.

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[Dosant]

I think the problem is in this update:

kibana/x-pack/plugins/lens/server/content_management/lens_storage.ts

Line 220 in 2c491b5

const partialSavedObject = await soClient.update<LensSavedObjectAttributes>(

It adds new properties and doesn't remove the old ones. I think the properties must be explicitly set to null to remove them.

The old code worked because it used create with overwrite:true:

kibana/x-pack/plugins/lens/public/persistence/saved_object_store.ts

Lines 60 to 68 in 93256a7

	const result = await this.client.create(
	DOC_TYPE,
	attributes,
	savedObjectId
	? {
	references,
	overwrite: true,
	id: savedObjectId,
	}

Interesting that, for example, dashboard so won't have the same problem because panels are stored as a json string.