Soft deleted entity ML Model still appears in the UI

[jazzblue]

Describe the bug
After soft deleting mlModel entity still appears on UI despite what the docs say here.

To Reproduce
Steps to reproduce the behavior:

1. Follow these instructions to spin up Datahub locally (quickstart) and install Python dependencies.
2. Run the following Python code to create ml modelGroup, mlModel and then soft delete that mlModel:

import datahub.metadata.schema_classes as models
import datahub.emitter.mce_builder as builder
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter


rest_emitter = DatahubRestEmitter(gms_server="http://localhost:8080")

model_group_urn = builder.make_ml_model_group_urn(
    platform="myplatform", group_name="mymodelgroup", env="PROD"
)


model_urn = builder.make_ml_model_urn(
    platform="myplatform", model_name="mymodel", env="PROD"
)

# Create model group
metadata_change_proposal = MetadataChangeProposalWrapper(
    entityType="mlModelGroup",
    changeType=models.ChangeTypeClass.UPSERT,
    entityUrn=model_group_urn,
    aspectName="mlModelGroupProperties",
    aspect=models.MLModelGroupPropertiesClass(
        description="My model Group",
    ),
)
rest_emitter.emit_mcp(metadata_change_proposal)

# Create model
metadata_change_proposal = MetadataChangeProposalWrapper(
    entityType="mlModel",
    changeType=models.ChangeTypeClass.UPSERT,
    entityUrn=model_urn,
    aspectName="mlModelProperties",
    aspect=models.MLModelPropertiesClass(
        description="My model", groups=[model_group_urn]
    ),
)
rest_emitter.emit_mcp(metadata_change_proposal)

# Delete model
metadata_change_proposal = MetadataChangeProposalWrapper(
    entityType="mlModel",
    changeType=models.ChangeTypeClass.UPSERT,
    entityUrn=model_urn,
    aspectName="status",
    aspect=models.StatusClass(removed=True),
)
rest_emitter.emit_mcp(metadata_change_proposal)

3. Navigate to the UI: http://localhost:9002/
4. Drill down: ML Groups -> mymodelgroup
5. You will see that the ML model we soft deleted above is still shown in the UI.

Expected behavior
The above soft deleted ML model should not be shown in the UI.

Screenshots

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[aditya-radhakrishnan]

Hey @jazzblue sorry for the delayed response here. Were you able to verify that the entity was indeed soft deleted in the database?

[aditya-radhakrishnan]

Ah it looks like we don't support the status aspect on MLModels, thanks for the catch. Let me talk to my colleagues and get back to you on what next steps are!

[aditya-radhakrishnan]

Hey @jazzblue so I realized that our docs are misleading. Soft deletion only means that an entity won't show up in Search or Browse, but its entity page will still be viewable. I created this PR to make that clear #7117

That said, supporting soft deletion for ML models is a different matter, and I will get back to you soon on this!

[aditya-radhakrishnan]

It actually looks like we do support soft deletion for ML models via the legacy snapshot aspects, as shown here https://github.com/datahub-project/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/metadata/aspect/MLModelAspect.pdl

Given this, I'm going to close this one out!

[jazzblue]

Hey @jazzblue so I realized that our docs are misleading. Soft deletion only means that an entity won't show up in Search or Browse, but its entity page will still be viewable. I created this PR to make that clear #7117

That said, supporting soft deletion for ML models is a different matter, and I will get back to you soon on this!

@aditya-radhakrishnan I would disagree, soft deleted is deleted and should not show up in the UI. The meaning of soft is that it can be restored, but while it is in soft deleted state it is deleted as far as any user whether API or UI. This should be fixed. I will reopen this issue.