Change Container related tables purging to use archiving and destroy_all on archived records as they age out

[jrafanie]

Container related tables

These tables aren't being purged with the purger for the principal container* models.

• container_conditions
• container_env_vars
• container_volumes
• security_contexts
• container_port_configs
• custom_attributes (also used by vms)

Even with these configured in my db:

:keep_archived_entities: 2.months
:keep_archived_quotas: 2.months

and running:

Container.purge_timer
ContainerNode.purge_timer
ContainerGroup.purge_timer
ContainerImage.purge_timer
ContainerProject.purge_timer
ContainerQuota.purge_timer
ContainerQuotaItem.purge_timer

I'm still seeing several tables with orphaned records:

Container: 49240 deleted, remaining: 555
ContainerNode: 6 deleted, remaining: 6
ContainerGroup: 47687 deleted, remaining: 351
ContainerImage: 3895 deleted, remaining: 386
ContainerProject: 74 deleted, remaining: 75
*ContainerEnvVar: 0 deleted, remaining: 132107
*ContainerCondition: 0 deleted, remaining: 193093
ContainerQuota: 1 deleted, remaining: 1
ContainerQuotaItem: 5 deleted, remaining: 5
*ContainerVolume: 0 deleted, remaining: 156509
*SecurityContext: 0 deleted, remaining: 48517
*ContainerPortConfig: 0 deleted, remaining: 38715
*ContainerService: 0 deleted, remaining: 150

We don't implement purge_associated_records in many of these models so the association tables are left orphaned. I think we can add CustomAttribute to that list also.

[jrafanie]

@Fryguy suggested:

"we remove purging from container_group, container_image, container_node, container_project, container, keep the schedule, but in the schedule it just does a destroy_all - probably don't need batching at all as long as each record is committed"

Basically, we can ensure archiving works for the container* primary tables and then make sure the archived rows are removed and when removed, they should use destroy_all so the associated tables are cascaded, leaving no orphaned associations in other tables. Note, we'll need to make sure the destroy_all happens in a small enough batch size so the work item doesn't timeout trying to remove the data associated with the record.

[jrafanie]

I found the following workaround I could run in bin/rails console in dev or on appliances to clean up some of the container related tables with orphaned rows after the purger removed the higher level container entities but not the associated tables.

ContainerCondition.include PurgingMixin
ContainerVolume.include PurgingMixin
CustomAttribute.include PurgingMixin
SecurityContext.include PurgingMixin

ContainerCondition.purge_by_orphaned(:container_entity, 1000)
ContainerVolume.purge_by_orphaned(:parent, 1000)
CustomAttribute.purge_by_orphaned(:resource, 1000)
SecurityContext.purge_by_orphaned(:resource, 1000)

The remaining possibly big ones are non-polymorphic so would need a tweaked version of purge_by_orphaned.

• ContainerEnvVar (non-polymorphic, uses container)
• ContainerPortConfig (non-polymorphic, uses container)