Fix AsyncCheckpointIO race condition

[awonak]

Provide a mechanism for calling Checkpoint teardown when using manual checkpoint saves with AsyncCheckpointIO.

When using a large layer count, the save_checkpoint() task was not completing prior to calling load_checkpoint(), resulting in a race condition where load_checkpoint() would return a 404. This was caused by the removal of the teardown step, which issues a blocking shutdown call to the Threadpool executor, waiting for all tasks to complete before returning. We still want to override teardown so that it is not called during trainer.fit(), but we want to make sure the Threadpool tasks complete before making any calls to load_checkpoint().

This PR introduces a new invocation point for the teardown method, which can be called after all save_checkpoint()
calls and before any load_checkpoint() calls.

• Tests pass
• Appropriate changes to documentation are included in the PR