Unsealing enhancements

[rjan90]

Context: This issue originally started out as Investigaton of storing tree_r_last in Lotus-Miner to see how we could make unsealing better, which lead us to a longer path of unsealing enhancements after discussions with the proofs-team. This issue ticket has therefore been repurposed reflect that epic.

User story:
As a storage provider (SP) who stores sealed sectors, I need an efficient and reliable unsealing process which does not require manual work to attach workers to where the sealed sector is located, allowing me to focus on handling large volume of sealed sectors being unsealed for clients.

Acceptence criteria:

• Unsealing of every sector type (Deals, SnapDeal) functions without bugs.
• The unsealing process should not require manual intervention for unsealing to work on unsealing-workers; such as attaching the location where the sealed sector is stored to a worker.

Techincal scope

Tasks

Preview Give feedback

1. Investigate storing tree_r_last in Lotus-Miner
2. fix: unseal: check if sealed/update sector exists #10639
   +0
3. feat: api: Add API and CLI to unseal sector #10626
   +0
4. feat: piecereader: Allow parallel access #10913
   +0
5. Method to generate sector key rust-fil-proofs#1666
   +0
6. Don't create the sealed file in precommit1 rust-fil-proofs#1665
   +0
7. Additional data encode/decode methods rust-fil-proofs#1667
   +0
8. Implement new SnapDeals decodes API for unsealing #10994
   3 of 3
   +0
   
9. “Split PC1” for unsealing.

Point estimation:

[rjan90]

Closing this issue now that #11011 has been merged, and is on its way to users with the Lotus v1.23.4 release train.

The linked PR introduces significant improvements and bugfixes with regards to unsealing, and ensures that unsealing operates as anticipated. Consequently, the unsealing of deal sectors, Snap-sectors without sector keys, and Snap-sectors with sector keys now all function seamlessly.

Additional improvements/bugfixes with this PR includes:

• Enabling unsealing on workers with only sealing paths. 🎉
• Transferring unsealed files to long-term storage upon successful unsealing. ➡️
• Ensuring no residual files in sealing paths post a successful unsealing operation. 🧹

One area to explore further is if want to spend additional time splitting the "PC1-step" (TreeD building and SDR) in the unsealing pipeline. See the More efficient deal data pipeline section in https://pl-strflt.notion.site/Improved-APIs-for-Lotus-9bd4fa443e254701a081f51ad301fddf.