[3/4] Properly cleanup unclosed files within generator function

[ejguan]

There is a case that file.close is never called because when generator function has never reached to the end. A simple example would be zip two datepipes with different length. The longer DataPipe would never reach the end of generator and then it will be cleaned up by gc. So, the line of file.close is not executed. (This is the reason that Vitaly has to create this hack to retrieve all remaining data to make sure generator function is fully executed)

However, this hack introduces another problem where an infinite datapipe would make zip never end as it would try to deplete the infinite iterator. See: #865

So, in this PR, I am adding a try-finally clause to make sure the file.close is always executed during the destruction of generator object. Then, we don't need the hack within zip any more.

• [1/4][DataPipe] Properly cleanup unclosed files within generator function pytorch#89973
• [DataPipe] Properly cleanup unclosed files within generator function vision#6997
• [4/4][DataPipe] Remove iterator depletion in Zipper pytorch#89974

[facebook-github-bot]

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[NivekT]

Just one question on this, but overall, this LGTM!

Will look at the rest soon

[wenleix]

Thanks!

[facebook-github-bot]

@ejguan has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.