set mtime to video publish date

[codl]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.12.03. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2018.12.03

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

Description of your issue, suggested solution and other information

this is a followup to #18383 in which I found out that the mtime is only set according to the Last-Modified HTTP headers, and doesn't work for videos downloaded with DASH. I find this counter-intuitive: having downloaded videos and seen that the mtime was set to — as far as I could tell — the upload date, I expected this to happen all the time until it randomly didn't work for some videos.

I think that, whenever the upload time is available, it should be used for the final file's mtime in place of Last-Modified.

[dotysan]

I just tested and it appears that all DASH segments indeed had the same/proper mtime when fetched via http.py. However when the segments are stitched together, the mtime is lost.

[dotysan]

Not a pull request yet, because I'm new to the code in this project and haven't run the unit tests. But this seems to do the job!

diff --git a/youtube_dl/downloader/fragment.py b/youtube_dl/downloader/fragment.py
index 917f6dc01..37edcc900 100644
--- a/youtube_dl/downloader/fragment.py
+++ b/youtube_dl/downloader/fragment.py
@@ -116,6 +116,10 @@ class FragmentFD(FileDownloader):
         finally:
             if self.__do_ytdl_file(ctx):
                 self._write_ytdl_file(ctx)
+            try:
+                os.utime(ctx['tmpfilename'], (time.time(), os.stat(ctx['fragment_filename_sanitized']).st_mtime))
+            except Exception:
+                pass
             if not self.params.get('keep_fragments', False):
                 os.remove(encodeFilename(ctx['fragment_filename_sanitized']))
             del ctx['fragment_filename_sanitized']

In summary, it leverages the fact that every fragment/segment already has an mtime set from the last-modified header. So it just re-sets the mtime of the working/temp file from the fragment file. Which in turn, seems to be preserved in the finalized file of all fragments.

Feedback?

[reallyuniquename]

Has this been addressed? I'm running latest version and it seems mtime has no effect on fragmented Twitch VODs.

[dotysan]

Has this been addressed? I'm running latest version and it seems mtime has no effect on fragmented Twitch VODs.

PR #27138 addresses this via trivial patch proposed above last year.

[codl]

By the way, fragmented videos are not the only ones to have inconsistent mtimes. I have encountered youtube videos that had Last-Modified/mtime many years off from their upload date (I assume because they have been re-encoded since)

As an example:

$ youtube-dl --ignore-config "https://www.youtube.com/watch?v=U7eLBwCAwmo"
[youtube] U7eLBwCAwmo: Downloading webpage
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: You Cannot Remove Your Fingerprints With Pineapple-U7eLBwCAwmo.f135.mp4
[download] 100% of 13.97MiB in 00:06
[download] Destination: You Cannot Remove Your Fingerprints With Pineapple-U7eLBwCAwmo.f251.webm
[download] 100% of 5.24MiB in 00:02
[ffmpeg] Merging formats into "You Cannot Remove Your Fingerprints With Pineapple-U7eLBwCAwmo.mkv"
Deleting original file You Cannot Remove Your Fingerprints With Pineapple-U7eLBwCAwmo.f135.mp4 (pass -k to keep)
Deleting original file You Cannot Remove Your Fingerprints With Pineapple-U7eLBwCAwmo.f251.webm (pass -k to keep)
$ youtube-dl -j "https://www.youtube.com/watch?v=U7eLBwCAwmo" | jq '.upload_date'
"20090315"
$ stat You\ Cannot\ Remove\ Your\ Fingerprints\ With\ Pineapple-U7eLBwCAwmo.mkv | grep Modify
Modify: 2018-11-18 20:02:28.000000000 +0100

[dotysan]

@codl so maybe the .upload_date is the first date the upload was attempted. But the mtime of each fragment is the last time that fragment was uploaded.

This seems to be what I've observed. Therefore setting the download mtime to the last fragment is most logical.

I agree the json/metadata is missing info such as re-encoding...