Skip to content
This repository has been archived by the owner on Oct 4, 2024. It is now read-only.

Error on second run #98

Closed
mrzoops opened this issue Jul 30, 2019 · 10 comments
Closed

Error on second run #98

mrzoops opened this issue Jul 30, 2019 · 10 comments

Comments

@mrzoops
Copy link

mrzoops commented Jul 30, 2019

My first run seemed to pull everything. Now on my second run I am not getting my updated photos/videos. Can anyone help me understand what this error is?

Microsoft Windows [Version 10.0.18362.239]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\Plexbox>cd c:\photosync

c:\Photosync>.\env\Scripts\activate

(env) c:\Photosync>gphotos-sync "D:\Google Drive\GPBACKUP"
07-30 08:23:59 Indexing Google Photos Files ...
07-30 08:24:04 Indexing Shared (titled) Albums ...
07-30 08:25:13 Indexed 572 Shared (titled) Albums
07-30 08:25:13 Indexing Albums ...
07-30 08:25:18 Indexed 41 Albums
07-30 08:25:19 Downloading Photos ...
07-30 08:25:20 Null response in mediaItems.batchGet dict_keys(['AEGUry6SSBiKbNke0pzC89cuWdhjD_ktwds9mIJcfujFQP27IJOoU77sD4WRiAKXRxknV009TJMb4GFsp1D7mlNMw3atWv', 'AEGUry5S3CyGkNWyuAy3SvhvX_G9tdBTF28xcZ2KSxOmr4X5cDsFAfguVqYD0ywlIZPq5mQSz3OiqhvdxhYhLz8gBkkv_0_', 'AEGUry4zWZEoLWThybuzBB9ky813LJASvdxlzzXgVTVAMnbLuahLZjReOotrce56diEClV4Zz073-zLZjchuHlUWrjk-YFP', 'AEGUry7dNfL19k9_nxB0Yfzw2JJpUCD6w9BQq_dLTkUcSq9487ItSfgIXCXO4HtcH6Tvy3YMkqIv', 'AEGUry5oHf5M-9lGQQF2ulWsPzkccQuFJlC_tHTm2PwAPpEXOT6Ku_XAkUrpjh8Gk3qydvA9NWR3', 'AEGUry7ShflIJUHx9npk81pnILbr_sqmoDxqP5VlpMkGYzymgxCF2HUjEiTQRaUNoy4xpeeL2OI', 'AEGUry4udlv9GKdQ17pcQRQrliHLANJ7t169-wmHomQOPg7XGUkOXo4NKaUSLeSksnSSouJRJEW2', 'AEGUry4C2qMqZKKzlr3CUL_4aSYIVtqo_v5N99X36f8P7sj11etIxhSyYR3cru_jeB0jbx8qEnDc', 'AEGUry5kfm4Kp7KHZUu2SKPidCkgKq4COEmnV7-PUW9m5octW6r7eoSpNvu0pNidpzGF0G0pOYlLX0c-Y0t5i992BTeINFHsw', 'AEGUry4mhxkHl5TIiH-7L5x8dAS4TYCtIcXyL33gUOLbx5RH-VhLjqj2eD9xJnux_uUqQ4E0Vjs-jsFim6dK4GPJGm_zu5ZSDA'])
07-30 08:25:20 Null response in mediaItems.batchGet dict_keys(['AEGUry6SSBiKbNke0pzC89cuWdhjD_ktwdsp9mIJcfujFQP27IJOoU77sD4WRiAKXRxknV009TJMb4GFsp1D7mlNMw3atWvTWA', 'AEGUry5S3CyGkNWyuAy3SvhvX_G9tdBTF28xcZ2KSxOmr4X5cDsFAfguVqYD0ywlIZPq5mQSz3OiqhvdxhYhLz8gBkkv_0_qqg', 'AEGUry4zWZEoLWThybuzBB9ky813LJASvdxlzzXgVTVAMnbLuahLZjReOotrce56diEClV4Zz073-zLZjchuHlUWrjk-YFPcJQ', 'AEGUry7dNfL19k9_nxB0Yfzw2JJpUCD6w9BQq_dLTkUcSq9487ItSfgIXCXO4HtcH6Tvy3YMkqIv', 'AEGUry5oHf5M-9lGQQF2ulWsPzkccQuFJlC_tHTm2PwAPpEXOT6Ku_XAkUrpjh8Gk3qydvA9NWR3', 'AEGUry7ShflINJUHx9npk81pnILbr_sqmoDxqP5VlpMkGYzymgxCF2HUjEiTQRaUNoy4xpeeL2OI', 'AEGUry4udlv9GKdQ17pcQRQrliHLANJ7t169-wmHomQOPg7XGUkOXo4NKaUSLeSksnSSouJRJEW2', 'AEGUry4C2qMqZKKzlr3CUL_4aSYIVtqo_v5N99X36f8P7sj11etIxhSyYR3cru_jeB0jbx8qEnDc', 'AEGUry5kfm4Kp7KHZUu2SKPidCkgEK4COEmnV7-PUW9m5octW6r7eoSpNvu0pNidpzGF0G0pOYlLX0c-Y0t5i992BTeINFHsw', 'AEGUry4mhxkHl5TIiH-7L5x8dAS4TYCtIcXyL33gUOLbx5RH-VhLjqj2eD9xJnux_uUqQ4E0Vjs-jsFim6dK4GPJGm_zu5ZSDA'])
07-30 08:25:20 Null response in mediaItems.batchGet dict_keys(['AEGUry6SSBiKbNke0pzC89cuWdhjD_ktwdsp9mIJcfujFQP27IJOoU77sD4WRiAKXRxknV009TJMb4GFsp1D7mlNMw3atWvTWA', 'AEGUry5S3CyGkNWyuAy3SvhvX_G9tdBTF28xcZ2KSxOmr4X5cDsFAfguVqYD0ywlIZPq5mQSz3OiqhvdxhYhLz8gBkkv_0_qqg', 'AEGUry4zWZEoLWThybuzBB9ky813LJASvdxlzzXgVTVAMnbLuahLZjReOotrce56diEClV4Zz073-zLZjchuHlUWrjk-YFPcJQ', 'AEGUry7dNfL19k9_nxB0Yfzw2JJpUCD6w9BQq_dLTkUcSq9487ItSfgIXCXO4HtcH6Tvy3YMkqIv', 'AEGUry5oHf5M-9lGQQF2ulWsPzkccQuFJlC_tHTm2PwAPpEXOT6Ku_XAkUrpjh8Gk3qydvA9NWR3', 'AEGUry7ShflINJUHx9npk81pnILbr_sqmoDxqP5VlpMkGYzymgxCF2HUjEiTQRaUNoy4xpeeL2OI', 'AEGUry4udlv9GKdQ17pcQRQrliHLANJ7t169-wmHomQOPg7XGUkOXo4NKaUSLeSksnSSouJRJEW2', 'AEGUry4C2qMqZKKzlr3CUL_4aSYIVtqo_v5N99X36f8P7sj11etIxhSyYR3cru_jeB0jbx8qEnDc', 'AEGUry5kfm4Kp7KHZUu2SKPidCkgKq4COEmnV7-PUW9m5octW6r7eoSpNvu0pNidpzGF0G0pOYlLX0c-Y0t5i992BTeINFHsw', 'AEGUry4mhxkHl5TIiH-7L5x8dAS4TYCtIcXyL33gUOLbx5RH-VhLjqj2eD9xJnux_uUqQ4E0Vjs-jsFim6dK4GPJGm_zu5ZSDA'])
07-30 08:25:20 Null response in mediaItems.batchGet dict_keys(['AEGUry6SSBiKbNke0pzC89cuWdhjD_ktwdsp9mIJcfujFQP27IJOoU77sD4WRiAKXRxknV009TJMb4GFsp1D7mlNMw3atWvTWA', 'AEGUry5S3CyGkNWyuAy3SvhvX_G9tdBTF28xcZ2KSxOmr4X5cDsFAfguVqYD0ywlIZPq5mQSz3OiqhvdxhYhLz8gBkkv_0_qqg', 'AEGUry4zWZEoLWThybuzBB9ky813LJASvdxlzzXgVTVAMnbLuahLZjReOotrce56diEClV4Zz073-zLZjchuHlUWrjk-YFPcJQ', 'AEGUry7dNfL19k9_nxB0Yfzw2JJpUCD6w9BQq_dLTkUcSq9487ItSfgIXCXO4HtcH6Tvy3YMkqIv', 'AEGUry5oHf5M-9lGQQF2ulWsPzkccQuFJlC_tHTm2PwAPpEXOT6Ku_XAkUrpjh8Gk3qydvA9NWR3', 'AEGUry7ShflINJUHx9npk81pnILbr_sqmoDxqP5VlpMkGYzymgxCF2HUjEiTQRaUNoy4xpeeL2OI', 'AEGUry4udlv9GKdQ17pcQRQrliHLANJ7t169-wmHomQOPg7XGUkOXo4NKUSLeSksnSSouJRJEW2', 'AEGUry4C2qMqZKKzlr3CUL_4aSYIVtqo_v5N99X36f8P7sj11etIxhSyYR3cru_jeB0jbx8qEnDc', 'AEGUry5kfm4Kp7KHZUu2SKPidCkgEKq4COEmnV7-PUW9m5octW6r7eoSpNvu0pNidpzGF0G0pOYlLX0c-Y0t5i992BTeINFHsw', 'AEGUry4mhxkHl5TIiH-7L5x8dAS4TYCtIcXyL33gUOLbx5RH-VhLjqj2eD9xJnux_uUqQ4E0Vjs-jsFim6dK4GPJGm_zu5ZSDA'])
07-30 08:25:20 Null response in mediaItems.batchGet dict_keys(['AEGUry6SSBiKbNke0pzC89cuWdhjD_ktwdsp9mIJcfujFQP27IJOoU77sD4WRiAKXRxknV009TJMb4GFsp1D7mlNMw3atWvTWA', 'AEGUry5S3CyGkNWyuAy3SvhvX_G9tdBTF28xcZ2KSxOmr4X5cDsFAfguVqYD0ywlIZPq5mQSz3OiqhvdxhYhLz8gBkkv_0_qqg', 'AEGUry4zWZEoLWThybuzBB9ky813LJASvdxlzzXgVTVAMnbLuahLZjReOotrce56diEClV4Zz073-zLZjchuHlUWrjk-YFPcJQ', 'AEGUry7dNfL19k9_nxB0Yfzw2JJpUCD6w9BQq_dLTkUcSq9487ItSfgIXCXO4HtcH6Tvy3YMkqIv', 'AEGUry5oHf5M-9lGQQF2ulWsPzkccQuFJlC_tHTm2PwAPpEXOT6Ku_XAkUrpjh8Gk3qydvA9NWR3', 'AEGUry7ShflINJUHx9npk81pnILbr_sqmoDxqP5VlpMkGYzymgxCF2HUjEiTQRaUNoy4xpeeL2OI', 'AEGUry4udlv9GKdQ17pcQRQrliHANJ7t169-wmHomQOPg7XGUkOXo4NKaUSLeSksnSSouJRJEW2', 'AEGUry4C2qMqZKKzlr3CUL_4aSYIVtqo_v5N99X36f8P7sj11etIxhSyYR3cru_jeB0jbx8qEnDc', 'AEGUry5kfm4Kp7KHZUu2SKPidCkgEKq4COEmnV7-PUW9m5octW6r7eoSpNvu0pNidpzGF0G0pOYlLX0c-Y0t5i992BTeINFHsw', 'AEGUry4mhxkHl5TIiH-7L5x8dAS4TYCtIcXyL33gUOLbx5RH-VhLjqj2eD9xJnux_uUqQ4E0Vjs-jsFim6dK4GPJGm_zu5ZSDA'])
07-30 08:25:26 FAILURE 1 downloading photos\2017\10\VID_20170930_155255.mp4
07-30 08:25:26 FAILURE 2 downloading photos\2017\10\VID_20170930_155255.mp4
07-30 08:25:26
Process failed.
Traceback (most recent call last):
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 214, in do_download_file
response.raise_for_status()
File "c:\photosync\env\lib\site-packages\requests\models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://video-downloads.googleusercontent.com/ABzBgw0U5egbpG1sJgqQK-X0hOokG1DfYPMfW-SaB3hVAen6nALUchGUUi1wuRFM2pTzJF9vNbgOv0ooiDXutqWVnLT7IoH27aSs4NjFTQpNBJKbR3Nu0ytG8wgThoFeDHKb50csJVk7c_AcEGOPpPNmtRkbmIng9dCOjSRxsAi-nDSvVGOhnVkzvUjt73DtU-KLr9uyg6wrOFaCnXhLI1waadQUpCkHULQP3722rEtY9WG1eV8hMjgkgV5k3yHG4JSDE2UgLyFl6NXseUlnYV3EumzviVHiMKHT9gA0AubK_u6EBnMpfMfdqFbu2mtlV5ZtsBwzaSz86kHDMOnhxbchp1945ckpn26S7wanZL5RSLiDRpqgFCYr5NARgfjUfp4c7hudn23u_AWIACPZu4DBsg5A6Tt6iif7anFgoMG27M_D4U1iRsm0sEwGx3YFK5uSyq8xC2ELFOnQGt4OcEw6iIUqUTjfC5yrXOte4BfIsBt8jtHd7T2TydSqgj_ePwbvXs_YpVb2Io7v5QrR1n_sSLZAeOt-6RVdxVKgpwVmZFwB6c8aga-mX7bxH_jhgcxCgd1JXBTuda7S0nq-lE2tkCQl6r6gNIKmcUgbNIz-mG1A-Q1UulDymm3foPqnsD3Jr5F1-bjLzV89BPoeEgstXutEyeajzNbibmipIzDE-XOhIF_NkC-i9k5yv8CXpmT3wftB7P14aSEKSJQPM0BzLtrqg_DsYqmEy0cVCQxNOErn16992pqX-h434_gBSYBVX06oniKbPsKtdRWLhOBy_PEgarkVTijKeUsWe3nEVtbg_y-QIpQnU9hpPUJXMcmf-pkzdYkDhQXwMSaFRVYZINw8ag88ZH1XRs0hLjb_Ut3ujppl9IRZJaR1FsTAet8auhIoBClSBILXQ0r1rUxmgDvmrqI7w

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\photosync\env\lib\site-packages\gphotos\Main.py", line 329, in main
self.start(args)
File "c:\photosync\env\lib\site-packages\gphotos\Main.py", line 301, in start
self.do_sync(args)
File "c:\photosync\env\lib\site-packages\gphotos\Main.py", line 285, in do_sync
self.google_photos_down.download_photo_media()
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 126, in download_photo_media
self.do_download_complete(futures_left)
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 250, in do_download_complete
raise e
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 122, in download_photo_media
self.download_batch(batch)
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 156, in download_batch
self.download_file(media_item, media_item_json)
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 187, in download_file
self.do_download_complete(done_list)
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 250, in do_download_complete
raise e
File "C:\Users\Plexbox\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "c:\photosync\env\lib\site-packages\gphotos\GooglePhotosDownload.py", line 227, in do_download_file
t_path.unlink()
File "C:\Users\Plexbox\AppData\Local\Programs\Python\Python37-32\lib\pathlib.py", line 1294, in unlink
self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'D:\Google Drive\GPBACKUP\photos\2017\10\tmp40fhuhdt'
07-30 08:25:26 Done.

(env) c:\Photosync>
@gilesknap
Copy link
Owner

gilesknap commented Jul 30, 2019 via email

gilesknap added a commit that referenced this issue Jul 30, 2019
@gilesknap
#98 close temp file before unlink when download fails
8eab21d
@gilesknap
Copy link
Owner

I have found the issue.

You are getting a failure to download some items as per https://issuetracker.google.com/issues/116842164. It looks like this is happening in an album that is shared to you.

I handle this by adding these files to a list in gphotos.bad_ids.yaml and then don't try to download them again.

However, I was failing to close the temporary download file before deleting it when this happens. I get away with that on Linux but windows does not allow it.

I have implemented a fix. Please can you try the feature branch here https://github.com/gilesknap/gphotos-sync/tree/feature.

Cheers, giles.

@gilesknap
Copy link
Owner

Todo: add a regression test for this issue.

@mrzoops
Copy link
Author

mrzoops commented Jul 31, 2019

Awesome. I will test. Whats the best way to update when I am not using pipenv? I am just using a virtual env?

@gilesknap
Copy link
Owner

gilesknap commented Jul 31, 2019 via email

@gilesknap
Copy link
Owner

gilesknap commented Jul 31, 2019 via email •
edited
Loading

@gilesknap
Copy link
Owner

By the way. Another possible interpretation of your log is that you lost your network connection during the run. Did you try a few times and get the same error?

@gilesknap
Copy link
Owner

gilesknap commented Jul 31, 2019 •
edited
Loading

There is an additional issue here. Sometimes Google API fails intermittently then all attempted downloads will get put into the bad_ids file and never be downloaded. The easy workaround is to delete the bad_ids file but it would be better to somehow distinguish between a 'permanent' issue with a particular file and an intermittent one.

I just had an example of the most recent Travis test that failed with the same errors that mrzoops got and then succeeded the second time I ran it with no changes.

07-31 06:26:32 Null response in mediaItems.batchGet dict_keys(['ABfvn ...
07-31 06:26:32 gphotos.GooglePhotosDownload WARNING  Null response in mediaItems.batchGet dict_keys(['ABfv ...
07-31 06:26:32 Null response in mediaItems.batchGet dict_keys(['ABfv ...
07-31 06:26:32 gphotos.GooglePhotosDownload WARNING  Null response in mediaItems.batchGet dict_keys(['ABfv ...

...

I think in this instance it is possible to distinguish the NULL response from the 500 error that we normally get on permanent mediaItems.batchGet issues. But when the file download error was a 500 and I guess that could be intermittent or permanent.

Opening a new issue on this.

@gilesknap gilesknap mentioned this issue Jul 31, 2019
Closed
@gilesknap
Copy link
Owner

The file left open fix has now been released to pypi at 2.9.2

@mrzoops
Copy link
Author

mrzoops commented Jul 31, 2019

OK I will review all of this and be testing very shortly.

@gilesknap gilesknap mentioned this issue Jul 31, 2019
Closed
@gilesknap gilesknap mentioned this issue Aug 8, 2019
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gilesknap @mrzoops