This repository has been archived by the owner on Oct 4, 2024. It is now read-only.
Failure downloading , already downloaded #221
Assignees
Labels
Comments
|
Hi, Would you be OK checking out a branch and trying that? There is a branch called 429-reponse in which I was working on an issue where downloads would fail when a URL quota was hit. I do not think you are hitting quota using an RPi but the fix included extra error reporting in the download threads. I think this would give us more info on the root of your issue. |
|
I cloned gphotos-sync and checked out 429-reponse. Here are the logs: |
|
This looks like a different issue to what you had before.
It looks like your file system has declared that it supports links but the
account does not have permission to use them
If you can link files from the command line with
ln -s sourcefile destlink
Then it should work.
If your filesystem is Windows then you need to enable links.
…On Fri, 17 Apr 2020, 15:47 Jordan R Thompson, ***@***.***> wrote:
I cloned gphotos-sync and checked out 429-reponse.
I ran it and got this:
04-17 09:55:02 WARNING gphotos-sync 2.14.0+4.37fad11 2020-04-17 09:54:59.309912
04-17 09:55:03 WARNING Indexing Google Photos Files ...
04-17 09:55:05 WARNING indexed 6 items
04-17 09:55:05 WARNING Downloading Photos ...
04-17 10:10:44 WARNING Downloaded 6 Items, Failed 0, Already Downloaded 49597
4-17 10:10:44 WARNING Indexing Shared (titled) Albums ...
04-17 10:15:41 WARNING Indexed 184 Shared (titled) Albums ..
4-17 10:15:41 WARNING Indexing Albums ...
04-17 10:18:41 WARNING Indexed 96 Albums ..
04-17 10:18:41 WARNING Downloading Photos ...
04-17 10:33:45 WARNING Downloaded 6 Items, Failed 0, Already Downloaded 99200
04-17 10:33:45 WARNING Creating album folder links to media ...
04-17 10:35:04 ERROR
Process failed.
Traceback (most recent call last):
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 418, in main
self.start(args)
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 369, in start
self.do_sync(args)
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 358, in do_sync
self.google_albums_sync.create_album_content_links()
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 322, in create_album_content_links
follow_symlinks=False,
PermissionError: [Errno 1] Operation not permitted
04-17 10:35:04 WARNING Done.
Here are the logs:
04-17 09:55:02 gphotos.Main WARNING gphotos-sync 2.14.0+4.37fad11 2020-04-17 09:54:59.309912
04-17 09:55:02 gphotos.Utils DEBUG MINIMUM_DATE = 1970-01-01 00:00:00
04-17 09:55:02 gphotos.Checks INFO Target filesystem /mnt/googlephotos/mojgan is cifs
04-17 09:55:02 gphotos.Checks DEBUG Checking if is filesystem supports symbolic links...
04-17 09:55:02 gphotos.Checks DEBUG attempting to symlink /mnt/googlePhotos/Mojgan/test_src_442191337 to /mnt/googlePhotos/Mojgan/test_dst_2193620574
04-17 09:55:02 gphotos.Checks DEBUG Checking if File system supports unicode filenames...
04-17 09:55:03 gphotos.Checks INFO Filesystem supports Unicode filenames
04-17 09:55:03 gphotos.Checks DEBUG Checking if File system is case insensitive...
04-17 09:55:03 gphotos.Checks INFO Case insensitive file system found
04-17 09:55:03 gphotos.Checks INFO Max Path Length: 4096
04-17 09:55:03 gphotos.Checks INFO Max filename length: 255
04-17 09:55:03 gphotos.Main INFO version: 2.14.0+4.37fad11, database schema version 5.7
04-17 09:55:03 gphotos.GooglePhotosIndex WARNING Indexing Google Photos Files ...
04-17 09:55:03 gphotos.GooglePhotosIndex INFO searching for media start=2020-04-16 17:21:47, end=None, videos=True
04-17 09:55:03 gphotos.GooglePhotosIndex DEBUG mediaItems.search with body:
{'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2020, 'month': 4, 'day': 16}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}}
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 1 photos/2020/04/mvimg_20200417_094657.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 2 photos/2020/04/mvimg_20200417_094610.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 3 photos/2020/04/mvimg_20200417_094427.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 4 photos/2020/04/mvimg_20200417_094328.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 5 photos/2020/04/mvimg_20200417_094147.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 6 photos/2020/04/mvimg_20200417_093948.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG Skipped Index (already indexed) 1 photos/2020/04/mvimg_20200416_132147.jpg
04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG search_media parsed 7 media_items with 100 PAGE_SIZE
04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG mediaItems.search with body:
{'pageToken': 'CmwKRHR5cGUuZ29vZ2xlYXBpcy5jb20vZ29vZ2xlLnBob3Rvcy5saWJyYXJ5LnYxLlNlYXJjaE1lZGlhSXRlbXNSZXF1ZXN0EiQQZCIgChQSEgoHCOQPEAQYEBIHCLgXEAEYARoDCgEAOgMKAQASjAFBSF91UTQydERYbmJXM01KNTNVZmdNaVh1aGFlcmRvaWNPYkJoUm9Vbkl3ZVVVZWViSHNBNDBkRGxVaHhELVJNUGJvRW9aQWY4NlF3bWNENzV3WlNpVjBNVS1LdzV6bmc3cnJQQ3p6QzRvNnZYaHo2Z1N6bjZpamJrR200V2JJQVJUV1N5U3MzM08zVxobMVkwQ0RkMnRIcXNSNUlOak1BdWRSXy1HcWFV', 'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate': {'year': 2020, 'month': 4, 'day': 16}, 'endDate': {'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia': False}}
04-17 09:55:05 gphotos.GooglePhotosIndex WARNING indexed 6 items
04-17 09:55:05 gphotos.GooglePhotosDownload WARNING Downloading Photos ...
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 1 photos/2020/04/mvimg_20200415_180725.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 2 photos/2020/04/mvimg_20200415_180719.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 3 photos/2020/04/mvimg_20200415_115554.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 4 photos/2020/04/mvimg_20200415_002211.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 5 photos/2020/04/mvimg_20200415_002208.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 6 photos/2020/04/screenshot_20200414-153029.png
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 7 photos/2020/04/mvimg_20200414_145912.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 8 photos/2020/04/mvimg_20200414_125147.jpg
04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 9 photos/2020/04/mvimg_20200414_124717.jpg
..
... more ...
..
04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99189 photos/2019/07/MVIMG_20190726_174745.jpg
04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99190 photos/2019/07/MVIMG_20190726_174749.jpg
04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99191 photos/2019/07/MVIMG_20190726_174757.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99192 photos/2019/07/MVIMG_20190726_174804.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99193 photos/2019/07/MVIMG_20190726_174946.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99194 photos/2019/07/MVIMG_20190726_175040.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99195 photos/2020/04/mvimg_20200417_094657.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99196 photos/2020/04/mvimg_20200417_094610.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99197 photos/2020/04/mvimg_20200417_094427.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99198 photos/2020/04/mvimg_20200417_094328.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99199 photos/2020/04/mvimg_20200417_094147.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file exists) 99200 photos/2020/04/mvimg_20200417_093948.jpg
04-17 10:33:45 gphotos.GooglePhotosDownload WARNING Downloaded 6 Items, Failed 0, Already Downloaded 99200
04-17 10:33:45 gphotos.GoogleAlbumsSync WARNING Creating album folder links to media ...
04-17 10:33:45 gphotos.GoogleAlbumsSync DEBUG removing previous album links tree
04-17 10:35:03 gphotos.GoogleAlbumsSync DEBUG adding album link ../../../photos/2017/02/IMG_20170131_194132.jpg -> /mnt/googlePhotos/Mojgan/albums/2017/0205 Trip to Havana/0000_IMG_20170131_194132.jpg
04-17 10:35:03 gphotos.GoogleAlbumsSync DEBUG new album folder /mnt/googlePhotos/Mojgan/albums/2017/0205 Trip to Havana
04-17 10:35:03 gphotos.LocalData INFO Saving Database ...
04-17 10:35:04 gphotos.LocalData INFO Database Saved.
04-17 10:35:04 gphotos.Main ERROR
Process failed.
Traceback (most recent call last):
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 418, in main
self.start(args)
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 369, in start
self.do_sync(args)
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 358, in do_sync
self.google_albums_sync.create_album_content_links()
File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/GoogleAlbumsSync.py", line 322, in create_album_content_links
follow_symlinks=False,
PermissionError: [Errno 1] Operation not permitted
04-17 10:35:04 gphotos.Main WARNING Done.
04-17 10:35:04 gphotos.Main INFO Elapsed time = 0:40:05.521218
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHLRW45LTYVKEQV7OXHDRDRNBTWZANCNFSM4MKMTGXQ>
.
|
|
The problem is that it runs for a while (downloading files) and then quits.
The Pi is running Raspbian (Debian 4.19.108-v7) and the NAS is
running Debian 8.3.0-6. Nothing is changing on either end.
…On Fri, Apr 17, 2020 at 1:07 PM Giles Knap ***@***.***> wrote:
This looks like a different issue to what you had before.
It looks like your file system has declared that it supports links but the
account does not have permission to use them
If you can link files from the command line with
ln -s sourcefile destlink
Then it should work.
If your filesystem is Windows then you need to enable links.
On Fri, 17 Apr 2020, 15:47 Jordan R Thompson, ***@***.***>
wrote:
> I cloned gphotos-sync and checked out 429-reponse.
> I ran it and got this:
>
> 04-17 09:55:02 WARNING gphotos-sync 2.14.0+4.37fad11 2020-04-17
09:54:59.309912
> 04-17 09:55:03 WARNING Indexing Google Photos Files ...
> 04-17 09:55:05 WARNING indexed 6 items
> 04-17 09:55:05 WARNING Downloading Photos ...
> 04-17 10:10:44 WARNING Downloaded 6 Items, Failed 0, Already Downloaded
49597
> 4-17 10:10:44 WARNING Indexing Shared (titled) Albums ...
> 04-17 10:15:41 WARNING Indexed 184 Shared (titled) Albums ..
> 4-17 10:15:41 WARNING Indexing Albums ...
> 04-17 10:18:41 WARNING Indexed 96 Albums ..
> 04-17 10:18:41 WARNING Downloading Photos ...
> 04-17 10:33:45 WARNING Downloaded 6 Items, Failed 0, Already Downloaded
99200
> 04-17 10:33:45 WARNING Creating album folder links to media ...
> 04-17 10:35:04 ERROR
> Process failed.
> Traceback (most recent call last):
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 418,
in main
> self.start(args)
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 369,
in start
> self.do_sync(args)
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 358,
in do_sync
> self.google_albums_sync.create_album_content_links()
> File
"/home/jordan/gphotos-sync/gphotos-sync/gphotos/GoogleAlbumsSync.py", line
322, in create_album_content_links
> follow_symlinks=False,
> PermissionError: [Errno 1] Operation not permitted
> 04-17 10:35:04 WARNING Done.
>
> Here are the logs:
>
> 04-17 09:55:02 gphotos.Main WARNING gphotos-sync 2.14.0+4.37fad11
2020-04-17 09:54:59.309912
> 04-17 09:55:02 gphotos.Utils DEBUG MINIMUM_DATE = 1970-01-01 00:00:00
> 04-17 09:55:02 gphotos.Checks INFO Target filesystem
/mnt/googlephotos/mojgan is cifs
> 04-17 09:55:02 gphotos.Checks DEBUG Checking if is filesystem supports
symbolic links...
> 04-17 09:55:02 gphotos.Checks DEBUG attempting to symlink
/mnt/googlePhotos/Mojgan/test_src_442191337 to
/mnt/googlePhotos/Mojgan/test_dst_2193620574
> 04-17 09:55:02 gphotos.Checks DEBUG Checking if File system supports
unicode filenames...
> 04-17 09:55:03 gphotos.Checks INFO Filesystem supports Unicode filenames
> 04-17 09:55:03 gphotos.Checks DEBUG Checking if File system is case
insensitive...
> 04-17 09:55:03 gphotos.Checks INFO Case insensitive file system found
> 04-17 09:55:03 gphotos.Checks INFO Max Path Length: 4096
> 04-17 09:55:03 gphotos.Checks INFO Max filename length: 255
> 04-17 09:55:03 gphotos.Main INFO version: 2.14.0+4.37fad11, database
schema version 5.7
> 04-17 09:55:03 gphotos.GooglePhotosIndex WARNING Indexing Google Photos
Files ...
> 04-17 09:55:03 gphotos.GooglePhotosIndex INFO searching for media
start=2020-04-16 17:21:47, end=None, videos=True
> 04-17 09:55:03 gphotos.GooglePhotosIndex DEBUG mediaItems.search with
body:
> {'pageToken': None, 'pageSize': 100, 'filters': {'dateFilter':
{'ranges': [{'startDate': {'year': 2020, 'month': 4, 'day': 16}, 'endDate':
{'year': 3000, 'month': 1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes':
['ALL_MEDIA']}, 'featureFilter': {'includedFeatures': ['NONE']},
'includeArchivedMedia': False}}
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 1
photos/2020/04/mvimg_20200417_094657.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 2
photos/2020/04/mvimg_20200417_094610.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 3
photos/2020/04/mvimg_20200417_094427.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 4
photos/2020/04/mvimg_20200417_094328.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 5
photos/2020/04/mvimg_20200417_094147.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex INFO Indexed 6
photos/2020/04/mvimg_20200417_093948.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG Skipped Index (already
indexed) 1 photos/2020/04/mvimg_20200416_132147.jpg
> 04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG search_media parsed 7
media_items with 100 PAGE_SIZE
> 04-17 09:55:04 gphotos.GooglePhotosIndex DEBUG mediaItems.search with
body:
> {'pageToken':
'CmwKRHR5cGUuZ29vZ2xlYXBpcy5jb20vZ29vZ2xlLnBob3Rvcy5saWJyYXJ5LnYxLlNlYXJjaE1lZGlhSXRlbXNSZXF1ZXN0EiQQZCIgChQSEgoHCOQPEAQYEBIHCLgXEAEYARoDCgEAOgMKAQASjAFBSF91UTQydERYbmJXM01KNTNVZmdNaVh1aGFlcmRvaWNPYkJoUm9Vbkl3ZVVVZWViSHNBNDBkRGxVaHhELVJNUGJvRW9aQWY4NlF3bWNENzV3WlNpVjBNVS1LdzV6bmc3cnJQQ3p6QzRvNnZYaHo2Z1N6bjZpamJrR200V2JJQVJUV1N5U3MzM08zVxobMVkwQ0RkMnRIcXNSNUlOak1BdWRSXy1HcWFV',
'pageSize': 100, 'filters': {'dateFilter': {'ranges': [{'startDate':
{'year': 2020, 'month': 4, 'day': 16}, 'endDate': {'year': 3000, 'month':
1, 'day': 1}}]}, 'mediaTypeFilter': {'mediaTypes': ['ALL_MEDIA']},
'featureFilter': {'includedFeatures': ['NONE']}, 'includeArchivedMedia':
False}}
> 04-17 09:55:05 gphotos.GooglePhotosIndex WARNING indexed 6 items
> 04-17 09:55:05 gphotos.GooglePhotosDownload WARNING Downloading Photos
...
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 1 photos/2020/04/mvimg_20200415_180725.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 2 photos/2020/04/mvimg_20200415_180719.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 3 photos/2020/04/mvimg_20200415_115554.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 4 photos/2020/04/mvimg_20200415_002211.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 5 photos/2020/04/mvimg_20200415_002208.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 6 photos/2020/04/screenshot_20200414-153029.png
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 7 photos/2020/04/mvimg_20200414_145912.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 8 photos/2020/04/mvimg_20200414_125147.jpg
> 04-17 09:55:06 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 9 photos/2020/04/mvimg_20200414_124717.jpg
> ..
> ... more ...
> ..
> 04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99189 photos/2019/07/MVIMG_20190726_174745.jpg
> 04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99190 photos/2019/07/MVIMG_20190726_174749.jpg
> 04-17 10:33:44 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99191 photos/2019/07/MVIMG_20190726_174757.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99192 photos/2019/07/MVIMG_20190726_174804.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99193 photos/2019/07/MVIMG_20190726_174946.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99194 photos/2019/07/MVIMG_20190726_175040.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99195 photos/2020/04/mvimg_20200417_094657.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99196 photos/2020/04/mvimg_20200417_094610.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99197 photos/2020/04/mvimg_20200417_094427.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99198 photos/2020/04/mvimg_20200417_094328.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99199 photos/2020/04/mvimg_20200417_094147.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload DEBUG SKIPPED download (file
exists) 99200 photos/2020/04/mvimg_20200417_093948.jpg
> 04-17 10:33:45 gphotos.GooglePhotosDownload WARNING Downloaded 6 Items,
Failed 0, Already Downloaded 99200
> 04-17 10:33:45 gphotos.GoogleAlbumsSync WARNING Creating album folder
links to media ...
> 04-17 10:33:45 gphotos.GoogleAlbumsSync DEBUG removing previous album
links tree
> 04-17 10:35:03 gphotos.GoogleAlbumsSync DEBUG adding album link
../../../photos/2017/02/IMG_20170131_194132.jpg ->
/mnt/googlePhotos/Mojgan/albums/2017/0205 Trip to
Havana/0000_IMG_20170131_194132.jpg
> 04-17 10:35:03 gphotos.GoogleAlbumsSync DEBUG new album folder
/mnt/googlePhotos/Mojgan/albums/2017/0205 Trip to Havana
> 04-17 10:35:03 gphotos.LocalData INFO Saving Database ...
> 04-17 10:35:04 gphotos.LocalData INFO Database Saved.
> 04-17 10:35:04 gphotos.Main ERROR
> Process failed.
> Traceback (most recent call last):
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 418,
in main
> self.start(args)
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 369,
in start
> self.do_sync(args)
> File "/home/jordan/gphotos-sync/gphotos-sync/gphotos/Main.py", line 358,
in do_sync
> self.google_albums_sync.create_album_content_links()
> File
"/home/jordan/gphotos-sync/gphotos-sync/gphotos/GoogleAlbumsSync.py", line
322, in create_album_content_links
> follow_symlinks=False,
> PermissionError: [Errno 1] Operation not permitted
> 04-17 10:35:04 gphotos.Main WARNING Done.
> 04-17 10:35:04 gphotos.Main INFO Elapsed time = 0:40:05.521218
>
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <
#221 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/AAHLRW45LTYVKEQV7OXHDRDRNBTWZANCNFSM4MKMTGXQ
>
> .
>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUEQU2SXHLPYFANGIQ3RNCEGBANCNFSM4MKMTGXQ>
.
|
|
I've tried deleting all of the files, logs, configurations, sqlite database, etc and started fresh. Same thing happens: 40K-50K files are downloaded and it errors out. |
|
You have two issues here. In this log #221 (comment) It completed downloading all files and then errored while creating albums because symbolic links were not permitted. Albums are created as a set of links to the existing files in the photos folder. You can fix this by using The other issue logged here #221 (comment) is a failure during download of the file photos/2014/06/vid_20140626_155604.mp4. I would be interested to see the log from your most recent failure because you now have the version of gphotos-sync with better logging. I'm going to guess that this might be some kind of timeout issue, perhaps vid_20140626_155604.mp4 is particularly large? |
|
I am in the middle of yet another test. I was able to confirm that I can create symbolic links on the share from my rpi: When it fails again, I will grab the logs. Is there any way I can send you the complete logs directly? |
|
Hi, Good it looks like symlinks are working, so I do not currently understand the second traceback you sent me. If you would like to send entire logs, you can try https://www.file.io/ or use dropbox or google drive and send to my email [email protected]. |
|
Is there any sensitive data in the logs that I should be aware of?
…On Sat, Apr 18, 2020 at 10:00 AM Giles Knap ***@***.***> wrote:
Hi,
Good it looks like symlinks are working, so I do not currently understand
the second traceback you sent me.
If you would like to send entire logs, you can try https://www.file.io/
or use dropbox or google drive and send to my email ***@***.***
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUFL7CRQV6BBGYITER3RNGW7LANCNFSM4MKMTGXQ>
.
|
|
Thanks for your logs. I can see that it is not failing on creating the links but rather on setting the timestamp on the links. I'm not sure of the reason for this but I have added an exception handler for this case in the branch 429-reponse. Can you try pulling that and having another go please. You will not need to delete what you have downloaded, it might help to delete the albums folder first and you may also need to use --rescan. So we still have not seen the cause of your failure downloading an mp4 file, only failure during the album creation after all files are downloaded. It could have been a transient network issue. So lets see if this fix gets you up and running. Re sensitive info. It will show the names of your albums and photos plus URLs to your photos but the URLs require authentication so not an issue. I won't keep your logs :-) If you get loads of warnings, I'll alter the code to only report the first instance. |
|
I forgot to mention. I m not exactly running it the way I did the other
day. My wife stole my laptop for a meeting and I had to run it on the pi
directly. Here is the script I am using:
#!/bin/bash
echo Backing up Jordans photos
rm -f /home/jordan/gphotos-sync/Jordan/gphotos.lock
export PIPENV_TIMEOUT=240
export EXE=/home/jordan/gphotos-sync/gphotos-sync/gphotos-sync
$EXE --retry-download --progress --db-path
/home/jordan/gphotos-sync/Jordan/ --secret
/home/jordan/gphotos-sync/Jordan/client_secret.json
/mnt/googlePhotos/Jordan/
…On Sat, Apr 18, 2020 at 2:55 PM Jordan Thompson ***@***.***> wrote:
Hi Giles,
I git pulled and switched to 429-reponse
Here is the output:
./gphotos-sync.sh
Backing up Jordans ohotos
04-18 14:43:40 WARNING gphotos-sync 2.14.0+4.37fad11 2020-04-18
14:43:39.833141
04-18 14:43:40 WARNING Indexing Google Photos Files ...
04-18 14:43:41 WARNING indexed 0 items
04-18 14:43:41 WARNING Downloading Photos ...
04-18 14:48:19 WARNING Null response in mediaItems.batchGet
dict_keys(['AMyd5lvwRFQtN2Jje_TwpdWsBfnUSYJfkVvsNojohcSbniPZG5tjnZk2o_EYs1mwTTLohC-SeN7XLpZFeKvjlxUraAigOnY82kulDv3Xy1L_IBa4k0q4KZU',
'AMyd5luDEgSbCcCvaOUQ40IZFe-q2C2sQHyxK-hJbDZ175Gao5xdDC9VqwjGbzqHutbNaB5hUc7Gzbrxm9DgIi3FVuWAijWItg'])
04-18 14:48:19 WARNING Null response in mediaItems.batchGet
dict_keys(['AMyd5lvwRFQtN2Jje_TwpdWsBfnUSYJfkVvsNojohcSbniPZG5tjnZk2o_EYs1mwTTLohC-SeN7XLpZFeKvjlxUraAigOnY82kulDv3Xy1L_IBa4k0q4KZU',
'AMyd5luDEgSbCcCvaOUQ40IZFe-q2C2sQHyxK-hJbDZ175Gao5xdDC9VqwjGbzqHutbNaB5hUc7Gzbrxm9DgIi3FVuWAijWItg'])
04-18 14:48:19 ERROR FAILURE 1 downloading
photos/2014/06/vid_20140626_155604.mp4
04-18 14:48:19 WARNING Downloaded 0 Items, Failed 1, Already Downloaded
33131
04-18 14:48:19 WARNING Done.
I wound up running it twice (if the logs look confusing)
thanks again,
Jordan
On Sat, Apr 18, 2020 at 1:46 PM Giles Knap ***@***.***>
wrote:
> Thanks for your logs. I can see that it is not failing on creating the
> links but rather on setting the timestamp on the links.
>
> I'm not sure of the reason for this but I have added an exception handler
> for this case in the branch 429-reponse. Can you try pulling that and
> having another go please. You will not need to delete what you have
> downloaded, it might help to delete the albums folder first and you may
> also need to use --rescan.
>
> So we still have not seen the cause of your failure downloading an mp4
> file, only failure during the album creation after all files are
> downloaded. It could have been a transient network issue. So lets see if
> this fix gets you up and running.
>
> Re sensitive info. It will show the names of your albums and photos plus
> URLs to your photos but the URLs require authentication so not an issue. I
> won't keep your logs :-)
>
> If you get loads of warnings, I'll alter the code to only report the
> first instance.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#221 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADKSXUGODS2TIHXBPSAKGCDRNHRNZANCNFSM4MKMTGXQ>
> .
>
|
|
Di you get a chance to try the update to branch 429-reponse? |
|
I sent you the updated info but I just checked and saw that it had
bounced. Let me try another way to get the data
…On Sun, Apr 19, 2020 at 11:03 AM Giles Knap ***@***.***> wrote:
Di you get a chance to try the update to branch 429-reponse?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUE6BHTN3S5LRWQULB3RNMHFXANCNFSM4MKMTGXQ>
.
|
|
OK I picked up the new logs from Drive. Did the album links work for you this time? |
|
Hi Giles,
I may have had a user error with git earlier. I am pretty sure that I have
the correct version of it this time (may be a good idea to include the
version in the logs, or at least from the command line ;-) - if it's there
already I missed it).
It is definitely getting further downloading photos this time (still not
gotten to the albums yet.)
I'll keep you posted...
Thanks,
Jordan
…On Sun, Apr 19, 2020, 12:32 PM Giles Knap ***@***.***> wrote:
OK I picked up the new logs from Drive.
It looks to me as though your download completed successfully but that you
have one corrupt media ID for the file
photos/2014/06/vid_20140626_155604.mp4. If I read the logs correctly then
everything else downloaded OK.
Did the album links work for you this time?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUFKYWQQ2OOWRBKLZ3LRNMRRFANCNFSM4MKMTGXQ>
.
|
|
To answer your earlier question, no, there are no new album links created (just the one album with a single file from a few days ago) |
|
@jorythompson I have changed the download threads to never raise errors and only report them. Please give it a try. Your library is also getting the dreaded 'Null response in batch,get' error. There used to be a bug in the API which meant that this would happen occasionally and I had some complex code for working around it. I may have to re-instate this code. (the long story is that the details of photos are requested in batches to reduce the number of requests and improve performance. Batches used to fail because of one bad file ID and this would mean that all of the items in that batch would get missed. I used to individually get every item when this happened and mark the bad item in a bad ids file and always skip it in future scans) |
|
I'm not too happy with never raising errors because a network disconnect will then spew errors for ages. I may need to review this fix. |
|
I should have said the fix has been pushed to master. |
|
Are your changes on the master branch?
Do I need to update any libraries while I'm at it?
…On Mon, Apr 20, 2020 at 1:29 PM Giles Knap ***@***.***> wrote:
@jorythompson <https://github.com/jorythompson> I have changed the
download threads to never raise errors and only report them.
I hope that this will allow your download to complete.
Please give it a try.
Your library is also getting the dreaded 'Null response in batch,get'
error. There used to be a bug in the API which meant that this would happen
occasionally and I had some complex code for working around it. I may have
to re-instate this code.
(the long story is that the details of photos are requested in batches to
reduce the number of requests and improve performance. Batches used to fail
because of one bad file ID and this would mean that all of the items in
that batch would get missed. I used to individually get every item when
this happened and mark the bad item in a bad ids file and always skip it in
future scans)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUABYN3SNMMFGVBWRJ3RNSBAPANCNFSM4MKMTGXQ>
.
|
|
Yes its the master branch. There are no changes to libraries.
On Mon, 20 Apr 2020 at 18:32, Jordan R Thompson <[email protected]>
wrote:
… Are your changes on the master branch?
Do I need to update any libraries while I'm at it?
On Mon, Apr 20, 2020 at 1:29 PM Giles Knap ***@***.***>
wrote:
> @jorythompson <https://github.com/jorythompson> I have changed the
> download threads to never raise errors and only report them.
> I hope that this will allow your download to complete.
>
> Please give it a try.
>
> Your library is also getting the dreaded 'Null response in batch,get'
> error. There used to be a bug in the API which meant that this would
happen
> occasionally and I had some complex code for working around it. I may
have
> to re-instate this code.
>
> (the long story is that the details of photos are requested in batches to
> reduce the number of requests and improve performance. Batches used to
fail
> because of one bad file ID and this would mean that all of the items in
> that batch would get missed. I used to individually get every item when
> this happened and mark the bad item in a bad ids file and always skip it
in
> future scans)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <
#221 (comment)
>,
> or unsubscribe
> <
https://github.com/notifications/unsubscribe-auth/ADKSXUABYN3SNMMFGVBWRJ3RNSBAPANCNFSM4MKMTGXQ
>
> .
>
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAHLRW2D4K23XJBVXKLI4VDRNSBKPANCNFSM4MKMTGXQ>
.
|
|
Ran the app. I get these errors occasionally:
04-20 14:45:22 ERROR bad link to
/mnt/googlePhotos/Jordan/photos/2013/03/img_20130303_171017 (2).jpg
and plenty of these (note the log format is still off):
04-20 14:46:58 WARNING cant set date on {link_file}
and some of these:
04-20 14:54:14 ERROR bad link to
/mnt/googlePhotos/Jordan/photos/2013/02/img_20130209_000302 (2).jpg
It completed without raising an exception (yay!)
Do you need anything else from me?
|
|
Awesome. I will tidy up the exception handling at some point and ask you to check it still works for you. Also I may need to restore the badIDs feature and have you test that. The bad link errors are probably files missing due to your null response from mediaitem.batch errors. Are there alot of them? Do you think you have most of your library backed up? |
|
There are quite a few.
I think most of the photos are backed up but I don't mind deleting them and
starting over if it helps.
Let me know what I can do to help.
…On Mon, Apr 20, 2020, 3:44 PM Giles Knap ***@***.***> wrote:
Awesome.
I will tidy up the exception handling at some point and ask you to check
it still works for you.
Also I may need to restore the badIDs feature and have you test that. The
bad link errors are probably files missing due to your null response from
mediaitem.batch errors. Are there alot of them? Do you think you have most
of your library backed up?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUETZ5CFJUUQKOHPMOTRNSQ3RANCNFSM4MKMTGXQ>
.
|
|
Good morning. I have today restored the Bad Ids feature which was implemented to work around a bug in the Google API. The bug meant that it would deliver some library media IDs that would always fail to download. The bug was fixed but your library seems to be exhibiting a similar issue. What it does is keep a list of google library identifiers that have had an error during download in a file called gphotos.bad_ids.yaml. It will never again try to download the files that appear in that list. It is perfectly OK to delete that file, then all bad IDs will be retried on the next --rescan. In particular when an attempt to download a batch of media items fails, it will crack open the batch and download all of its contents individually, in order to find which item caused the issue. Previous to this feature, when a batch failed because of a single bad ID, all of the items in that batch would not get downloaded. I think it is likely that this is what is happening to you and that is why you are missing files. This was moderately hard to do so I'm not 100% convinced it will work, although I did also restore the related system test and that is passing. Please give it a go and let me know how you get on. I would be interested to see your logs and your gphotos.bad_ids.yaml file from the run. The changes are in a branch called restore-bad-ids. Thanks, giles |
|
One more point. I have restored stricter error handling in the download threads so ... There is a good chance that the initial test is going to fail, I need to capture what class of exception you are getting on these failures. Unfortunately you have binned the previous logs you sent me and I can't access then anymore. So if it fails just send me the error message you get and I'll add that exception to the handler. |
|
Hi Giles,
here are the results from running:
git checkout restore-bad-ids
from the latest version of gphotos-sync
thanks,
Jordan
…On Sun, Apr 26, 2020 at 3:22 AM Giles Knap ***@***.***> wrote:
One more point. I have restored stricter error handling in the download
threads so ...
There is a good chance that the initial test is going to fail, I need to
capture what class of exception you are getting on these failures.
Unfortunately you have binned the previous logs you sent me and I can't
access then anymore. So if it fails just send me the error message you get
and I'll add that exception to the handler.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUHB7ZYVOZYTVNCGDF3ROPOJ3ANCNFSM4MKMTGXQ>
.
|
|
Hi, I don't see any attachments or links in your previous post. |
|
Sorry for the delay, I introduced an issue in the system tests with one of the changes I made for you and it has taken this long to get around to fixing it. There is a new full release 2.14.2 which rolls up all the changes we discussed in this ticket.This now includes coping with a coma in the name for windows filesystems and removing the spam messages when unable to set the file date on album links, Please give it a go and reopen this ticket if you still have issues. Thanks |
|
Just updated and ran. Looks good so far!
Thanks for your efforts,
Jordan
…On Tue, May 12, 2020 at 4:07 PM Giles Knap ***@***.***> wrote:
Sorry for the delay, I introduced an issue in the system tests with one of
the changes I made for you and it has taken this long to get around to
fixing it.
There is a new full release 2.14.2 which rolls up all the changes we
discussed in this ticket.This now includes coping with a coma in the name
for windows filesystems and removing the spam messages when unable to set
the file date on album links,
Please give it a go and reopen this ticket if you still have issues.
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#221 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADKSXUEXTXQWVJEE2TRDKOLRRGUBBANCNFSM4MKMTGXQ>
.
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I am running this command from my laptop on a raspberry pi. The target file system is on a NAS.
ssh megabyte export PIPENV_TIMEOUT=240 ; nohup /usr/local/bin/gphotos-sync --db-path /home/jordan/gphotos-sync/Jordan/ --retry-download --progress --secret /home/jordan/gphotos-sync/Jordan/client_secret.json /mnt/googlePhotos/Jordan/I continue to get errors after downloading thousands of files. I finally tried deleting the sqlite database and other files (keeping the downloaded photos, videos, and albums) and tried again:
Here is the log:
Any suggestions?
The text was updated successfully, but these errors were encountered: