some photos not indexed

[gilesknap]

One user has given me log files that show most of his photos not being indexed despite the fact that they are seen in (non-shared) albums.

I think this is similar to the index vs shared bug I have seen in the API before. A workaroud would be to always index all contents of albums as well as using mediaitems.index/search.

[beele]

I seem to have a similar issue.
I synced everything and see that only from now till 2016 is being synced. All photo's before that are not present (1998-2015)

Edit:
It does seem to log all the photos in the log file, it just doesn't download them.

[gilesknap]

Hi. Did you originally upload your earlier photos to Google drive?

…

On Fri, 26 Jul 2019, 16:59 Kevin Van den Abeele, ***@***.***> wrote: I seem to have a similar issue. I synced everything and see that only from now till 2016 is being synced. All photo's before that are not present (1998-2015) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#89?email_source=notifications&email_token=AAHLRW4OAI7QN6ZOQRGGPX3QBMNPJA5CNFSM4IFTGQWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25ASAI#issuecomment-515508481>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRW4PET4BQNZMHQQ5XMLQBMNPJANCNFSM4IFTGQWA> .

[beele]

No all my photo's are uploaded either through the google photos website or through my phone via the photos app. (I did have the option active to show all my google photos in a folder inside my drive, but I never added any new photos though that)

If you want I can provide a log file and or help (I'm a dev myself)

EDIT: I've checked the log file a bit better. It seems the photos are all listed when it's indexing the albums but they are absent during the earlier indexing of the photos themselves

[gilesknap]

@beele thanks for the additional info. It does sound exactly like the same problem the other user is having.

I'm working on a change so that all photos found in albums will get indexed and downloaded even if they are not seen in the main index. However, this would not help for those of your photos that are pre-2016 and not in an album.

Please do send me a log of a new run with --flush-index option (and no other options).

Thanks.

[beele]

Most, if not all of my pre 2016 photos are all in albums If I'm not mistaken.

I've rerun the command and attached the log file here. If there's anything else I can help with, do let me know.

gphotos.log

[gilesknap]

@beele thanks.

I have reproduced the mediaitems.list vs mediaitems.search issue in my test library. This may not be the same bug since both users in this thread see missing files for list and search. However, it looks similar.

Also, I'm testing the 'index all album links' fix and it looks OK, its just trashed my tests and highlighted the above.

[gilesknap]

@beele (@ Fred) if you would like to try out the indexing of all albums option then there is a version that does this in https://github.com/gilesknap/gphotos-sync/tree/testing-index-all-albums.

It does not pass system tests but that is because it highlights related issues with my test library.

[beele]

Thx! I tried it and it indeed it started to download more files, however it gave an error on a duplicate file (the file was not there before, since the last folder downloaded was 2016/10)

07-27 11:22:43 FAILURE 1 downloading photos\2016\07\IMG_0016.JPG
07-27 11:22:43 FAILURE 2 downloading photos\2016\07\IMG_0016.JPG

FileExistsError: [WinError 183] Kan geen bestand maken dat al bestaat: 'G:\\Data\\google_photos\\photos\\2016\\07\\tmp8nyz8gdi' -> 'G:\\Data\\google_photos\\photos\\2016\\07\\IMG_0016.JPG'
07-27 11:22:44 Done.

The error translates to: 'Cannot make a file that already exists'

[gilesknap]

@beele thanks for the update. Would you mind sending me your gphotos.sqlite file? You can email it to [email protected] if you are OK with that.

[gilesknap]

Also, the log file from the failed attempt if you have it, please.

[gilesknap]

Thanks for the files. This is a new bug for windows only :-)
You have files called IMG_0016.JPG and IMG_0016.jpg in the same folder. I see these as unique and indeed they are on the linux filesystem. It seems they are not on your filesystem.

We can do a fix to the duplicate matching to ignore case.

I'll push something soon.

[gilesknap]

Please try the latest code in https://github.com/gilesknap/gphotos-sync/tree/testing-index-all-albums.

Before merging this I need to:

• make the change conditional on windows os otherwise peoples local copies may change unexpectedly
• add tests to CI for duplicate names since there is none currently (include one that has case changes only for windows check)
• fixup the shared folder tests item counting and verify I'm happy with the new regime
• consider 'index all albums' as an option instead??

[beele]

It worked perfectly this time, I'm still going though it but it seems it's all there!
Thanks!

[gilesknap]

That's great. I'd still like to try and diagnose why the Google API does not pick up your photos. So have a couple of questions. Are any of your albums shared with others? Do you share libraries between other accounts? Can you think of anything that changed at the cutoff date? (Switched phone, started sync with Google drive etc.) Thanks.

…

On Sat, 27 Jul 2019, 12:32 Kevin Van den Abeele, ***@***.***> wrote: It worked perfectly this time, I'm still going though it but it seems it's all there! Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#89?email_source=notifications&email_token=AAHLRW2WNI6U3Y2O2ON5UT3QBQW45A5CNFSM4IFTGQWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26JZFA#issuecomment-515677332>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHLRW3DEZMSJR5E5EHMNUDQBQW45ANCNFSM4IFTGQWA> .

[fstanland]

The update also solved my problem. I went from 89 downloaded to 1888 downloaded. Thanks.

[beele]

That's great. I'd still like to try and diagnose why the Google API does not pick up your photos. So have a couple of questions. Are any of your albums shared with others? Do you share libraries between other accounts? Can you think of anything that changed at the cutoff date? (Switched phone, started sync with Google drive etc.) Thanks.
…
On Sat, 27 Jul 2019, 12:32 Kevin Van den Abeele, @.***> wrote: It worked perfectly this time, I'm still going though it but it seems it's all there! Thanks! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#89?email_source=notifications&email_token=AAHLRW2WNI6U3Y2O2ON5UT3QBQW45A5CNFSM4IFTGQWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26JZFA#issuecomment-515677332>, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHLRW3DEZMSJR5E5EHMNUDQBQW45ANCNFSM4IFTGQWA .

Some of my albums are shared with other photos users or publicly viewable but not all (and not all of those that were previously missing).
My entire library is shared with my partner.
I din't change phone recently. I did ad most of the missing albums all around the same one or two day period. In the past I've also had issues with the sync in drive (new photos not showing up there, removed the folder and let it sync to drive again from scratch, after contact with google support)

[gilesknap]

So I guess you uploaded the missing folders with Google backup and sync?

How about the newer folders that worked?

If your library is shared with your partner, is it still your account that uploads or did photos come from both accounts.

Thanks for all the info.

[beele]

I didn't upload any photos via backup and sync. All via the web-gui of google photos (creating an album, then adding all the photos)

Of the ones that did show up, I can't really tell what's different.

All photos are uploaded through my account, shared with my partner just for access.

[gilesknap]

(creating an album, then adding all the photos)

That is not a workflow I use and it could be related @fstanland do you use this workflow?

I've tried adding an album to my test account using this approach and it does not exhibit the problem but perhaps the issue has been fixed since you made these uploads.

Anyway, the very good news is that this should mean that all of your photos are now indexed since the fix will cover all instances of your workflow.

[gilesknap]

Merged into master.

• I added --no-album-index to turn off the new indexing just in case - but the default is to always do this now.
• tidied up tests and added tests for case-insensitive file systems and duplicates