Indexed only photos in albums #112
Comments
|
That does not sound good. Please can you verify that you are passing no parameters except for the root folder? Independent of my code you can see what the Google API is doing by going to https://developers.google.com/photos/library/reference/rest/v1/mediaItems/list Thanks. |
|
Ok, so I checked the API result like you asked but unfortunately through the explorer I can only get files from the last month. When I compared them I have all the ones from August which is odd but I only have 28/61 items from July and unfortunately the explorer won't let me get a result for July. I was thinking it had something to do with Google Drive and Photos no longer syncing but these photos were all uploaded through the android Photos app. It's odd too because my wife's account downloaded 29k/30k files but mine only has <2k/32k. Is there a way I can request the older items? If I have some time I might be able to figure out an API call myself but if you know of something that would be better. |
|
This seems to be a bug in Google's API and one that is not yet reported. I have had other people report similar things but have yet to determine what the common factor is so its not easy to describe a bug to google. See #89 - in that case the users had uploaded all their photos into newly created albums and so the cause does not seem to be the same as for you. Your case seems worrying since it is new photos uploaded in the most standard way. Note that using the API Explorer you will only get the first page of results. You can set the pageSize to 100 which is the maximum. Then you can cut and paste the nextPageToken from the end of the results into the pageToken input field to get the next page. I'm not sure if I fully understand the status of your photos please could you answer the following. I believe we can do a workaround for the missing files if there are not too many but this will be a pain if the API continues to skip newly uploaded files. Simply create an album for the missing files, gphotos-sync should then be able to find them. Thanks, |
|
Here are the answers: So yeah, thinking about it I would agree that there seems to be a bug in the API that isn't letting me get anything older than the current month and that's why I don't have all my files. This does seem to be the opposite of #89 since I think your fix for that is why I'm getting any older files in albums at all :-P |
|
From your description I notice one thing that I won't have tested.
You said that your uploads are in original size. So that means you are
using your storage quota, you are not using the "high quality" free quota
option?
If that is the case. Then I have never tested that so it might be the key.
…On Sat, 24 Aug 2019, 16:12 Matthew Morrow, ***@***.***> wrote:
Here are the answers:
a. I started using Photos in 2012 and also uploaded some older ones, I
believe all in Original size.
b. I can see all of them in the Web. Google dashboard (
https://myaccount.google.com/dashboard) reports and 28921 photos in 1672
albums and ~190GB
c. I only get ~1700 from gphotos-sync. Anything older than this month
doesn't seem to be coming back unless it is in an album. It's only about
~8GB of files.
d. With the API explorer I can only get 76 results back, as early as
2019-08-02. If I put in the next page result I get blank pages.
So yeah, thinking about it I would agree that there seems to be a bug in
the API that isn't letting me get anything older than the current month and
that's why I don't have all my files.
This does seem to be the opposite of #89
<#89> since I think your
fix for that is why I'm getting any older files in albums at all :-P
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#112?email_source=notifications&email_token=AAHLRWYYF4KMXBC3PCVDHXLQGFFXVA5CNFSM4IOWDOUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CCAOI#issuecomment-524558393>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHLRW5XEHATXLVCERGUQPDQGFFXVANCNFSM4IOWDOUA>
.
|
|
I just tried uploading an original quality file and was able to sync it with gphotos-sync. |
|
Oh! I figured something out. So in the API I only get 76 items on the first "page" and the next page is blank. But if I continue putting in To test this, I pulled down the repository and removed the break when |
|
Definitely a bug in the API but an easy workaround.
I shall put your fix in the code and also report to Google.
Thanks for a really insightful investigation.
…On Sun, 25 Aug 2019, 08:07 Matthew Morrow, ***@***.***> wrote:
Oh! I figured something out. So in the API I only get 76 items on the
first "page" and the next page is blank. But if I continue putting in
nextPageToken repeatedly, I eventually get more results items!
To test this, I pulled down the repository and removed the break when
media_json is empty and it goes on to grab the next page until
nextPageToken is empty. When I ran it again I got 7660 items this time
which is 5x what I got before. So maybe the code should only rely on
nextPageToken to stop?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#112?email_source=notifications&email_token=AAHLRWY4AXW3YDMFWR6YT63QGIVURA5CNFSM4IOWDOUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5CN3ZI#issuecomment-524606949>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAHLRW6UMHLZTVO36SF3CALQGIVURANCNFSM4IOWDOUA>
.
|
|
I'm still curious why I get 7k/32k photos but I feel like this issue is closed when that change get's added. On a side note, when it was available, I used Google's Backup and Sync with "upload to Google Photos" for some photos and those all seem to be coming back fine now too. Those empty pages are bizarre. |
|
OK, I have added the change to the branch 'feature' please try it out. In testing this I did note some weird behaviour with low pageSize numbers and was able to get similar behaviour to you with pageSize=2. |
|
Would you mind reporting the bug you have found to Google? To do so click on New Issue on this page https://issuetracker.google.com/issues?q=componentid:385336%20status:open When I reported https://issuetracker.google.com/issues/116842164 they asked for permission to access my library for testing and then did eventually fix the issue. Since your library is showing the problem it would be easiest for you to do the report if you are comfortable with that. |
|
@matthewmmorrow did you get a chance to see if feature branch fixed your issue? |
|
no news from @matthewmmorrow. I'm pretty sure this will fix the issue and seems to be a benign change anyway. Todo - merge into master for next release. |
|
merged to master |
From what I can tell, after running this the only photos that I have are ones that are from albums. I don't have any from the main list. I get the same result with the
--flush-indexoption.The text was updated successfully, but these errors were encountered: