Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for /sync/v4 #311

Closed
d2r opened this issue Jul 27, 2024 · 10 comments · Fixed by #312
Closed

Support for /sync/v4 #311

d2r opened this issue Jul 27, 2024 · 10 comments · Fixed by #312

Comments

@d2r
Copy link

d2r commented Jul 27, 2024

RM Software is 3.13.1.2
rmfakecloud version 0.0.18
Setup: rmfakecloud running behind nginx reverse proxy; DNS is handled by resolver rather than modifying /etc/hosts on the tablet

My sync is failing 100% of the time after upgrading the tablet to 3.13.1.2. It failed with rmfakecloud v0.0.17, and upgrading to v0.0.18 did not help.

Packet capture reveals the following exchange. Could this be preventing sync?

GET /sync/v4/root HTTP/1.1
X-Forwarded-For: <REDACTED IP address>
Host: internal.cloud.remarkable.com
Connection: Upgrade
authorization: Bearer <REDACTED>
rm-filename: roothash
user-agent: xochitl/3.13.1.2 (codex 4.0.617-2-ga920cbd6)
accept-encoding: zstd, gzip, deflate
accept-language: en,*

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 485
Content-Type: text/html; charset=utf-8
Date: Sat, 27 Jul 2024 16:36:11 GMT

<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="RM FakeApi"/><title>rmfakecloud</title><script defer="defer" src="/static/js/main.9c2de5b1.js"></script><link href="/static/css/main.d94d89ba.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

The client, xochitl, closes the TCP connection immediately after acknowledging this response. I don't see anything in the xochitl debug logs that gives any clues, but it seemed suspicious to me. Do we need a new syncGetRootV4?
@Eeems
Copy link
Contributor

Eeems commented Jul 27, 2024 •
edited
Loading

rmfakecloud doesn't have any v4 routes yet, so yes, all the new v4 routes would need to be added to routes.go

As for if there needs to be a new method for them, that depends on if the API itself has changed for that call or not.

From the readme:

The current release of rmfakecloud support file synchronization for SW <= 3.10.2. Newer releases have not been tested yet.

@d2r d2r changed the title HTTP 200 for /sync/v4/root/ Support for /sync/v4 Jul 27, 2024
@Silloky
Copy link

Silloky commented Aug 6, 2024

Can we expect the v4 routes to work in the same way as v3 ?
In which case, all we need to do is to copy-paste the v3 lines and edit to v4...
I'll try and keep you informed.

@martinetd
Copy link
Contributor

FWIW I had tried to copy the v3 routes to v4 and all I can say is sync still is unhappy (root was the same so that was a fair attempt, but if it's really the same it would just keep using the v3 endpoint, I assume the v4 one returns more informations)

I don't have an account on the official cloud to take traces here so didn't try anything else; sending notes via mail works and I assume I can still send files over usb if I really need to so didn't press this further.
If someone with a real account could take traces I'd assume this won't be too hard to fill in the gaps...

@nemunaire nemunaire mentioned this issue Aug 6, 2024
Merged
@nemunaire
Copy link
Collaborator

FWIW I had tried to copy the v3 routes to v4 and all I can say is sync still is unhappy (root was the same so that was a fair attempt, but if it's really the same it would just keep using the v3 endpoint, I assume the v4 one returns more informations)

Are you on 3.13.1?

I just made the upgrade and with just the new route GET /sync/v4/root, the file sync works well. I didn't see any change in the format, and I don't record any other new route.

@martinetd
Copy link
Contributor

Hmm, the 'check sync' option fails with something similar to what you've done and I don't see new files when uploading with rmapi so I'm surprised it works for you; I'll check again when I have a moment.
Will be a few days so don't wait for me, if someone else can confirm this works on 3.13.x it's great

@martinetd
Copy link
Contributor

Found some time before bed -- upgraded to 0.0.19 (which contains #312) and can confirm it's still broken for me, but then again it's possibly something local to my device -- I upgraded from 3.0 (!) and after upgrade some files re-appeared that I had deleted (fairly old, pretty sure they weren't just in trash), and the device might just considers it has been unsync'ed or something.
With my normal user on the "Check sync" button I get "There are some issues with the cloud connection. Please try again later"; I have this in the journal

Aug 06 13:07:30 reMarkable rm-sync[223]: rm.synchronizer:  -- execute requested: 2024-08-06T13:07:30.818
Aug 06 13:07:30 reMarkable rm-sync[223]: rm.synchronizer:  -- running sync, batch: 0/1, minimal:false
Aug 06 13:07:30 reMarkable rm-sync[223]: rm.synchronizer:  - read "/home/root/.local/share/remarkable/xochitl/.tree" in 00.000037749s (failed)
Aug 06 13:07:34 reMarkable rm-sync[223]: rm.synchronizer: mismatch with node marking and token on disk:23e6b174-ec95-405b-8f4b-186461adb2f3 disk: true marked:false
Aug 06 13:07:34 reMarkable rm-sync[223]: rm.synchronizer: Archived state signalled for:23e6b174-ec95-405b-8f4b-186461adb2f3 paths:["23e6b174-ec95-405b-8f4b-186461adb2f3.metadata"]

I tried creating a new user and it gets a 404 error:

Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  -- execute requested: 2024-08-06T13:05:28.348
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  -- running sync, batch: 0/1, minimal:false
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  - read "/home/root/.local/share/remarkable/xochitl/.tree" in 00.000057874s (failed)
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer: Root hash error without proper error reply
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  -> content not found when transferring for .: {"message":"root not found"}
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  -> failing rmsync::DownloadRootHashJob for . with RequestNetworkError, 404, "{\"message\":\"root not found\"}"
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer: network job failed: path: ., code: RequestNetworkError, domainError: 404, details: {"message":"root not found"} job:1, availability=Availability::Online
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer: Oh no!!!, rmsync failed.. reason="Network failure: 404", detail="{\"message\":\"root not found\"}"
Aug 06 13:05:28 reMarkable rm-sync[223]: rm.synchronizer:  -- execute ended: 2024-08-06T13:05:28.417
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  -- execute requested: 2024-08-06T13:05:34.608
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  -- running sync, batch: 0/1, minimal:false
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  - read "/home/root/.local/share/remarkable/xochitl/.tree" in 00.000035874s (failed)
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer: Root hash error without proper error reply
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  -> content not found when transferring for .: {"message":"root not found"}
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  -> failing rmsync::DownloadRootHashJob for . with RequestNetworkError, 404, "{\"message\":\"root not found\"}"
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer: network job failed: path: ., code: RequestNetworkError, domainError: 404, details: {"message":"root not found"} job:1, availability=Availability::Online
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer: Oh no!!!, rmsync failed.. reason="Network failure: 404", detail="{\"message\":\"root not found\"}"
Aug 06 13:05:34 reMarkable rm-sync[223]: rm.synchronizer:  -- execute ended: 2024-08-06T13:05:34.674

I tried removing the .tree file (after backing up ...) and sync appears to work again, except the phantom files don't appear in the cloud, so I better delete them again from the remarkable...

Anyway, sorry for the noise; I'm still not convinced they'd go out of their way to create a v4 endpoint if it's identical to v3 but it could be something for their own convenience on the backend side so I'll pretend I never saw this and happily use the fake cloud. Thanks!

@nemunaire
Copy link
Collaborator

nemunaire commented Aug 6, 2024 •
edited
Loading

Sure, they didn't make a v4 for nothing. The caveat is to play with the official route to identify the differences. One notable difference seems to be at the beginning of the account setup, as observed from your test when no sync has been done before. The 404 error you encountered was expected in v3, but since checking that behavior requires a new official cloud account, it's quite difficult to verify what is expected for v4.
reMarkable mentions they reworked their onboarding process, so perhaps now the cloud account is populated with a dummy file?

To move forward, perhaps the v4 route behaves like our v3 did in the past. Reverting 98c7da8 might help, or we might need to create an empty index and return it.

Regarding archived files, I experience the same issue. The fakecloud doesn't handle them properly yet; it works as a side effect. However, it doesn't stop the sync process.

As for the phantom files, it could be related to https://ddvk.github.io/rmfakecloud/usage/diff-sync/#deal-with-file-lost?

@Silloky
Copy link

Silloky commented Aug 6, 2024

Oh, this is great, thanks @nemunaire !
That's exactly what I was planning on trying 10 minutes ago, but I thought I should check the issue first, in case other people have already fixed it.
Also, what do you mean the archive doesn't work well ? I used it yesterday on SW version 3.12 and it was working fine. Is this 'side-effect' related to the current fix with v4 ?

@nemunaire
Copy link
Collaborator

nemunaire commented Aug 6, 2024 •
edited
Loading

Also, what do you mean the archive doesn't work well ?

When the tablet send an archive request, the fakecloud returns OK, but doesn't perform any action under the hood.
Later, on resync/version upgrade, this creates incomprehension on the tablet because archived files are not listed as archived (as we don't store the archived state). The archive feature works (the file is correctly removed from the tablet and can be retrieved as expected), but the archived state doesn't survive a factory reset.

@Silloky
Copy link

Silloky commented Aug 6, 2024

Right, ok, got it.
The archived status is only stored on the tablet, not in the cloud.

@d2r d2r mentioned this issue Sep 14, 2024
Closed
@lowgos lowgos mentioned this issue Oct 15, 2024
Closed
@nemunaire nemunaire mentioned this issue Oct 17, 2024
Merged
@davidpkj davidpkj mentioned this issue Nov 7, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

New route for v3.13.1 nemunaire/rmfakecloud
5 participants
@nemunaire @d2r @Eeems @martinetd @Silloky