Option to back up all apps by pretending to be a device to device transport

[zroug]

Seedvault is bound to Android's backup API which allows apps to exclude themselves from getting backed up. We want to implement another backup mechanism that will allow us to backup more apps by using an API loophole: pretending to do a device-to-device backup.

• Handle a larger number of apps getting backed up in a single backup pass
• Warn about insufficient storage space on backup medium (more likely when backing up more apps without storage quota)
• Pretend (to the system) to do a device-to-device backup run instead of cloud backups

---

Android 11 ignores the allowBackup manifest attribute for device to device transfers (https://developer.android.com/about/versions/11/behavior-changes-11#device-to-device-file-transfer). That means the local backup transport supports it in principle and the apps must support it in principle too. I think it would be a nice addition to Seedvault to make this behavior a (configurable) option. If this is indeed possible of course. This is partly related to #36.

[grote]

Seedvault would have to pretend to be a device-to-device backup agent. Not sure that's something we would want to do, but it should be as easy as changing a single line of code.

[IzzySoft]

Well, maybe you could consider at least a locally attached drive (Flashdrive, SDCard) as "device" – so it would be a device2device transfer? 😄 A power-user-option behind a bunch of warnings would also be acceptible as long as the goal is reached. Backups should be as complete as possible. Of course this would be welcome to < 11 as well 😉

[chirayudesai]

One thing to note here is that just ignoring this flag won't let all apps actually backup properly.

There's multiple reasons for an app to disallow backups. Sometimes it's about not wanting to send backups to Google, sometimes it's about not wanting unencrypted copies (both Google and Seedvault encrypt, and there's an explicit option for this as well), or maybe something else.

Other times it may be because the app might not actually be able to get backed up this way.

The apps would get backed up when the flag is ignored, and it would even say "backup successful". But then when you actually try to restore the app, it may or may not work. And that'd be worse than not having it backed up at all - because you may have made alternate arrangements knowing that, but now if it looks like backup worked maybe not, only to find out later.

Point being, this would need testing. Testing a device-to-device migration would help, since it ignores this flag - seeing what it backs up / restores, what it skips, and if everything works / something doesn't work would be helpful.

Google's change is only for apps' targeting Android 11, of which there may not be many right now (especially with allowBackup=false)

From the Play store's target API level requirements, for targetSdkVersion=30 we have:

• August 2, 2021: Required for new apps
• November 1, 2021: Required for app updates

[IzzySoft]

And that'd be worse than not having it backed up at all

I get your reasoning here – and though we all know (at least at a theoretical level) that one should always make restore tests (in business even at least once a year), that rarely happens in reality. So I'm fully with you putting a big red warning box crying "danger" and asking for additional approval (maybe even requiring to type in, literally, "I understand the risks" – or, if you prefer, the fitting Potter for the Marauder's map). Making clear that just because the backup succeeds doesn't implicate the restore will as well should help to at least think about additional measures. If not – well, "not my fault – you have been warned" 😜

If you insist, there could even be a warning summary at each backup run, in red letters: "… 11 apps might not be able to restore …", or something like that, as a reminder. Turning your phrase around: it's better to have a backup which might not fully restore than to have none at all (you might still be able to "carve" parts out of it). Especially without root, there are apps you cannot backup by other means.

Point being, this would need testing.

Agreed. And if there are some "known black sheep", they could be, uh, blacklisted. OTOH apps targeting 11 might even be white-listed by default if they should be expected to work with this.

[thestinger]

Apps can still presumably still blacklist their files from being backed up. They could blacklist all of their files if none of them should be backed up.

[IzzySoft]

The question is: should they? And who owns the data? I still don't like the idea that an app tells me what I may gratefully backup and what not. On my Linux system, there are no such restrictions. And nobody would keep me from backing up even e.g. /dev or /proc. And no, I wouldn't try restoring THAT from a backup either. If I would not know about such culprit, I'd expect a warning. But in the end I'd be the one to decide. Make it an advanced option in the "danger zone" if you like. Etc, I've already outlined it above 😉

[thestinger]

So if the user's computer is compromised and they've whitelisted ADB access, adb backup should be able to steal all of their data?

[thestinger]

Also, a lot of data is not portable across devices and there are cases where it would be insecure to move it. Either way, an app can encrypt data with hardware keystore keys and you won't be able to decrypt that outside the device. For example, bypassing this for Signal won't accomplish anything. The database is encrypted with hardware keystore keys.

[IzzySoft]

If your server is compromised… well, you just answered that yourself: use encryption. And we're speaking about a local backup here, not adb backup called from the computer, not a backup to the cloud. And yes, we've discussed all this before – hence the warning. To my experience, 90% of the apps disallowing backups have no issues to either backup or restore (I've overridden this before via Xposed – and I'm using Titanium Backup for more than 10 years to backup AND RESTORE).

[callahan22]

One thing to note here is that just ignoring this flag won't let all apps actually backup properly.

There's multiple reasons for an app to disallow backups. Sometimes it's about not wanting to send backups to Google, sometimes it's about not wanting unencrypted copies (both Google and Seedvault encrypt, and there's an explicit option for this as well), or maybe something else.

Other times it may be because the app might not actually be able to get backed up this way.

The apps would get backed up when the flag is ignored, and it would even say "backup successful". But then when you actually try to restore the app, it may or may not work. And that'd be worse than not having it backed up at all - because you may have made alternate arrangements knowing that, but now if it looks like backup worked maybe not, only to find out later.

Point being, this would need testing. Testing a device-to-device migration would help, since it ignores this flag - seeing what it backs up / restores, what it skips, and if everything works / something doesn't work would be helpful.

Google's change is only for apps' targeting Android 11, of which there may not be many right now (especially with allowBackup=false)

From the Play store's target API level requirements, for targetSdkVersion=30 we have:

• August 2, 2021: Required for new apps
• November 1, 2021: Required for app updates

I've just come across this same issue with Calyx using SeedVault for backup. I was naively under the impression it would just back up everything I have. I totally understand the reasons it won't and accept that it is what it is and your reasoning makes sense. A false sense of security you get from everything appearing to backup leads to to you potentially taking risks you otherwise wouldn't.

My suggestion would be that there is the option to output a text file or similar that includes all the apps that were not backed up from the device. At least this way, when the worst happens and we are rebuilding our phones from scratch, we have an up-to-date list of all the apps missing from the backup. This would speed up the recovery massively. Google's App Store has something similar to this, in that one of it's tabs lists all the apps you had previously downloaded thereby giving you something to work with rather than trying to remember 2 years + of apps you had installed at point of crash/wipe/theft/tinkering...

[thestinger]

It shouldn't be ignoring the flag completely and probably doesn't have an option to do that anyway. Rather, it makes sense to ignore it for apps targeting API 30+, which is probably all that this app can do anyway. Those apps are expected to move to blacklisting files that are not supposed to be backed up. They can still exclude everything from backups. It just requires them to be more explicit about it rather than simply turning off backups.

[grote]

Does anybody know an app that already targets Android 11, but disallows backups? So we can test how it behaves?

[thestinger]

You could fairly easily make an example by adding allowBackup="false" to an API 30 app. It will be ignored for device-to-device backup. It will still respect blacklisting files, overriding the backup service implementation with a custom one (which can be used to convert to a portable format), etc.

[grote]

So I tried this and it didn't work. When trying to find out why I fail to find anything in the AOSP source code supporting that document behavior change.

Here allowBackup=false is still sufficient to consider the app ineligable for backup: https://cs.android.com/android/platform/superproject/+/android-11.0.0_r3:frameworks/base/services/backup/java/com/android/server/backup/utils/AppBackupUtils.java;l=56

If you search for FLAG_DEVICE_TO_DEVICE_TRANSFER, you find only this as the only relevant place: https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/android/app/backup/FullBackup.java;l=606?q=FLAG_DEVICE_TO_DEVICE_TRANSFER which doesn't lead anywhere either.

So in short, I couldn't find any evidence in the public AOSP source for the documented behavior change and it practice, it doesn't work.

[grote]

You could fairly easily make an example by adding allowBackup="false" to an API 30 app. It will be ignored for device-to-device backup.

Turns out I can't. At least not without also publishing this app to Google Play, because their d2d data transfer is only offering to copy apps from Google play. Sideloaded apps or apps signed with a different key simply aren't offered.

So if anybody knows a allowBackup="false" app that is already targetting API 30 on Google Play, please let me know!

[chirayudesai]

So if anybody knows a allowBackup="false" app that is already targetting API 30 on Google Play, please let me know!

https://chromium.googlesource.com/chromium/src/+/18c24f658c374c1aa581b80d43782e26dabda2bf/chrome/android/java/AndroidManifest.xml#167

Something based on Chromium might work.

[grote]

I tried with Signal v5.4.12 which should be targeting API 30: signalapp/Signal-Android@75062a
and also does not use any backup file excludes: https://github.com/signalapp/Signal-Android/blob/75062a/app/src/main/AndroidManifest.xml#L94-L101

The result was that contrary to other apps that do allow backup, Signal started like it had no data. So either there's a check in the source code that detects that it is on a different phone or this forced d2d backup is a myth.

[chirayudesai]

The result was that contrary to other apps that do allow backup, Signal started like it had no data. So either there's a check in the source code that detects that it is on a different phone or this forced d2d backup is a myth.

At this point given that this is all Play Services and not AOSP I would say we can personally try to just go with the spirit of what they wanted, and do this anyway.

However, one of the reasons I didn't just want to ignore allowBackup willy nilly is cases like these - where it seems to have backed up but it won't actually, which means the user thinks there's a backup but there isn't.

I can think of 2 ways to test this

1. Ignore allowBackup in Seedvault for either all apps or just API >= 30, and then see how well that restores.
2. Submit a simple app to Google Play where we can control what data it has and see how this works with that. There might be some other hidden criteria which we can only speculate on at this point.

[grote]

Ignore allowBackup in Seedvault for either all apps or just API >= 30, and then see how well that restores.

We can't ignore allowBackup in Seedvault, we can only tell the system that we are a D2D backup transport. I had tried that as written above and it did not have any effect.

Submit a simple app to Google Play where we can control what data it has and see how this works with that. There might be some other hidden criteria which we can only speculate on at this point.

So I used a "internal test" version of an app from Google Play. Upgraded on the source phone to that version which targets API 30. Then I factory reset the target phone, did the setup wizard and d2d app copying. The app is offered as an option and gets copied. In Google play it shows as the correct internal test version. When starting it, it starts without any previous app data.

I think we've finally arrived at a point where I can call a lie even though it is officially documented.

[thestinger]

It's possible it's a bug in either the code (wasn't shipped as intended) or documentation (feature was delayed but they didn't remove the documentation).

[Victor239]

Android documentation says that it's an upcoming change in Android 12 to ignore android:allowBackup="false" for D2D transfers.

[rugk]

BTW there is a Beta of Android 12 available, did anyone already try or dig into whether that will be supported there?
https://developer.android.com/about/versions/12

[thestinger]

https://developer.android.com/about/versions/12/backup-restore

[IzzySoft]

That basically means no restrictions at all to D2D unless explicitly specified for D2D – in other words, apps not having that section would be fully available to D2D backup.

Yes, I saw the restriction that this only applies to apps targeting Android 12+ (as before that, this section simply could not exist) – so I don't want to start a discussion on 11 and below again (we had that above). Just mentioning one reason for the exclusion from backup, as the linked page states for its only example, quote:

That allows you, for example, to exclude a file or directory from Google Drive backups while still transferring it during D2D transfers. This may be useful if you have files which would be too large to back up to the cloud, but which can be transferred between devices without issue.

Unfortunately, there's no field for the "exclude reason" to go by so it could be ignored for "size" while being honored for "TPM" (as with that, restore might fail).

[pshirshov]

I've tested this patch and figured out that it breaks things. It makes backup service to backup everything, including system apps. When backup service kills "vendor rro" before backing it up the backup fails.

I've come up with another patch which actually lets the backup to finish and even restore works (I've tested it on Signal).

The idea is quiet simple: instead of patching package parser I teach backup service to disrespect the flag for all the non-system apps (thus running backup on them all) but still respect the flag for system apps.

There is just one issue with this patch - seedvault UI still displays the apps as "disallowing backups" and does not let me to exclude some from backup. This is, well, expected behaviour though I think that it may be a good idea to discuss how to properly combine my patch with some UI changes so we may have first class backup experience...

backup2.diff.txt

[grote]

What patch did you test exactly? It sounds like you are modifying AOSP and not Seedvault.

[pshirshov]

Yes, this is the same as in case of allowBackup patch. From what I can see, Seedvault is a frontend for android backup facilities and has no control over these aspects.

In order to be able to perform backup the system needs to be patched, not seedvault. Though while backup works with this patch, seedvault UI gets unaligned with that and I'm not sure what would be the best way to address that.

[grote]

@pshirshov this ticket is about making seedvault back up more apps by pretending to be a device-to-device transport. OS patches can be done in custom ROMs as well, but are then unrelated to seedvault itself.

[grote]

Just realized that I had never pushed the branch where I tried to pretend to be a d2d transport. Here it is, in case anybody wants to try this with Android 12: grote@78f417b

[grote]

BackupManager now has a new #requestBackup() method that takes an @OperationType int where we can pass in OperationType.MIGRATION to get all app data, if we consider this a safe thing to do and can handle the hundreds of MB or GB this might give us.

• https://gitlab.com/CalyxOS/platform_frameworks_base/-/commit/2e5a1c222fab8b6f302a01249a92af58b088fbea
• https://gitlab.com/CalyxOS/platform_frameworks_base/-/commit/2f2e13c1b6cae5fde3e8bfe8a08e7badab9c243a

Edit: It seems that even restore needs to be started in migration mode.

• https://gitlab.com/CalyxOS/platform_frameworks_base/-/commit/9bdd2f67192def131adf015b6f5adcc3b112c847

[grote]

Ok, I am reading the commits sorted by date and they have reverted the @OperationType parameter again now simply relying on the transport flags which is unfortunate, because then we can't simply do hacks like re-calling requestBackup for apps that go over quota when using MIGRATION type backup.

[grote]

https://gitlab.com/CalyxOS/platform_frameworks_base/-/commit/5c3913d9c1e51e8043c131891da43ae280955aeb

A cloud transport can still be responsible for restore of D2D data. Instead, determine the operation type for each restore set individually.

Our getAvailableRestoreSets() method would need to be updated to use the new constructor which can pass in backupTransportFlags for the RestoreSet.

[doak]

Why is #473 stuck? Does it need a "maintainer"?

[chirayudesai]

Why is #473 stuck? Does it need a "maintainer"?

It needs further testing and review, and all of the developers are currently busy with one thing or the other. It's on our list, but it'll take some time to get this done.