artifact url improvment for f-droid binaries

[thecockatiel]

Hi
I'm trying to do the necessary work for releasing this app on f-droid, regarding issue #1700
it would be nice if there could be some adjustments made to the website to help ease stuff a bit for reproducible builds
currently the url is like this: https://download.electrum.org/4.5.5/Electrum-4.5.5.0-arm64-v8a-release.apk

it would be nice if you can make https://download.electrum.org/4.5.5.0/Electrum-4.5.5.0-arm64-v8a-release.apk also point to the same file ( 4.5.5.0 -> 4.5.5 redirection)

I'll also open a PR for adding the fastlane metadata files for f-droid, which can help you control the metadata without opening MRs on f-droid
if you don't like the idea of metadata living inside the repo, I can still add it directly to f-droid repo, It's your decision

[thecockatiel]

I have made the required metadata to build the app, but I'm having a hard time reproducing the exact apk
any tips ?
I have used the same env, sdk, ndk ,ant and steps of the dockerfile inside the gitlab runner, but it's producing different apk by a few kilobytes
most notably the pyc files inside the apk have a slightly different size than the one produced from docker image
the python version seems to be the same, so I am having a hard time tracking down the issue

[thecockatiel]

can you list possible things that can cause the build to vary ?

[thecockatiel]

I even tried replacing the base image with registry.gitlab.com/fdroid/fdroidserver:buildserver-bookworm and deleting the existing user with id 1000 and tried building the image and it still fails

[thecockatiel]

the private.tar that is included in the apk is one part of the issue. the path at which the project is built affects the reproduction. for example inside the main.pyc inside the private.tar that is included in the assets of the apk we see this path:

which should not be there I think, but is.

my guess is that it's maybe the same kind of issue happening in .so files in lib

[SomberNight]

Thank you for looking into this!

it would be nice if there could be some adjustments made to the website to help ease stuff a bit for reproducible builds
currently the url is like this: https://download.electrum.org/4.5.5/Electrum-4.5.5.0-arm64-v8a-release.apk

it would be nice if you can make https://download.electrum.org/4.5.5.0/Electrum-4.5.5.0-arm64-v8a-release.apk also point to the same file ( 4.5.5.0 -> 4.5.5 redirection)

I think we could drop the trailing .0 from the android apks. It is a historical thing that we don't need/utilise anymore. (Some years ago, prior to reproducible builds at least, sometimes we had .1 and .2 releases just for android-specific bugs.)

electrum/electrum/version.py

Lines 1 to 2 in 9a5edfd

	ELECTRUM_VERSION = '4.5.5' # version of the client package
	APK_VERSION = '4.5.5.0' # read by buildozer.spec

We could have APK_VERSION = ELECTRUM_VERSION, however it is not that easy, as it also affects the android versionCode.
python-for-android is used to generateversionCode from this version string, see e.g. SomberNight/python-for-android@edb7e4f

Maybe we could pass in the versionCode ourselves via an environment variable (APP_ANDROID_NUMERIC_VERSION might work, but have to double-check what buildozer expects) in make_apk.sh. We would need to calculate it from ELECTRUM_VERSION and the arch.

[thecockatiel]

hmmm
I think it would be fine if the website only do the redirect mentioned (if not hard), then we probably don't have to make any other change
but it should be guaranteed from now on that the trailing digit does not change anymore (so that the f-droid builds don't fail)

[thecockatiel]

and I think as you pointed out dropping the trailing 0 would cause the generated versionCode number to be less than previous builds, and needs manual generation with taking into account the dropped 0

[thecockatiel]

I have opened the MR: https://gitlab.com/fdroid/fdroiddata/-/merge_requests/15858

[thecockatiel]

F-droid has requested that the arch number to be put at the end of the version code:
Per arch number should be at the end of the versionCode, the metadata is sorted by versionCode, eg. Good = 40505002, Bad = 24050500

but it makes me wonder if its safe to do so at this point

since the major version is at 4, I guess we need to bump it to 5 to be fine (as x86_64 is probably at 44050500, we need to go higher than that)

[SomberNight]

F-droid has requested that the arch number to be put at the end of the version code:
Per arch number should be at the end of the versionCode, the metadata is sorted by versionCode, eg. Good = 40505002, Bad = 24050500

Ok, so that's another reason to change the versionCode calculation (in addition to dropping the .0 suffix). I will do that.

[thecockatiel]

hmm,
@SomberNight can you add create a branch on top of 4.5.5 and add the metadata commits to it, so we can reference the commit for releasing 4.5.5 ?
otherwise we have to release 4.5.6 and update the merge request

[SomberNight]

can you add create a branch on top of 4.5.5 and add the metadata commits to it, so we can reference the commit for releasing 4.5.5 ?

I don't know if it's still possible to reproduce the same binary as 4.5.5 if we include a commit such as 6b71367.

I will try to cherry-pick stuff and then have a 4.5.6 released.

[thecockatiel]

hmm, with exclusion commit and fastlane metadata on top of 4.5.5 tag, its possible to get the same release I believe, (build spec is excluded from the build so I don't think it affects the content)
But I guess waiting for 4.5.6 release is probably a better idea if it's around the corner, as we can go and enable the auto update on the MR (so we don't need to make another MR once 4.5.6 releases)

[SomberNight]

Ok, well, if you want to try, and in case 4.5.6 takes longer, I pushed a branch now: https://github.com/spesmilo/electrum/commits/4.5.5-fdroid/

[thecockatiel]

Uh. I completely forgot about SOURCE_DATE_EPOCH in makefile. can you make an amend with a new commit with the date of the one on 4.5.5 ? is it possible ?

[thecockatiel]

Also, I'm not sure, but maybe it's a good idea to fix the SOURCE_DATE_EPOCH on future releases to a specific date ?

[thecockatiel]

oh okay, we can just checkout 4.5.5 in prebuild it seems. you don't need to do anything.

[SomberNight]

Uh. I completely forgot about SOURCE_DATE_EPOCH in makefile. can you make an amend with a new commit with the date of the one on 4.5.5 ? is it possible ?
Also, I'm not sure, but maybe it's a good idea to fix the SOURCE_DATE_EPOCH on future releases to a specific date ?

I thought about this a bit now.
The problem is that, as the comment there says, p4a's logic determining whether to update the code running on a device depends on the version+timestamp. A hardcoded timestamp would mean that if someone built their own apk in release mode, they would not be able to upgrade unless they manually change either the version number or the hardcoded timestamp. I think this has the potential for the occasional major confusion (even for the people actively working on the project).

Maybe we could change p4a not to rely on the timestamp when deciding whether to unpack new files. I guess the naive way would be to have it hash the archive that it is considering to unpack, and use that hash as basis of comparison (instead of version+timestamp). I imagine they are not doing this because it is expensive to read and hash the whole archive on every app start. But maybe there is a faster alternative, e.g. it might be possible to grab a crc32 checksum in constant time (if it's stored as an attribute of the archive) and use that for comparison.

[thecockatiel]

I see
the current way is not that bad after I gave it another thought. its just a little bit surprising when you forget about it
personally I have no idea about how your suggestion would work, but the current way seems to be very efficient

and we wouldn't need to do stuff like this after the f-droid build is stable, and everyone trying to reproduce will always checkout the tag, so yeah I guess It's okay

[thecockatiel]

Also I think as we no longer need to change the url, I'll close this issue