FOSS Version

[shtrophic]

Hi, I wanted to raise some awareness for some objectives that the Android-FOSS community would enjoy seeing. Those are:

1. Support for ntfy-push. This should count as the greatest FOSS-priority.
2. Non-Fdroid FOSS releases, preferrably on GitHub
3. Split APKs

About Nr. 2: While Element-Android has github releases that one can use, they only provide the playstore version of the app. Furthermore, the version available on fdroid often lags behind playstore by a significant timespan. You might make the argument that fdroid only receives stable versions of the app after being tested by playstore users; However for pre-releases, github already provides the necessary facilities.

I hope this could gain some traction in the early development of Element-X.

[bmarty]

Hello,

This is the tasks to achieve to be able to publish on F-Droid:

• Add gradle flavorDimensions store with two values gplay and fdroid
• Update all the CI script so that it understand this new flavorDimensions and build both variant
• Assign value to BuildMeta.flavorDescription and BuildMeta.flavorShortDescription (here)
• Bug report: uncomment builder.addFormDataPart("label", buildMeta.flavorDescription) (here)
• Make sure the fdroid variant does not include the module pushproviders.firebase
• Make sure the fdroid variant configure properly the unified push provider. UnifiedPush support has been implemented and has to be used at this point. Note: if there is no installed distributors, no pusher will be registered. There is no UI to select a push provider and a distributor for now, this is out of scope of this issue.
• When it's ready, ping the F-Droid team to make them build the app and publish it.
• Check the user agent value. Maybe use a different one depending on the flavor.

Already done:

• The store assets are already committed in the project, this must be used to publish to the F-Droid store. (English only for now)

In the long term: Element X Android codebase will be used to upgrade Element Android (the existing app). So at some point this work has to be done, to be able to release the new Element Android to F-Droid

[licaon-kter]

Current status is rather clean ( ahem https://gitlab.com/fdroid/rfp/-/issues/2556#note_1593343486 ), I mean one rm and 3 seds, but yeah, a flavor would be nicer so we don't need to hunt down future changes 😉

[Mynacol]

I'd like to encourage you to try to realize reproducible builds. It eliminates trust in F-Droid and ensures you are in control of the published software. It avoids issues when end users are using multiple stores for updates, but the same app is signed with different keys (assuming you'd/can use the same key for the play store and F-Droid variants). Publishing to F-Droid first non-reproducible and later turning on reproducible builds has its drawbacks, a cleaner approach would be to publish reproducibly from the start.

For that you have to add the required code changes (new flavor, removed unfree dependencies, only/additionally building split APKs) and publishing those APKs somewhere (GitHub releases?). And maybe some compilation adjustments to make it reproducible.

[licaon-kter]

@Mynacol think you are missing some context.

If this app is repro, great.

But this app will die after a while and current app (non X) will be replaced with this one.

Yet, to have that reproducible then, all the current users will need to uninstall & reinstall.

There's no UI that informs as such and guide the users, at least not yet.

The TWIF (this week in fdroid) blog posts are used to inform about such changes but not sure everybody is RSS subscribed to that :)

[K4LCIFER]

It looks like Element X is to be published to F-Droid, with support for UnifiedPush, for the release of v0.4.2:

Next release 0.4.2 should also be published on F-Droid with push support using UnifiedPush.

Related PR: #2248

[Guanran928]

@linsui told me they (fdroid) are still waiting for Element team to publish FOSS flavor's apk, so they can do reproducible build.

https://matrix.to/#/!JqpjuzzVfwdybKQWQg:f-droid.org/$Z2CIcmMm-uvxvnp1keBP7DADRsuPAStDG5eWEC8Y3E4?via=mozilla.org&via=t2bot.io&via=matrix.org

[licaon-kter]

@K4LCIFER read this thread at least https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13829#note_1595843143 maybe https://gitlab.com/fdroid/rfp/-/issues/2556 too

[Mek101]

Only another 20 thousand years...

[SamyDjemai]

Hi @bmarty, from what I see in LoggedInPresenter.kt and in my tests, the UnifiedPush provider seems to be properly working now. Are there any other steps we can help you with before pinging the F-Droid team? :)

[licaon-kter]

@SamyDjemai ping who about what? https://gitlab.com/fdroid/fdroiddata/-/merge_requests/13829

[SamyDjemai]

@licaon-kter I was referencing this message which seems to be outdated: #1418 (comment)
I've only now noticed that builds were already in progress on the F-Droid side 😅 Any blockers there? I'm noticing that the checkupdates GitLab CI job is failing, would we need to change something in this repo to comply?

[licaon-kter]

Expand the threads and read F-Droid feedback and ara4n's answers, that's the status I guess...

would we need to change something in this repo to comply?

yes, a place for F-Droid to grep the versionName and versionCode

[SamyDjemai]

I think that instead of hardcoding the versionName and versionCode in this repo, we could use UpdateCheckData in HTTP mode and scrape the GitHub API for latest releases.
As an example, we could have:

UpdateCheckMode: HTTP
UpdateCheckData: 'https://api.github.com/repos/element-hq/element-x-android/releases?per_page=1|"name": "(\d+)|.|"tag_name": "v((?:[0-9]\d*)\.(?:[0-9]\d*)\.(?:[0-9]\d*))"'

This way, the release process stays unchanged, and we can dynamically fetch versionCode and versionName from the latest GitHub release

[licaon-kter]

@SamyDjemai that line yields ruamel.yaml.scanner.ScannerError: mapping values are not allowed in this context so it might need some tweaking :)

[jspricke]

This works for me:

UpdateCheckData: 'https://api.github.com/repos/element-hq/element-x-android/releases?per_page=1|"name":"(\d+)|.|"tag_name":"v((?:[0-9]\d*)\.(?:[0-9]\d*)\.(?:[0-9]\d*))"'

[SamyDjemai]

I've also reworked mine with simpler matching :)

UpdateCheckMode: HTTP
UpdateCheckData: https://api.github.com/repos/element-hq/element-x-android/releases/latest|"name":"(\d+)-|.|"tag_name":"v(\d+\.\d+\.\d+)"

I'd suggest using https://api.github.com/repos/element-hq/element-x-android/releases/latest instead of using the ?per_page=1.

I'm getting this error:

2024-03-22 12:32:41,036 ERROR: ...checkupdate failed for io.element.android.x : current version is newer: old vercode=40004064, new vercode=40004060

@licaon-kter would we track specific commits of the repo, or only releases? If we do the latter, we should clean up the Builds section of the manifest file

[licaon-kter]

I'm getting this error:

I've fixed that, let me test the rest

would we track specific commits of the repo, or only releases?

we usually get a Tag on checkupdate and use the commit hash

but, with HTTP we can't get commit...

[SamyDjemai]

What if we use AutoUpdateMode: Version v%v? According to this documentation, we can directly use the version tag, and we don't need the commit hash

[licaon-kter]

there's no v in versionName: https://github.com/element-hq/element-x-android/blob/v0.4.6/plugins/src/main/kotlin/Versions.kt#L63

this works if we just ignore versionName (high goal: https://gitlab.com/fdroid/fdroidserver/-/issues/1047) or if upstream has Tag = versionName

[SamyDjemai]

But the Git tags are in the vX.X.X format, hence the need for v%v if I'm not mistaken, since according to the doc:

The pattern is used to generate a value (tag name) used for the commit: property of new build blocks. It is simply text in which %v and %c are replaced with the required version name and version code respectively. The resulting string must match an existing tag in the app’s repo, which then will be used by F-Droid to build the corresponding version.

[licaon-kter]

yup that does it

...
AutoUpdateMode: Version v%v
UpdateCheckMode: HTTP
VercodeOperation:
  - 1 + %c
  - 2 + %c
  - 3 + %c
  - 4 + %c
UpdateCheckData: https://api.github.com/repos/element-hq/element-x-android/releases/latest|"name":"(\d+)-|.|"tag_name":"v(\d+\.\d+\.\d+)"
CurrentVersion: 0.4.6
CurrentVersionCode: 40004064

[SamyDjemai]

Great, thanks! From what I've assessed, in order to be compliant with F-Droid's inclusion policy, we still need to do the following:

• build F-Droid version without proprietary components
  • Sentry: remove services/analyticsproviders/sentry in the F-Droid manifest Sentry SDK is open source
  • Mapbox: Maplibre is used
• get approval from the upstream team for F-Droid release (clear up the existing confusion regarding release)

@licaon-kter do you see any other blockers? do we want reproducible builds?
@bmarty is the Element X team OK for F-Droid release?

[licaon-kter]

build F-Droid version without proprietary components

current recipe does that

Sentry: remove services/analyticsproviders/sentry in the F-Droid manifestJ

not a NEED, if it's on by default or opt-out, it will get a Tracking anti-feature, if it's off by default and opt-in, it will get nothing

get approval from the upstream team for F-Droid release (clear up the existing confusion regarding release)

mostly this yes

reproducible builds

the big IF

[licaon-kter]

Maybe reopen until those 2-3 repro issues get fixed?

[bmarty]

Thanks for the diff @licaon-kter .

• maptiler_api_key
• maptiler_dark_map_id
• maptiler_light_map_id

The values are store as GitHub secret so you do not have it when you build yourself.
We know that the secrets are easy to find, but the idea is to limit the usage of those keys by forks.
Is it possible to provide the value in your build chain? You will need to set those 3 env variables:

• ELEMENT_ANDROID_MAPTILER_API_KEY
• ELEMENT_ANDROID_MAPTILER_LIGHT_MAP_ID
• ELEMENT_ANDROID_MAPTILER_DARK_MAP_ID

For the diff in DefaultVersionFormatter, I think that: buildMeta.gitBranchName == "main" is not true in your build env. The value is equal to "HEAD".

Is it possible that your build chain patch the code here?
So in your case the line will change from

fun Project.gitBranchName() = runCommand("git rev-parse --abbrev-ref HEAD")
to
fun Project.gitBranchName() = "main"

This is the same root cause for the following diffs.

There is also a diff in versionCode. On our build, the value is 40004130 and on your build, the value is 40004132. The last digit is for the device architecture.

We should not use BuildConfig.VERSION_CODE, (since it will not be impacted by this code) but read the value from the Manifest. I'll create a PR about it. I am not sure to understand why there is a diff in the build though, but it should fix the issue. I am still investigating this.

[licaon-kter]

The values are store as GitHub secret so you do not have it when you build yourself.

https://f-droid.org/docs/Inclusion_Policy/

F-Droid does not sign up for any API keys. Even if provided by a third party, we include them in both, binary and sourcecode releases.

they need to be here in your code or in the F-Droid recipe, no secrets allowed

Is it possible that your build chain patch the code here?
So in your case the line will change from

we can do, yes

I am still investigating this.

great

[bmarty]

they need to be here in your code or in the F-Droid recipe, no secrets allowed

Yes, you can put the values in the F-Droid recipe, and set environment variables as described, this is fine (sorry if it was not clear). The 3 values are in the diff you provided.

Alternatively, you can put the values in the local.properties file. The values are read here and it's documented here.

[bmarty]

#2911 should help on the versionCode diff.

[bmarty]

@licaon-kter #2911 has been merged, so I think the plan for us is to publish F-Droid APKs for the next release on Github v0.4.14. This should occur on June 4th.

On your side: change the recipe about the 2 points mentioned above: the map keys and the git branch name.

Hopefully the APKs will have the reproducible build green flag.

[licaon-kter]

I've already tested the map injection fix, will test the testother fixes when the Tag drops. Feel free to ping me.

[licaon-kter]

@bmarty tick-tock, 4-th was gone

[bblacher]

@licaon-kter They just bumped the version, in a few minutes the release will be out. 4b27cda

[ganfra]

@licaon-kter https://github.com/element-hq/element-x-android/releases/tag/v0.4.14
Have fun.

[licaon-kter]

Started a build

First issue though, I can't know the versionCode for autoupdates, we used to pick it up from the APK name in past releases, eg. https://github.com/element-hq/element-x-android/releases/tag/v0.4.13 -> 40004130.apk -> 40004130

but that's not a thing in latest https://github.com/element-hq/element-x-android/releases/tag/v0.4.14

[ganfra]

Ok, can rename the file.

-> Done.

[licaon-kter]

Ok, thanks, the question is, do you need that file or? maybe there's a way to name it so it's usable both by F-Droid and users that drop by (eg. Obtainium etc)

[ganfra]

This is the universal apk for gplay variant. I just renamed it manually so it was clear. The original name is the one provided by the playstore.

[licaon-kter]

Here's the first diff log: elem0414_2.log

this looks like https://f-droid.org/docs/Reproducible_Builds/#embedded-build-paths with some ways to fix it here: https://gist.github.com/obfusk/61046e09cee352ae6dd109911534b12e

not sure if the AndroidManifest.xml will be fixed too once baseline is

[linsui]

Given that we can't get the release files without using the api which is rate limited, can you add a version.txt with version code in it?

[licaon-kter]

This has been merge in F-Droid metadata 🎆

FYI https://gitlab.com/fdroid/wiki/-/wikis/FAQ#how-long-does-it-take-for-my-app-to-show-up-on-website-and-client

/close this

[linsui]

It has been published.

[networkException]

For me it says the signing key has changed, is that expected?

[bmarty]

Thanks! I was actively waiting for https://f-droid.org/packages/io.element.android.x/ to return something.
Let's close the issue, please open new issues in case of trouble.

[licaon-kter]

@networkException if you installed in the past from my builds then that's expected

F-Droid hosts the verified reproducible builds from upstream now.

[licaon-kter]

@bmarty it does now, it takes a bit more after the index is published for the website to be generated

[networkException]

@licaon-kter Should've been from the Google Play previously. Is upstream signing these differently?

[licaon-kter]

@networkException well yes, Google signs in Play, not the developer.... they forced all developers at some point, for new apps since Nov 2021 or so