Allow building GRDBCipher with SwiftPM

[hartbit]

Pull Request Checklist

• This pull request is submitted against the development branch.
• Inline documentation has been updated.
• README.md or another dedicated guide has been updated.
• Changes are tested.

Changes

• I ran the amalgamation step of the latest SQLCipher release (v4.2.0) and copied the generated header and source file in a new SQLCipher folder.
• I updated the Package.swift manifest:
  • I added a new SQLCipher target, a new GRDBCipher target that depends on it, and a GRDBCipher product that makes it available.
  • I bumped the swift-tools-version version to 5.0 to allow specifying compiler defines necessary for building SQLCipher and for building GRDB with SQLITE_HAS_CODEC.
  • I had to create a GRDBCipher folder that symlinks to GRDB to avoid a current SwiftPM issue (https://bugs.swift.org/browse/SR-10974).
  • I defined the minimum platforms GRDB supports, bumping macOS to 10.10 in the process, as SwiftPM doesn't allow you depending back to 10.9.
• I added missing import Foundation statements in FTS5.swift, FTS5Tokenizer.swif, and FTS5WrapperTokenizer.swift which were missing but necessary.
• I updated the README to explain how to depend on GRDBCipher in SwiftPM.

Issues

The are two remaining issues:

• SQLCipher generates a bunch of warnings when building which I can't silence without dropping Xcode support :-( It seems like the:

.unsafeFlags([
    "-Wno-ambiguous-macro",
    "-Wno-shorten-64-to-32",
    "-Wno-#warnings"
])

configuration necessary to silence the warnings makes is not supported in Xcode (perhaps for security reasons).

I opened a SwiftPM issue regarding this: https://bugs.swift.org/browse/SR-10978

• Several GRDB files generated the following warnings which I didn't quick understand and which might not be related:

<unknown>:0: warning: imported declaration 'UITableViewDiffableDataSourceCellProvider' could not be mapped to 'UITableViewDiffableDataSourceReference.CellProvider'
<unknown>:0: warning: imported declaration 'UICollectionViewDiffableDataSourceCellProvider' could not be mapped to 'UICollectionViewDiffableDataSourceReference.CellProvider'

[groue]

Hello @hartbit, you were fast :-)

I ran the amalgamation step of the latest SQLCipher release (v4.2.0) and copied the generated header and source file in a new SQLCipher folder.

This is my biggest concern. That's a big new 7Mb file.

Shouldn't we export the amalgamation Package in its own repository? For example, GRDB supports custom SQLite builds by depending on http://github.com/swiftlyfalling/SQLiteLib. This is a nice separation of concerns.

I also hope that it makes clear that this is not a regular PR. This one isn't free! Someone will have to maintain SQLCipher, embedded here, or in a GRDB fork, or in its own external repository. Your position on this would be welcome.

I bumped the swift-tools-version version to 5.0 to allow specifying compiler defines necessary for building SQLCipher and for building GRDB with SQLITE_HAS_CODEC.

GRDB still claims support for Swift 4.2 (and tests it). Thanks to Version-specific manifest selection, we can keep the previous Package.swift file, renamed to [email protected].

EDIT: if Swift 4.2 does not support version-specific manifests, then we'll force SPM users to Swift 5. This will be a breaking change, but one I'll be able to live with. I just wish we could avoid it if possible.

SQLCipher generates a bunch of warnings when building which I can't silence

You've done your best effort 😅

[hartbit]

This is my biggest concern. That's a big new 7Mb file.

Do you want to create the repo for SQLCipher? I can send a PR to that repo with everything to have a working Swift Package for SQLCipher.

I also hope that it makes clear that this is not a regular PR. This one isn't free! Someone will have to maintain SQLCipher, embedded here, or in a GRDB fork, or in its own external repository. Your position on this would be welcome.

I know :-( What I can do is create a script in the SQLCipher repo to simplify the process of fetching a specific version of SQLCipher, amalgamating it, and creating a new tagged version. This will at least make the process easy enough that we will be more inclined to keep it up to date.

GRDB still claims support for Swift 4.2 (and tests it). Thanks to Version-specific manifest selection, we can keep the previous Package.swift file, renamed to [email protected].

I'll do that change.

You've done your best effort 😅

I'll send a PR to SwiftPM to support an .silenceWarning option, but I don't know if it will make it into 5.1 or not.

[groue]

Hi again @hartbit, I've been busy shipping v4.1 so I could not address your last message.

And I'll postpone my answer again, because I'm leaving in vacations.

So let's put on hold this topic for a few weeks. I hope you are not blocked, and that your own explorations run smoothly!

[hartbit]

Hi @groue. No rush! Have a great vacation 🏝

[groue]

Back from vacations, but still postponing a bit. I can't help but thinking we're fighting with SPM. Nah, I'm preparing v4.2, so I'm all in my "strict phase". I'll soon be more open to experiments.

[hartbit]

We'll be fighting SPM until it supports running pre-build scripts, which we need for the amalgamation step. I'm just putting myself in the shoes of a GRDB user, where SPM in Xcode is now the easiest way to get started: It'd be a pity if the user had to abandon SPM and migrate to Cocoapods when they need to encrypt their database.

[groue]

Yes, David, you are totally right. Consider me sold.

[groue]

Hello @hartbit!

After a few weeks of vacations, and the release of v4.2, I'm ready to keep on discussing with you and find a way forward :-)

Let's start with a summary of our previous conversations:

• Yes, we want to be able to distribute GRDB+SQLCipher with SPM
• A way to expose SQLCipher to SPM is to define a package target that contains a previously built "amalgamation" C file. This file can be built from the raw SQLCipher sources and a shell script.
• I was reluctant to embed the amalgamation right into the GRDB repo because of its huge size.
• I was wonderig how we can secure the maintenance of the SQLCipher SPM target on the long run.

---

Let me propose the draft of a plan, and tell me how it sounds to your ears. I'll not make any assumption about who does what, because it's all too early yet. Scheduling and ETA are also out of scope.

Note In an ideal world, the SQLCipher SPM package we need would be hosted by Zetetic, the authors of SQLCipher. Shouldn't we try to bring them in this discussion? I assume we are "alone" in the rest of this message.

• Host somewhere (more on that below) a SQLCipher package that contains the SQLCipher amalgation.

  The following line looks like it builds the amalgamation from the raw SQLCipher sources: https://github.com/sqlcipher/sqlcipher/blob/v4.2.0/SQLCipher.podspec.json#L14

  In order to maintain this package, we'd need a recipe which tells how to ship a new release. For example:

  1. Download latest SQLCipher sources
  2. Run some_script
  3. Commit and tag with some version
  4. Push

  It is better if this recipe can be scripted, since it would become a no-brainer for anyone with push access to the repo.

2. Define a Github organization that would host GRDB-related repositories.

   For example... https://github.com/GRDB (ha, no, it is not available).

   This organization would host the SQLCipher package we need (see below).

   Existing repositories, GRDBCombine, CSQLite, GRDBObjc, and eventually GRDB itself would eventually move to this organization. But we can start just with SQLCipher.

   Disadvantage: URL changes are breaking changes. When other repositories will move, we'd need to bump major versions.

   But in the long run, there is a big advantage: this would blur my "ownership" of this ecosystem.

   It would make it much more natural that some of those repositories are maintained by the people that need it the most. My personal list of things to do on a release would stop growing. I wouldn't have to ask you whether you personally want to host the SQLCipher package under your Github account. But it would be possible to grant you all my privileges so that you don't need me when SQLCipher ships the feature or the bugfix you need.

   With the advent of Combine, my interest in RxGRDB will certainly fade out. Since I barely never use SQLCipher, I do not know it very closely. The last important SQLCipher-related evolutions of GRDB were all brought by users (Add additional SQLCipher configuration options #497, Allow Database Connection Configuration #508). What's common between RxGRDB and SQLCipher? Well, I don't lack energy, and I am able to prepare my releases the best I can. But I may not be able to maintain high quality APIs for features I don't use, or new features that I'm not even aware of, as those libraries evolve. Decoupling is good. Empowering experts in domains I do not master is good.

3. Setup the ability to use GRDB and SQLCipher with SPM, and be able to run tests.

   I don't know precisely what it means, because I don't know much about SPM yet. I guess this will change with this project ;-)

   And I don't even know which tests we'd need to run.

   Current tests of GRDB+SQLCipher+CocoaPods run the full test suite four times. Once with clear-text databases, once with encrypted databases. Repeat for SQLCipher 3 and 4. See the test_framework_SQLCipher in the Makefile. This is not a luxury, and it does test a great deal.

   Do we need more?

   Anyway, those eventual tests would be hosted in the main GRDB repo, even it makes it difficult to test the evolution of the SQLCipher package repo itself. This is because the natural place for the SQLCipher package is not under the GRDB umbrella, but under Zetetic's.

4. Document. Announce. Rest.

Does it sound like a correct direction to you? My wish is that nothing would prevent you from jumping on the board, for the duration you want. I need you, here.

[hartbit]

I've been on vacation the last few weeks. I'll reply ASAP.

[groue]

Closing due to lack of activity

[jcavar]

Hello, I was wondering what is the status of this issue?
Is there a recommended way to install GRDB + SQLCipher using SPM?

[groue]

Hello @jcavar. No, there isn't any recommended way to install GRDB + SQLCipher with SPM. It's up to you to build up a solution. If you do so, please share the results of your work!

[jzzocc]

Since @groue suggested we should share the results of our work if we have a solution, I'll share my makeshift one for using SQLCipher with GRDB installed via SPM. You can build an amalgamation yourself (using the instructions at https://www.zetetic.net/sqlcipher/ios-tutorial/) and fork and modify GRDB to use it instead of CSQLite (as seen at metabolist@ea3ed26).

I would recommend doing this yourself and not using my fork as I may modify or delete it without notice.

I've opened an issue with SQLCipher asking if they'd be open to maintaining an official SPM package so this would not be necessary in the future.