Reduce the number of checked in generated files

[robertbastian]

We have quite a few generated files checked in, with CI failing if they are out of date. This happens to me quite a lot, and it's a waste of my time and CI resources (a new commit has to run everything again).

Some files very cheap to generate on the fly (gn-gen and diplomat-gen), while some like testdata are not. Can we remove the cheap files?

[sffc]

Part of the reason I added the GN files to the repo is so that clients can view and download them directly from the repo without needing to install the tools themselves (which, although fast to run, are not particularly cheap to install). Another way to meet this end is to upload zip files containing the GN rules to certain tagged releases. I feel like it's a tradeoff between development velocity and release velocity.

[sffc]

Let's wait a few weeks to see exactly how Skia/Flutter is going to be downloading these GN files.

[sffc]

Discuss with:

• @robertbastian
• @Manishearth
• @sffc

[sffc]

I'm okay removing the GN files from the repo so long as the CI still generates and tests them on the fly. Bonus points if CI uploads the GN file as an artifact or at least dumps it to stdout.

[sffc]

Actually @Manishearth does Julia use our checked-in GN files?

[Manishearth]

They were, but I think they're switching to bazel now? I'm not fully sure.

[sffc]

GN-Gen

• @Manishearth We could remove the GN files normally but then add them back at release
• @robertbastian I think we don't really need to test these files on main; we could test them only at release time, like the cargo tutorials
• @Manishearth It seems the GN build is currently broken on nightly; the dependencies changed order. It can't be fixed without checking in twice the number of files. Adding back the genewrated files at release time is an easy change to make.
• @sffc What's wrong with keeping it as /ffi/gn since there is other content under /ffi like /ffi/npm?
• @robertbastian Because /ffi/npm is a package we plan to release
• @sffc I'm concerned about removing the Lockfile because it causes less stability in CI
• @Manishearth It seems useful to know and report back issues with Lockfile stability. Our users do not use our lockfiles, we should not be handcrafting perfect lockfiles for CI in the first place.
• @robertbastian Why do we need it in CI?
• @sffc GNaw has strange requirements about dependencies and proc macros which seem useful to test that we don't break. I'm okay running it on nightly CI or PR CI. We could keep it for now on PR CI and move it to nightly CI if it is too flaky without the Lockfile.

Proposal:

1. .gitignore all generated files in /ffi/gn which we believe to be icu4x/BUILD.gn and Cargo.lock
2. Move /ffi/gn to /docs/tutorials/gn
3. Continue running GN in CI

LGTM: @Manishearth @sffc @robertbastian

Diplomat-Gen

• @robertbastian I take this back, it's actually very convenient to have diffs for diplomat generated files
• @Manisearth GN-Gen has a diff when you touch any Cargo.toml file, whenever you add a dependency or anything like that. When you touch Cargo.toml, you aren't thinking about GN. However, when you touch Diplomat source files, you are intentionally changing Diplomat so therefore it is expected that there could be a diff in the Diplomat output files.
• @robertbastian Also the generated bindings are something we actually ship, so they should be there
• @Manishearth It's not great to publish bindings to like npm or whatever and when ecosystem members go look at the repo it shows them a shell script that generates the thing they're using.
• @robertbastian The directory structure has generated bindings for each language, and the npm package needs to copy the files into its own directory before it builds. I made a symlink in a recent PR which may have gone in unnoticed.
• @Manishearth I'm fine with the actual publish being a shell script, as long as the source code can be found somewhere in the repo
• @robertbastian Without the symlink, you can't run npm tests without running the copy command first

Proposal:

1. Don't change the set of checked-in files for Diplomat
2. Keep the symlink for now until it causes problems

LGTM: @sffc @Manishearth @robertbastian

[sffc]

@sffc to make the change for GN and add documentation as needed

[sffc]

I tried removing the lockfile but immediately hit the following error:

Generating GN file from /home/sffc/projects/icu4x/ffi/gn/Cargo.toml
Error: generating manifest

Caused by:
    GNaw config exists for crates that were not found in the Cargo build graph:
    
    library crate, package serde_derive version 1.0.160
    library crate, package memchr version 2.5.0
    library crate, package libm version 0.2.6
    library crate, package serde version 1.0.160
    library crate, package proc-macro2 version 1.0.63
    library crate, package quote version 1.0.26

It is because the Cargo.toml contains version-specific overrides for our dep tree, and refreshing the lockfile causes these deps to be pulled as different versions:

[gn.package.memchr."2.5.0"]
rustflags = []

[gn.package.proc-macro2."1.0.63"]
rustflags = ["--cfg=use_proc_macro", "--cfg=wrap_proc_macro", "--cfg=proc_macro_span"]

[gn.package.quote."1.0.26"]
rustflags = []

[gn.package.serde."1.0.160"]
rustflags = []

[gn.package.serde_derive."1.0.160"]
rustflags = []

[gn.package.libm."0.2.6"]

I can try to see if I can use the versions pinned in the top level Cargo.lock file.