WIP: Rewrite bootstrap #6719

bgamari · 2020-04-19T19:18:05Z

Given I've complained about this in the past, I felt compelled to pick up #6700. This currently requires a reasonably modern Python (Python 3.5 or newer) but if this is a problem we could likely relax this to Python >= 3.2.

The idea

A JSON file (generated with bootstrap.py --extract-plan) containing dependency and source information is built from plan.json. This is then used to fetch and build the bootstrapping dependencies. Since this is only to be used for bootstrapping on new architectures, we advise the users to only use the resulting cabal-install executable to build a proper cabal-install.

To-do

Figure out how to determine revision information; unfortunately this doesn't appear to be available from plan.json
Figure out story for distributing bootstrap-deps.json; should it be in the repository?
Document usage
Test in CI

phadej

This is good start.

At the end I'd keep generating of bootstrap-deps.json and "runner" as separate scripts, but that can be done later. (I'd also rewrite the generator later in Haskell, there is e.g .cabal-plan library etc.)

cabal-install/bootstrap.py

phadej · 2020-04-20T07:31:44Z

cabal-install/bootstrap.py

+from typing import Set, Optional, Dict, List, Tuple, \
+                   NewType, BinaryIO, NamedTuple
+
+ghc_path = Path('ghc')


One of problems with old bootstrap was that we were somewhat sensitive to the ghc version used. We should use ghc --version info.

Or better we should parse ghc --info and find ghc-pkg using LibDir, as perfectly boostrapping ghc should be configurable (i.e. we wouldn't need to rely on PATH).

Or better we should parse ghc --info and find ghc-pkg using LibDir, as perfectly boostrapping ghc should be configurable (i.e. we wouldn't need to rely on PATH).

This suggestion seems very fragile to me. The ghc-pkg wrapper in bin/ arranges to pass --global-package-db to the ghc-pkg executable in LibDir. We would need to duplicate this logic if we were to use the LibDir executable directly.

Moreover, on some platforms (e.g. NixOS) the executable in PATH is actually a distribution-provided wrapper which sets up other state. IMHO we shouldn't start poking around in LibDir. If anything we should just look for a ghc-pkg in the same directory as we found the ghc executable.

It actually turns out the that the --global-package-db flag that I noted above is added by the NixOS wrapper. Nevertheless, the point stands.

I'm sad to learn about this.

The simple heuristics don't work:

% $(dirname $(which ghc-8.10.1))/ghc-pkg --version GHC package manager version 8.8.3

we have better heuristics for these, but they don't work if suffix is arbitrary. I'd rather not to use heuristics, and for special (and I consider NixOS special) situations users would rather required to be explict.

NixOS probably uses wrapper because --global-package-db because they fake global installs, not actually do that. That doesn't affect bootstrap.sh because it doesn't use "extras".

cabal-install/bootstrap.py

23Skidoo · 2020-04-21T03:47:18Z

bootstrap.sh is also supposed to work when it's run from a Git checkout, I guess that this change means dropping that feature? Overall I'm +1 on making it more robust.

bgamari · 2020-04-22T14:44:57Z

bootstrap.sh is also supposed to work when it's run from a Git checkout, I guess that this change means dropping that feature?

Not necessarily. We could decide that we want to include a bootstrap dependencies description for a couple of supported bootstrap compilers in the repository. I'll leave this decision to you and @phadej.

bgamari · 2020-05-14T01:43:58Z

We still need to sort out if/how we want to package the build dependency sets in the tree.

bgamari · 2020-05-14T03:36:07Z

I cannot tell what the meta check is worried about.

phadej

I think this is a step into right direction.

cabal-install/bootstrap.py

bgamari · 2020-05-18T22:51:46Z

Addressed comments.

@phadej, how would you like to handle merging this? Should I drop the bootstrap-deps.json files and then we can reintroduce them in the release branch?

The plan: A JSON file (generated with bootstrap.py --extract-plan) containing dependency and source information is built from plan.json. This is then used to fetch and build the bootstrapping dependencies. Since this is only to be used for bootstrapping on new architectures, we advise the users to only use the resulting cabal-install executable to build a proper cabal-install.

bgamari · 2020-05-31T16:46:59Z

Ping @phadej.

blackgnezdo · 2020-06-28T21:26:35Z

@phadej, I'd love to use this for building the OpenBSD port. It would be awesome to merge before 3.4.

blackgnezdo · 2020-06-29T04:48:56Z

Quick observation: after rebasing the boostrap*json files have to be regenerated due to new dependencies on regex-base and regex-posix at HEAD (e.g. blackgnezdo@a4daad2)

phadej · 2020-07-20T11:32:38Z

I'm looking at this.

Few notes immediately, we have to record flag selection.
I have different cabal dir then $HOME/.cabal, so script doesn't work.

Both are easy enough to fix, and I'll do that.

phadej · 2020-07-21T10:37:05Z

Continued in #6979

bgamari force-pushed the rewrite-bootstrap branch 2 times, most recently from a37a342 to 08cb976 Compare April 19, 2020 19:23

bgamari mentioned this pull request Apr 19, 2020

Rewrite bootstrap.sh #6700

Closed

bgamari force-pushed the rewrite-bootstrap branch 3 times, most recently from ec42627 to 513356f Compare April 19, 2020 19:44

phadej reviewed Apr 20, 2020

View reviewed changes

bgamari mentioned this pull request Apr 20, 2020

What is purpose of bootstrap.sh? #6701

Closed

phadej mentioned this pull request May 6, 2020

Release 3.4 planning #6757

Closed

bgamari force-pushed the rewrite-bootstrap branch from 513356f to c2c6f8c Compare May 14, 2020 01:32

bgamari force-pushed the rewrite-bootstrap branch from ea93f10 to 0b4a20a Compare May 14, 2020 03:29

phadej approved these changes May 15, 2020

View reviewed changes

cabal-install/bootstrap.py Outdated Show resolved Hide resolved

blackgnezdo reviewed May 17, 2020

View reviewed changes

cabal-install/bootstrap.py Show resolved Hide resolved

blackgnezdo reviewed May 17, 2020

View reviewed changes

cabal-install/bootstrap.py Outdated Show resolved Hide resolved

bgamari added 3 commits May 20, 2020 20:43

Add bootstrap dependency metadata

e3287f3

Update CI for bootstrapping

12ab32d

bgamari force-pushed the rewrite-bootstrap branch from f79f951 to 12ab32d Compare May 21, 2020 00:48

phadej self-assigned this Jul 20, 2020

phadej closed this Jul 21, 2020

phadej mentioned this pull request Jul 21, 2020

More portable sed syntax in cabal-install #6846

Closed

ciez mentioned this pull request Mar 17, 2021

No bootstrap files in Hackage cabal-install tar ball #7289

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Rewrite bootstrap #6719

WIP: Rewrite bootstrap #6719

bgamari commented Apr 19, 2020 •

edited

Loading

phadej left a comment •

edited

Loading

phadej Apr 20, 2020

bgamari Apr 22, 2020

bgamari Apr 22, 2020

phadej Apr 22, 2020

23Skidoo commented Apr 21, 2020

bgamari commented Apr 22, 2020

bgamari commented May 14, 2020

bgamari commented May 14, 2020

phadej left a comment

bgamari commented May 18, 2020

bgamari commented May 31, 2020

blackgnezdo commented Jun 28, 2020

blackgnezdo commented Jun 29, 2020 •

edited

Loading

phadej commented Jul 20, 2020 •

edited

Loading

phadej commented Jul 21, 2020

WIP: Rewrite bootstrap #6719

WIP: Rewrite bootstrap #6719

Conversation

bgamari commented Apr 19, 2020 • edited Loading

The idea

To-do

phadej left a comment • edited Loading

Choose a reason for hiding this comment

phadej Apr 20, 2020

Choose a reason for hiding this comment

bgamari Apr 22, 2020

Choose a reason for hiding this comment

bgamari Apr 22, 2020

Choose a reason for hiding this comment

phadej Apr 22, 2020

Choose a reason for hiding this comment

23Skidoo commented Apr 21, 2020

bgamari commented Apr 22, 2020

bgamari commented May 14, 2020

bgamari commented May 14, 2020

phadej left a comment

Choose a reason for hiding this comment

bgamari commented May 18, 2020

bgamari commented May 31, 2020

blackgnezdo commented Jun 28, 2020

blackgnezdo commented Jun 29, 2020 • edited Loading

phadej commented Jul 20, 2020 • edited Loading

phadej commented Jul 21, 2020

bgamari commented Apr 19, 2020 •

edited

Loading

phadej left a comment •

edited

Loading

blackgnezdo commented Jun 29, 2020 •

edited

Loading

phadej commented Jul 20, 2020 •

edited

Loading