Add show-build-info command #2771

bgamari · 2015-08-16T21:53:42Z

Interaction with Cabal poses a few challenges to third-party tooling authors,

The interfaces are rather unstable and tend to be verbose
Some features are provided instead by cabal-install, which currently has no publicly accessible programmatic interfaces
Linking against Cabal directly has unfortunate interactions with cabal-install in the presence of upgrades

While (1) can be addressed with library support easing common use-casese (see, for instance, https://github.com/bgamari/cabal-ghc-dynflags), (2) and (3) pose severe impediments to tooling development. This patch seeks to address this by removing the need to link against Cabal to retrieve build information. It does this by introducing the show-build-info command, which produces a small JSON document describing the compiler arguments Cabal would use to build the various components of the package. For instance,

{ "cabal_version": "1.23.0.0",
  "compiler": {
    "flavor": "GHC",
    "compiler_id": "ghc-7.10.2",
    "path": "/usr/bin/ghc",
  },
  "components": [
    { "type": "library",
      "name": "CLibName",
      "compiler_args":
        ["-O", "-XHaskell98", "-Wall",
         "-package-id", "parallel-3.2.0.6-b79c38c5c25fff77f3ea7271851879eb"]
      "modules": ["Project.ModA", "Project.ModB", "Paths_project"],
      "source_files": [],
      "source_dirs": ["src"]
    }
  ]
}

The goal here is not to replace the Cabal library as the primary means of querying package information. Rather, this command is supposed to target the narrow but important use-case of querying for build information, only providing enough information to allow a third-party tool to reconstruct the compiler configuration and relevant modules without having to link against Cabal. The format is small with the goal of making it relatively easy to ensure compatibility going forwards.

bgamari · 2015-08-16T21:55:27Z

Bringing in @edsko, who provided the motivation for this feature.

bgamari · 2015-08-16T22:00:03Z

At the moment I think it is fair to say that this is a bit rough still. For instance, note the fact that Paths_project in the modules list above is generated by Cabal and therefore will not be found in any of the directories listed in source_dirs. Perhaps the solution here is just to eliminate source_dirs entirely (although I can envision a variety of uses for it).

23Skidoo · 2015-08-16T22:06:55Z

I guess that this needs a corresponding user guide update? And please also update the changelog. Code looks OK on a first glance.

May be worth making show-build-info a hidden command, since it's not something that ordinary users care about.

/cc @dcoutts

bgamari · 2015-08-16T22:22:01Z

Yes, absolutely. Really I would first like to get some feedback from other tooling authors. I know what I need but others may want information beyond what I provide here.

/cc @alanz @kazu-yamamoto

23Skidoo · 2015-08-16T22:39:15Z

@bgamari There's also a Travis failure.

ttuegel · 2015-08-16T22:46:06Z

cabal-install/Main.hs

+  maybeWithSandboxDirOnSearchPath useSandbox $
+    showBuildInfo verbosity config distPref buildFlags extraArgs
+
+-- | Actually do the work of building the package. This is separate from


This comment was probably copied from somewhere else and needs to be updated.

Ugh, thanks for catching that.

ttuegel · 2015-08-16T22:50:21Z

Except for the comment and the Travis failure, LGTM.

This allows users to get a JSON representation of various information about how Cabal would go about building a package. The output of this command is intended for external tools and therefore the format should remain stable.

bgamari · 2015-08-17T07:10:43Z

Brining in @GanielG as well.

DanielG · 2015-08-17T08:06:16Z

Who's @GanielG? :P

I fully support this in theory but just getting the complete set of renered GHC options turned out not to be sufficient for ghc-mod. I had to extract various subsets of options too: https://github.com/DanielG/cabal-helper/blob/master/CabalHelper/Main.hs#L95

bgamari · 2015-08-17T08:27:05Z

Oh dear, it's apparently still early in the morning.

Indeed you do scrape a lot out of Cabal. My original goal was to avoid reimplementing a JSON representation of the Cabal API and instead just focus on those cases which are complicated by linking considerations (e.g. user installs tool linked against Cabal-1.x.0, later updates cabal-install linking against Cabal-1.y.0; now every time the user switches between running tool and cabal-install the build information will need to be recomputed).

That was merely my original idea, however, and I'm not opposed to adding more surface area to the interface so long as we can do so in a way that is unlikely to pose compatibility issues going forward. Could you offer some more details on precisely what subsets of flags are needed and why?

My understanding is that there are a few classes of options that should be reasonably easy to isolate,

Language flags (e.g. -X...)
Package flags (e.g. -package-id)
Paths (e.g. -I, -i)
Perhaps CPP options (e.g. -DMACRO flags)

To get any more specific then this begins to wear away at Cabal's (perhaps not unnecessary) compiler independence. I'm not sure whether we want to enter this territory (@23Skidoo, thoughts?).

edsko · 2015-08-17T08:35:09Z

Yes yes yes. +1.

edsko · 2015-08-17T08:37:15Z

@chrisdone / @mgsloan I don't know if you guys still read the cabal data in stack-ide, but if you do, this may be relevant to you guys as well.

DanielG · 2015-08-17T08:43:38Z

write-autogen-files

Just calls writeAutogenFiles since cabal configure doesn't do that and
cabal build is too heavy.
ghc-options

Pretty much what you already do.
ghc-src-options

Only the search path related options so I can run module graph resolution
without also pulling in the package database and all the other stuff.
ghc-pkg-options

Only -package-db and -package flags. For ghc-mod commands like browse that
only need the package database.
ghc-merged-pkg-options

Like above but mconcat'ed together for contexts where we want the entire
project's dependencies in scope. It's a bit of a hack but would be much worse
to do at the flag level.
ghc-lang-options

-X flags essentially
package-db-stack

withPackageDB from LocalBuildInfo, we used to just resolve sandboxes
ourselfs but getting that this way turned out to make supporting stack much
easier and just seems like a better way to get that anyways.
entrypoints

List of components and source files / other- and exposed-modules
see componentEntrypoints in the cabal-helper source.

The other commands should be obvious.

23Skidoo · 2015-09-22T23:09:30Z

@bgamari I think this PR can go in since it's low-risk and marked as unstable anyway. Do you want me to merge it in the current form or do you think it still needs more work?

DanielG · 2015-09-23T04:26:11Z

I would still like to see the subsets of options added to the interface before this gets merged. Otherwise I'd not be able to use it in ghc-mod which would be a pitty. It would also be nice if I could more or less implement all the stuff in cabal-helper using this interface so we can have a smooth transition from current cabal-install versions to ones that have support for this natively.

I might be able to find some time to work on this unless @bgamari wants to do it. But my schedule is pretty packed so I'm not sure when that would be.

ghost · 2017-05-14T08:38:55Z

Cabal/Distribution/Simple/Utils/Json.hs

+  where
+    go []     = id
+    go [x]    = x
+    go (x:xs) = x . showString sep . go xs


(Just a casual observer, not a contributor) I assume the intention is to avoid adding another dependency to Cabal, but this still feels like a candidate for a separate library, even if it is a small library that is a minimal subset of Aeson... cabal-aeson-lite or something. I am sure others have already thought this through though..

@kanishka-azimi This part of the patch is actually going to vanish anyway, since we already have a aeson-ish mini library in cabal-install (which generates plan.json). But more generally, there is IMO no point to release a small trivial mini-aeson library unless somebody other than cabal wants to actually use it. But I don't see this happening, as aeson is so ubiquitous that virtually everyone just uses aeson if they need JSON capabilities. Last but not least, there's a significant cost involved in maintaining a library on Hackage in a responsible manner (writing proper documentation, testing, keeping it working with updated dependencies and compilers, being careful about API breakges, reacting to feature requests and bug reports etc.).

Since I wrote that comment I ended up releasing an extended version of my llightweight JSON library as https://hackage.haskell.org/package/microaeson as I ended up needing a dependency-minimal JSON library...

In any case, this PR ought to get rid of the redundant JSON impl in favor of the one we're using already for plan.json.

23Skidoo · 2017-06-11T14:16:44Z

@robinp BTW, you might be interesting in reviving this PR for haskell-indexer. The idea here is to add an easier way to integrate "processors" with Cabal, so that new ones could be added with zero/minimal changes to Cabal code.

robinp · 2017-06-11T14:20:37Z

@23Skidoo thank you, will consider. For now a GHC frontend plugin might work, but for long-term when we need to capture the precise inputs (for later compilation) this looks required.

merijn · 2017-07-07T09:35:37Z

So, essentially the only thing blocking this from being merged is the option subsets DanielG's described?

gbaz · 2018-04-09T17:53:50Z

what's the status on this?

23Skidoo · 2018-04-09T19:53:49Z

Stalled. Someone needs to take action to revive it.

cfraz89 · 2018-05-28T12:35:25Z

FWIW I've forked @bgamari's commit and updated it against cabal master, over here. @DanielG could you please explain the options subsets bit? Is it the equivalent of ghc-merged-pkg-options in cabal-helper?

There is now the additional complication of having this work against new-build however, at this point I'm well out of my depth. I've added the scaffolding for it, and will work on figuring it out. Could take a while though.

DanielG · 2018-09-16T18:17:15Z

@cfraz89 I've finally found some time to work on new-build support for my packages and ghc-merged-pkg-options is going to go away in the next release so don't worry about it.

samuelpilz · 2019-01-29T22:05:26Z

What is the status of this? If reasonable, @fendor and me could pick up at the current state during the next

Could someone familiar with this issue list a roadmap of next steps required for completing this PR?

mpickering · 2019-02-27T17:18:42Z

Can we please rebase and merge this? Any future changes can be made incrementally.

23Skidoo · 2019-02-27T17:21:22Z

OK with me, if it's clearly marked as unstable/experimental.

hvr

remove redundant JSON implementation

hvr · 2019-02-27T21:21:19Z

Cabal/Distribution/Simple/Utils/Json.hs

+  where
+    go []     = id
+    go [x]    = x
+    go (x:xs) = x . showString sep . go xs


Since I wrote that comment I ended up releasing an extended version of my llightweight JSON library as https://hackage.haskell.org/package/microaeson as I ended up needing a dependency-minimal JSON library...

In any case, this PR ought to get rid of the redundant JSON impl in favor of the one we're using already for plan.json.

ttuegel reviewed Aug 16, 2015
View reviewed changes

bgamari force-pushed the build-info branch from 66cf85f to ed9d56b Compare August 16, 2015 22:49

Add show-build-info command

203493a

This allows users to get a JSON representation of various information about how Cabal would go about building a package. The output of this command is intended for external tools and therefore the format should remain stable.

bgamari force-pushed the build-info branch from ed9d56b to 203493a Compare August 16, 2015 22:53

bgamari mentioned this pull request Aug 17, 2015

ghc-mod should check for CABAL_SANDBOX_CONFIG DanielG/ghc-mod#504

Closed

DanielG mentioned this pull request Sep 23, 2015

Transition to using cabal-install's show-buid-info (or similar) DanielG/cabal-helper#15

Open

ezyang added the Cabal: other label Sep 24, 2015

ezyang added the blocked: action-required label Dec 23, 2015

robinp mentioned this pull request May 14, 2016

Support for Cabal's (eventual) show-build-info commercialhaskell/stack#2139

Open

23Skidoo mentioned this pull request Sep 8, 2016

Check that new-build works with ghc-mod/haskell-mode #3731

Closed

This was referenced Sep 19, 2016

cabal new-build support DanielG/ghc-mod#835

Open

Better cabal (new-build) integration for ghc-mod #3872

Closed

ghost reviewed May 14, 2017

View reviewed changes

merijn mentioned this pull request Jul 3, 2017

hdevtools doesn't work with new-build hdevtools/hdevtools#55

Open

hvr mentioned this pull request Aug 3, 2017

cabal new-path #4661

Open

alanz mentioned this pull request Apr 11, 2018

cabal watch command #5252

Open

cfraz89 mentioned this pull request May 22, 2018

Support for cabal new-build / Per-project configuration of hie haskell/haskell-ide-engine#558

Closed

hvr mentioned this pull request Aug 1, 2018

Copy out Distribution.Solver from cabal-install into cabal-solver #5482

Closed

moll mentioned this pull request Oct 8, 2018

No GHC v8.4.1 compatibility due to base max version requirement deech/fltkhs#87

Open

hvr mentioned this pull request Nov 11, 2018

Add 'optional-target' field to plan.json #5659

Merged

3 tasks

hvr mentioned this pull request Jan 11, 2019

cabal v2-compile task #5834

Closed

hvr requested changes Feb 27, 2019

View reviewed changes

fendor mentioned this pull request Mar 20, 2019

Add new-show-build-info command #5954

Closed

4 tasks

phadej closed this Jul 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add show-build-info command #2771

Add show-build-info command #2771

bgamari commented Aug 16, 2015

bgamari commented Aug 16, 2015

bgamari commented Aug 16, 2015

23Skidoo commented Aug 16, 2015

bgamari commented Aug 16, 2015

23Skidoo commented Aug 16, 2015

ttuegel Aug 16, 2015

bgamari Aug 16, 2015

ttuegel commented Aug 16, 2015

bgamari commented Aug 17, 2015

DanielG commented Aug 17, 2015

bgamari commented Aug 17, 2015

edsko commented Aug 17, 2015

edsko commented Aug 17, 2015

DanielG commented Aug 17, 2015

23Skidoo commented Sep 22, 2015

DanielG commented Sep 23, 2015

ghost May 14, 2017

hvr May 14, 2017

hvr Feb 27, 2019

23Skidoo commented Jun 11, 2017

robinp commented Jun 11, 2017

merijn commented Jul 7, 2017

gbaz commented Apr 9, 2018

23Skidoo commented Apr 9, 2018

cfraz89 commented May 28, 2018

DanielG commented Sep 16, 2018

samuelpilz commented Jan 29, 2019

mpickering commented Feb 27, 2019

23Skidoo commented Feb 27, 2019 •

edited

Loading

hvr left a comment

hvr Feb 27, 2019

Add show-build-info command #2771

Add show-build-info command #2771

Conversation

bgamari commented Aug 16, 2015

bgamari commented Aug 16, 2015

bgamari commented Aug 16, 2015

23Skidoo commented Aug 16, 2015

bgamari commented Aug 16, 2015

23Skidoo commented Aug 16, 2015

ttuegel Aug 16, 2015

Choose a reason for hiding this comment

bgamari Aug 16, 2015

Choose a reason for hiding this comment

ttuegel commented Aug 16, 2015

bgamari commented Aug 17, 2015

DanielG commented Aug 17, 2015

bgamari commented Aug 17, 2015

edsko commented Aug 17, 2015

edsko commented Aug 17, 2015

DanielG commented Aug 17, 2015

23Skidoo commented Sep 22, 2015

DanielG commented Sep 23, 2015

ghost May 14, 2017

Choose a reason for hiding this comment

hvr May 14, 2017

Choose a reason for hiding this comment

hvr Feb 27, 2019

Choose a reason for hiding this comment

23Skidoo commented Jun 11, 2017

robinp commented Jun 11, 2017

merijn commented Jul 7, 2017

gbaz commented Apr 9, 2018

23Skidoo commented Apr 9, 2018

cfraz89 commented May 28, 2018

DanielG commented Sep 16, 2018

samuelpilz commented Jan 29, 2019

mpickering commented Feb 27, 2019

23Skidoo commented Feb 27, 2019 • edited Loading

hvr left a comment

Choose a reason for hiding this comment

hvr Feb 27, 2019

Choose a reason for hiding this comment

23Skidoo commented Feb 27, 2019 •

edited

Loading