Various improvements and fixes over shake install

.azure/linux-installhs-stack.yml

@@ -29,4 +29,12 @@ jobs:
   - bash: |
       source .azure/linux.bashrc
       stack install.hs help
-    displayName: Run help of `instal.hs`
+    displayName: Run help of `install.hs`
+  - bash: |
+      source .azure/linux.bashrc
+      stack install.hs stack-install-cabal
+    displayName: Run stack-install-cabal target of `install.hs`
+  - bash: |
+      source .azure/linux.bashrc
+      stack install.hs build-latest
+    displayName: Run build-latest target of `install.hs`

.azure/macos-installhs-stack.yml

@@ -29,4 +29,12 @@ jobs:
   - bash: |
       source .azure/macos.bashrc
       stack install.hs help
-    displayName: Run help of `instal.hs`
+    displayName: Run help of `install.hs`
+  - bash: |
+      source .azure/macos.bashrc
+      stack install.hs stack-install-cabal
+    displayName: Run stack-install-cabal target of `install.hs`
+  - bash: |
+      source .azure/linux.bashrc
+      stack install.hs build-latest
+    displayName: Run build-latest target of `install.hs`

.azure/macos-stack.yml

@@ -20,7 +20,7 @@ jobs:
       stack-8.4.2:
         YAML_FILE: stack-8.4.2.yaml
   variables:
-    STACK_ROOT: /Users/vsts/.stack
+    STACK_ROOT: $(Build.SourcesDirectory)/.stack
   steps:
   - task: CacheBeta@0
     inputs:

.azure/windows-installhs-stack.yml

@@ -27,4 +27,12 @@ jobs:
   - bash: |
       source .azure/windows.bashrc
       stack install.hs help
-    displayName: Run help of `instal.hs`
+    displayName: Run help of `install.hs`
+  - bash: |
+      source .azure/windows.bashrc
+      stack install.hs stack-install-cabal
+    displayName: Run stack-install-cabal target of `install.hs`
+  - bash: |
+      source .azure/linux.bashrc
+      stack install.hs build-latest
+    displayName: Run build-latest target of `install.hs`

README.md

@@ -214,7 +214,7 @@ cabal v2-run ./install.hs --project-file install/shake.project <target>
 
 Running the script with cabal on windows requires a cabal version greater or equal to `3.0.0.0`.
 
-Unfortunately, it is still required to have `stack` installed so that the install-script can locate the `local-bin` directory (on Linux `~/.local/bin`) and copy the `hie` binaries to `hie-x.y.z`, which is required for the `hie-wrapper` to function as expected.
+Unfortunately, it is still required to have `stack` installed so that the install-script can locate the `local-bin` directory (on Linux `~/.local/bin`) and copy the `hie` binaries to `hie-x.y.z`, which is required for the `hie-wrapper` to function as expected. There are plans to remove this requirement and let users build hie only with one build tool or another.
 
 For brevity, only the `stack`-based commands are presented in the following sections.
 
@@ -246,7 +246,6 @@ stack ./install.hs build-data
 
 The Haskell IDE Engine can also be built with `cabal new-build` instead of `stack build`.
 This has the advantage that you can decide how the GHC versions have been installed.
-However, this approach does currently not work for windows due to a missing feature upstream.
 To see what GHC versions are available, the command `stack install.hs cabal-ghcs` can be used.
 It will list all GHC versions that are on the path and their respective installation directory.
 If you think, this list is incomplete, you can try to modify the PATH variable, such that the executables can be found.

docs/Build.md

@@ -28,8 +28,10 @@ See the project's `README` for detailed information about installing `hie`.
 The build script `install.hs` defines several targets using the `shake` build system. The targets are roughly:
 
 * `hie-*`: builds and installs the `hie` binaries. Also renames the binaries to contain the correct version-number.
-* `build`: builds and installs `hie` binaries for all supported `ghc` versions.
+* `build-latest`: builds and installs `hie` for the latest available and supported `ghc` version.
+* `build-all`: builds and installs `hie` binaries for all supported `ghc` versions. This option may take a long time and computer resources so use it with caution.
 * `build-data`: builds the hoogle-db required by `hie`
+* `build`:  builds and installs `hie` for the latest supported `ghc` version (like `build-latest`) and the hoogle-db (like `build-data`)
 * `cabal-*`: execute the same task as the original target, but with `cabal` instead of `stack`
 
 Each `stack-*.yaml` contains references to packages in the submodules. Calling `stack` with one of those causes the build to fail if the submodules have not been initialized already. The file `shake.yaml` solves this issue invoking the `git` binary itself to update the submodules. Moreover, it specifies the correct version of `shake` and is used for installing all run-time dependencies such as `cabal` and `hoogle` if necessary.
@@ -38,7 +40,7 @@ Each `stack-*.yaml` contains references to packages in the submodules. Calling `
 
 `hie` depends on a correct environment in order to function properly:
 
-* `cabal-install`: If no `cabal` executable can be found or has an outdated version, `cabal-install` is installed via `stack`.
+* `cabal-install`: This dependency is required by `hie` to handle correctly projects that are not `stack` based (without `stack.yaml`). You can install an appropiate version using `stack` with the `stack-install-cabal` target.
 * The `hoogle` database: `hoogle generate` needs to be called with the most-recent `hoogle` version.
 
 ### Steps to build `hie`
@@ -47,16 +49,15 @@ Installing `hie` is a multi-step process:
 
 1. `git submodule sync && git submodule update --init`
 2. `hoogle generate` (`hoogle>=5.0.17` to be safe)
-3. ensure that `cabal-install` is installed in the correct version
-4. `stack --stack-yaml=stack-<X>.yaml install` or `cabal new-install -w ghc-<X>`
-5. rename `hie` binary to `hie-<X>` in `$HOME/.local/bin`, where `<X>` is the GHC version used
-6. repeat step 4 and 5 for all desired GHC versions
+3. `stack --stack-yaml=stack-<X>.yaml install` or `cabal v2-install -w ghc-<X>`
+4. rename `hie` binary to `hie-<X>` in `$HOME/.local/bin`, where `<X>` is the GHC version used
+5. repeat step 3 and 4 for all desired GHC versions
 
 This ensures that a complete install is always possible after each `git pull` or a `git clone`.
 
 #### Building `hie` with profiling support
 
-To build `hie` with profiling enabled `cabal new-install` needs to be used instead of `stack`.
+To build `hie` with profiling enabled `cabal v2-install` needs to be used instead of `stack`.
 
 Configure `cabal` to enable profiling by setting `profiling: True` in `cabal.project.local` for all packages. If that file does not already exist, create it as follows:
 
@@ -71,7 +72,7 @@ Then `hie` can be compiled for a specific GHC version:
 
 ```bash
 export GHCP=<path-to-ghc-binary>
-cabal new-install exe:hie -w $GHCP \
+cabal v2-install exe:hie -w $GHCP \
   --write-ghc-environment-files=never --symlink-bindir=$HOME/.local/bin \
   --overwrite-policy=always --reinstall
 ```
@@ -90,19 +91,17 @@ The final step is to configure the `hie` client to use a custom `hie-wrapper` sc
 The `install.hs` script performs some checks to ensure that a correct installation is possible and provide meaningful error messages for known issues.
 
 * `stack` needs to be up-to-date. Version `1.9.3` is required
+* `cabal` needs to be up-to-date. Version `3.0.0.0` is required for windows systems and `2.4.1.0` for other ones.
 * `ghc-8.6.3` is broken on windows. Trying to install `hie-8.6.3` on windows is not possible.
-* `cabal new-build` does not work on windows at the moment. All `cabal-*` targets exit with an error message about that.
 * When the build fails, an error message, that suggests to remove `.stack-work` directory, is displayed.
 
 ### Tradeoffs
 
 #### `stack` is a build dependency
 
-Currently, it is not possible to build all `hie-*` executables automatically without `stack`, since the `install.hs` script is executed by `stack`.
+Currently, `stack` is needed even if you run the script with `cabal` to get the path where install the binaries but there are plans to remove that dependency (see #1380).
 
-We are open to suggestions of other build systems that honor the requirements above, but are executable without `stack`.
-
-#### `install.hs` installs a GHC before running
+#### run `install.hs` with `stack` installs a GHC before running
 
 Before the code in `install.hs` can be executed, `stack` installs a `GHC`, depending on the `resolver` field in `shake.yaml`. This is necessary if `install.hs` should be completely functional right after a fresh `git clone` without further configuration.
 

install.hs

@@ -14,6 +14,7 @@ build-depends:
 -- * `stack install.hs <target>`
 
 -- TODO: set `shake.project` in cabal-config above, when supported
+-- (see https://github.com/haskell/cabal/issues/6353)
 
 import HieInstall (defaultMain)
 

install/src/Cabal.hs

@@ -31,20 +31,29 @@ cabalBuildData = do
   execCabal_ ["v2-build", "hoogle"]
   execCabal_ ["v2-exec", "hoogle", "generate"]
 
-cabalBuildHie :: VersionNumber -> Action ()
-cabalBuildHie versionNumber = do
-  ghcPath <- getGhcPathOf versionNumber >>= \case
+getGhcPathOfOrThrowError :: VersionNumber -> Action GhcPath
+getGhcPathOfOrThrowError versionNumber = 
+  getGhcPathOf versionNumber >>= \case
     Nothing -> do
       printInStars $ ghcVersionNotFoundFailMsg versionNumber
       error (ghcVersionNotFoundFailMsg versionNumber)
     Just p -> return p
+
+cabalBuildHie :: VersionNumber -> Action ()
+cabalBuildHie versionNumber = do
+  ghcPath <- getGhcPathOfOrThrowError versionNumber
   execCabal_
-    ["v2-build", "-w", ghcPath, "--write-ghc-environment-files=never", "--max-backjumps=5000", "--disable-tests"]
+    [ "v2-build"
+    , "-w", ghcPath
+    , "--write-ghc-environment-files=never"
+    , "--max-backjumps=5000"
+    , "--disable-tests"]
 
 cabalInstallHie :: VersionNumber -> Action ()
 cabalInstallHie versionNumber = do
   localBin <- getLocalBin
   cabalVersion <- getCabalVersion
+  ghcPath <- getGhcPathOfOrThrowError versionNumber
 
   let isCabal3 = checkVersion [3,0,0,0] cabalVersion
       installDirOpt | isCabal3 = "--installdir"
@@ -53,8 +62,9 @@ cabalInstallHie versionNumber = do
                     | otherwise = []
   execCabal_ $
     [ "v2-install"
+    , "-w", ghcPath
     , "--write-ghc-environment-files=never"
-    , installDirOpt ++ "=" ++ localBin
+    , installDirOpt, localBin
     , "exe:hie"
     , "--overwrite-policy=always"
     ]

install/src/Env.hs

@@ -8,7 +8,10 @@ import           Development.Shake.FilePath
 import           System.Info                              ( os
                                                           , arch
                                                           )
-import           Data.Maybe                               ( isJust )
+import           Data.Maybe                               ( isJust
+                                                          , isNothing
+                                                          , mapMaybe
+                                                          )
 import           System.Directory                         ( findExecutable
                                                           , findExecutables
                                                           , listDirectory
@@ -17,13 +20,13 @@ import           Data.Function                            ( (&)
                                                           , on
                                                           )
 import           Data.List                                ( sort
+                                                          , sortBy
                                                           , isInfixOf
                                                           , nubBy
                                                           )
+import           Data.Ord                                 ( comparing )
 import           Control.Monad.Extra                      ( mapMaybeM )
-import           Data.Maybe                               ( isNothing
-                                                          , mapMaybe
-                                                          )
+
 import qualified Data.Text                     as T
 
 import           Version
@@ -40,20 +43,32 @@ existsExecutable executable = liftIO $ isJust <$> findExecutable executable
 isWindowsSystem :: Bool
 isWindowsSystem = os `elem` ["mingw32", "win32"]
 
+-- | Defines all different hie versions that are supported.
+-- On windows, `8.6.3` is excluded as this version of ghc does not work there
+supportedGhcVersions :: [VersionNumber]
+supportedGhcVersions = sort (commonVersions ++ osVersions)
+  where commonVersions = ["8.4.2", "8.4.3", "8.4.4", "8.6.1", "8.6.2", "8.6.4", "8.6.5"]
+        -- the following lines exclude `8.6.3` on windows systems
+        osVersions | isWindowsSystem = []
+                   | otherwise = ["8.6.3"]
+
 findInstalledGhcs :: IO [(VersionNumber, GhcPath)]
 findInstalledGhcs = do
   hieVersions <- getHieVersions :: IO [VersionNumber]
   knownGhcs <- mapMaybeM
     (\version -> getGhcPathOf version >>= \case
-      Nothing -> return Nothing
-      Just p  -> return $ Just (version, p)
+        Nothing -> return Nothing
+        Just p  -> return $ Just (version, p)
     )
     (reverse hieVersions)
-  availableGhcs <- getGhcPaths
+  -- filter out not supported ghc versions
+  availableGhcs <- filter ((`elem` supportedGhcVersions) . fst) <$> getGhcPaths
   return
+    -- sort by version to make it coherent with getHieVersions
+    $ sortBy (comparing fst)
     -- nub by version. knownGhcs takes precedence.
     $ nubBy ((==) `on` fst)
-    -- filter out stack provided GHCs
+    -- filter out stack provided GHCs (assuming that stack programs path is the default one in linux)
     $ filter (not . isInfixOf ".stack" . snd) (knownGhcs ++ availableGhcs)
 
 -- | Get the path to a GHC that has the version specified by `VersionNumber`
@@ -63,7 +78,7 @@ findInstalledGhcs = do
 -- command fits to the desired version.
 getGhcPathOf :: MonadIO m => VersionNumber -> m (Maybe GhcPath)
 getGhcPathOf ghcVersion =
-  liftIO $ findExecutable ("ghc-" ++ ghcVersion) >>= \case
+  liftIO $ findExecutable ("ghc-" ++ ghcVersion <.> exe) >>= \case
     Nothing -> lookup ghcVersion <$> getGhcPaths
     path -> return path
 
@@ -87,7 +102,6 @@ ghcVersionNotFoundFailMsg versionNumber =
 -- | Defines all different hie versions that are buildable.
 --
 -- The current directory is scanned for `stack-*.yaml` files.
--- On windows, `8.6.3` is excluded as this version of ghc does not work there
 getHieVersions :: MonadIO m => m [VersionNumber]
 getHieVersions = do
   let stackYamlPrefix = T.pack "stack-"
@@ -99,8 +113,7 @@ getHieVersions = do
           & mapMaybe
               (T.stripPrefix stackYamlPrefix >=> T.stripSuffix stackYamlSuffix)
           & map T.unpack
-        -- the following line excludes `8.6.3` on windows systems
-          & filter (\p -> not isWindowsSystem || p /= "8.6.3")
+          & filter (\p -> p `elem` supportedGhcVersions)
           & sort
   return hieVersions
 

install/src/Help.hs

@@ -12,13 +12,23 @@ import           Version
 import           BuildSystem
 import           Cabal
 
+stackCommand :: TargetDescription -> String
+stackCommand target = "stack install.hs " ++ fst target
+
+cabalCommand :: TargetDescription -> String
+cabalCommand target = "cabal v2-run install.hs --project-file install/shake.project " ++ fst target
+
+buildCommand :: TargetDescription -> String
+buildCommand | isRunFromCabal = cabalCommand
+             | otherwise = stackCommand
+
 printUsage :: Action ()
 printUsage = do
   printLine ""
   printLine "Usage:"
-  printLineIndented "stack install.hs <target>"
+  printLineIndented (stackCommand templateTarget)
   printLineIndented "or"
-  printLineIndented "cabal new-run install.hs --project-file install/shake.project <target>"
+  printLineIndented (cabalCommand templateTarget)
 
 -- | short help message is printed by default
 shortHelpMessage :: Action ()
@@ -76,11 +86,12 @@ helpMessage versions@BuildableVersions {..} = do
   -- All targets with their respective help message.
   generalTargets = [helpTarget]
 
-  defaultTargets = [buildTarget, buildAllTarget, buildDataTarget]
+  defaultTargets = [buildTarget, buildLatestTarget, buildAllTarget, buildDataTarget]
     ++ map hieTarget (getDefaultBuildSystemVersions versions)
 
   stackTargets =
     [ stackTarget buildTarget
+      , stackTarget buildLatestTarget
       , stackTarget buildAllTarget
       , stackTarget buildDataTarget
       ]
@@ -90,6 +101,7 @@ helpMessage versions@BuildableVersions {..} = do
   cabalTargets =
     [ cabalGhcsTarget
       , cabalTarget buildTarget
+      , cabalTarget buildLatestTarget
       , cabalTarget buildAllTarget
       , cabalTarget buildDataTarget
       ]
@@ -99,6 +111,9 @@ helpMessage versions@BuildableVersions {..} = do
 emptyTarget :: (String, String)
 emptyTarget = ("", "")
 
+templateTarget :: (String, String)
+templateTarget = ("<target>", "")
+
 targetWithBuildSystem :: String -> TargetDescription -> TargetDescription
 targetWithBuildSystem system (target, description) =
   (system ++ "-" ++ target, description ++ "; with " ++ system)
@@ -114,17 +129,31 @@ hieTarget version =
   ("hie-" ++ version, "Builds hie for GHC version " ++ version)
 
 buildTarget :: TargetDescription
-buildTarget = ("build", "Builds hie with all installed GHCs")
+buildTarget = ("build", "Build hie with the latest available GHC and the data files")
+
+buildLatestTarget :: TargetDescription
+buildLatestTarget = ("build-latest", "Build hie with the latest available GHC")
 
 buildDataTarget :: TargetDescription
 buildDataTarget =
   ("build-data", "Get the required data-files for `hie` (Hoogle DB)")
 
 buildAllTarget :: TargetDescription
 buildAllTarget =
-  ("build-all", "Builds hie for all installed GHC versions and the data files")
+  ( "build-all"
+  , "Builds hie for all installed GHC versions and the data files. "
+  ++ buildAllWarning)
+
+buildAllWarning :: String
+buildAllWarning = "WARNING: This command may take a long time and computer resources"
+
+buildAllWarningAlt :: String
+buildAllWarningAlt = "Consider building only the ghc versions you need using:\n"
+                 ++ "  " ++ buildCommand (hieTarget "<ghc-version>") ++ "\n"
+                 ++ "or the latest available one with:\n"
+                 ++ "  " ++ buildCommand buildLatestTarget ++ "\n"
 
--- speical targets
+-- special targets
 
 macosIcuTarget :: TargetDescription
 macosIcuTarget = ("icu-macos-fix", "Fixes icu related problems in MacOS")
@@ -141,7 +170,9 @@ cabalGhcsTarget =
 installCabalTarget :: TargetDescription
 installCabalTarget =
   ( "install-cabal"
-  , "Install the cabal executable. It will install the required minimum version for hie (currently " ++ versionToString requiredCabalVersion ++ ") if it isn't already present in $PATH"
+  , "Install the cabal executable. It will install the required minimum version for hie (currently "
+  ++ versionToString requiredCabalVersion
+  ++ ") if it isn't already present in $PATH"
   )
 
 -- | Creates a message of the form "a, b, c and d", where a,b,c,d are GHC versions.