Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build workspace packages in top-sorted order #1018

Merged
merged 66 commits into from
Oct 5, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
9aff8e4
Build workspace packages in top-sorted order
JordanMartinez Sep 26, 2023
2a639d6
Drop unused With type
JordanMartinez Sep 26, 2023
4b9b936
Add tests for three cases
JordanMartinez Sep 26, 2023
cb7cb4c
Build leaves before roots
JordanMartinez Sep 26, 2023
b289a6c
Verify build order in test
JordanMartinez Sep 26, 2023
641e871
Fix dependency globs
JordanMartinez Sep 26, 2023
1eff3e5
Drop itOnly
JordanMartinez Sep 26, 2023
ca6a205
Allow Repl to work on all packages
JordanMartinez Sep 26, 2023
c5381dd
Only print info when building multiple packages
JordanMartinez Sep 26, 2023
46caa04
Duplicate code to simplify things temporarily
JordanMartinez Sep 27, 2023
9831a27
Inline getPackageConfigPath
JordanMartinez Sep 27, 2023
7fe0abd
Refactor error message; drop unneeded code
JordanMartinez Sep 27, 2023
3e36d05
Break cyclical module dependency: Repl.supportPackage
JordanMartinez Sep 27, 2023
f1b016b
Drop unused imports
JordanMartinez Sep 27, 2023
8f91238
Refactor Fetch.run to support Build.run's multi-package build
JordanMartinez Sep 27, 2023
990ef22
Drop old comments
JordanMartinez Sep 27, 2023
f92fda0
Revert `hasTests` removal
JordanMartinez Sep 27, 2023
5bd3e39
Merge remote-tracking branch 'origin/master' into build-via-topo-sort
JordanMartinez Sep 27, 2023
ab977f4
Merge branch 'master' into build-via-topo-sort
f-f Sep 28, 2023
1d65933
Update comment about ImportedPackages
JordanMartinez Sep 28, 2023
c4cc4a2
Refactor Graph to expose more internals
JordanMartinez Sep 28, 2023
97efc0e
Add duplicate module check when building workspace
JordanMartinez Sep 28, 2023
ccf3f91
Add xOrDieWith utils
JordanMartinez Sep 28, 2023
4d920de
Fix test module name
JordanMartinez Sep 28, 2023
e895f34
Include file path to duplicate module in error
JordanMartinez Sep 28, 2023
2dde00b
Drop itOnly
JordanMartinez Sep 28, 2023
c4c092d
Use OS-independent paths
JordanMartinez Sep 28, 2023
3f4e55b
Merge branch 'master' into build-via-topo-sort
JordanMartinez Sep 28, 2023
d0f5f85
Use record args; rename check functions
JordanMartinez Sep 28, 2023
f09f956
Cleanup ImportedPackages comment
JordanMartinez Sep 28, 2023
2d7e941
Drop runGraphCheck run code in build
JordanMartinez Sep 28, 2023
885e692
Make impossible state impossible
JordanMartinez Sep 28, 2023
37d4724
Make it easier to construct a config value
JordanMartinez Sep 28, 2023
8c34b10
Replace fixtures with inline code; move to own module
JordanMartinez Sep 28, 2023
7b6127a
Undo explicit re-exports change
JordanMartinez Sep 28, 2023
8d1ea05
Merge branch 'master' into build-via-topo-sort
f-f Sep 29, 2023
06b28e7
Drop dependencies
JordanMartinez Sep 29, 2023
6f7ce28
Get rid of misc compiler warnings
JordanMartinez Sep 29, 2023
344e2cf
Drop itOnly/describeOnly & run all tests
JordanMartinez Sep 29, 2023
48755d5
Merge remote-tracking branch 'origin/master' into build-via-topo-sort
JordanMartinez Sep 29, 2023
94c8ac7
Refactor to Fetch.getAllDependencies
JordanMartinez Oct 1, 2023
ec0e3be
Refactor supportPackage into Spago.Repl
JordanMartinez Oct 1, 2023
5ce8177
Fix typo: getToplogicallySortedWorkspacePackages
JordanMartinez Oct 1, 2023
1756c99
Use updated selected value
JordanMartinez Oct 1, 2023
d1c5b4b
Replace These with an isomorphic-but-renamed type
JordanMartinez Oct 1, 2023
a7e3038
Replace Either with PackageSelection type
JordanMartinez Oct 1, 2023
beadb4c
Unify getBuildGlobs/EntireWorkspaceGlobs
JordanMartinez Oct 2, 2023
323d183
Merge remote-tracking branch 'origin/master' into build-via-topo-sort
JordanMartinez Oct 2, 2023
47a4df5
Move depsOnly comment near first usage
JordanMartinez Oct 3, 2023
7fdf527
Write workspace `spago.yaml` in tests; explain why
JordanMartinez Oct 3, 2023
f1c92ef
Copy mermaid diagrams into module comment
JordanMartinez Oct 3, 2023
7568ac7
Add installingPackages for clarity of null check
JordanMartinez Oct 3, 2023
dad5b49
Use callback to finish single package; account for ensureRanges
JordanMartinez Oct 3, 2023
0f7f404
Define util function: ensure same representation
JordanMartinez Oct 3, 2023
f6c5152
Refactor duplicate modules error into function
JordanMartinez Oct 3, 2023
165bda7
Refactor code to prevent shadowed name on dependencies
JordanMartinez Oct 3, 2023
8ed928e
Revert back to original approach
JordanMartinez Oct 4, 2023
78474bd
Define type alias, rename function, rename to dependencies
JordanMartinez Oct 4, 2023
8b416e1
Call Config.getWorkspacePackages within getBuildGlobs
JordanMartinez Oct 4, 2023
c803de0
Merge remote-tracking branch 'origin/master' into build-via-topo-sort
JordanMartinez Oct 4, 2023
1db9ae2
Further reduce diff
JordanMartinez Oct 4, 2023
531ccaf
Move each mermaid diagram above test case
JordanMartinez Oct 4, 2023
90b7656
Nest cases 1-3 under topological build order
JordanMartinez Oct 4, 2023
34ea88c
Rename util function
JordanMartinez Oct 4, 2023
e053f18
Add ensure-ranges test in mono-/poly-repo setup
JordanMartinez Oct 4, 2023
927db28
Merge branch 'master' into build-via-topo-sort
JordanMartinez Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions bin/src/Main.purs
Original file line number Diff line number Diff line change
Expand Up @@ -819,10 +819,9 @@ mkReplEnv replArgs dependencies = do

purs <- Purs.getPurs

let
selected = case workspace.selected of
Just s -> [ s ]
Nothing -> Config.getWorkspacePackages workspace.packageSet
selected <- case workspace.selected of
Just s -> pure s
Nothing -> die "The REPL requires you to select a package"

pure
{ purs
Expand Down
54 changes: 26 additions & 28 deletions src/Spago/Command/Build.purs
Original file line number Diff line number Diff line change
Expand Up @@ -116,16 +116,6 @@ run opts = do
runCommands "Then" thenCommands
-}

let
globs = getBuildGlobs
{ dependencies
, depsOnly: opts.depsOnly
, withTests: true
, selected: case workspace.selected of
Just p -> [ p ]
Nothing -> Config.getWorkspacePackages workspace.packageSet
}

when (isJust $ Cmd.findFlag { flags: [ "-g", "--codegen" ], args: opts.pursArgs }) do
die
[ "Can't pass the `--codegen` option to purs, Spago already does that for you."
Expand All @@ -140,29 +130,37 @@ run opts = do
Nothing -> ",js,sourcemaps"
Just _ -> ""
]
Psa.psaCompile globs args psaArgs psaOptions

buildBackend

when workspace.buildOptions.pedanticPackages do
logInfo $ "Looking for unused and undeclared transitive dependencies..."
errors <- case workspace.selected of
Just selected -> Graph.runGraphCheck selected globs opts.pursArgs
Nothing -> do
-- TODO: here we could go through all the workspace packages and run the check for each
-- The complication is that "dependencies" includes all the dependencies for all packages
map Array.fold $ for (Config.getWorkspacePackages workspace.packageSet) \selected -> do
Graph.runGraphCheck selected globs opts.pursArgs
unless (Array.null errors) do
die' errors

let
selectedPackages = case workspace.selected of
Just p -> [ p ]
Nothing -> Config.getToplogicallySortedWorkspacePackages workspace.packageSet

for_ selectedPackages \selected -> do
let
globs = getBuildGlobs
{ dependencies
, depsOnly: opts.depsOnly
, withTests: true
, selected
}
Psa.psaCompile globs args psaArgs psaOptions

buildBackend

when workspace.buildOptions.pedanticPackages do
logInfo $ "Looking for unused and undeclared transitive dependencies..."
errors <- Graph.runGraphCheck selected globs opts.pursArgs
unless (Array.null errors) do
die' errors

-- TODO: if we are building with all the packages (i.e. selected = Nothing),
-- then we can use the graph to remove outdated modules from `output`!

type BuildGlobsOptions =
{ withTests :: Boolean
, depsOnly :: Boolean
, selected :: Array WorkspacePackage
, selected :: WorkspacePackage
, dependencies :: Map PackageName Package
}

Expand All @@ -171,11 +169,11 @@ getBuildGlobs { selected, dependencies, withTests, depsOnly } =
Set.fromFoldable $ projectGlobs <> monorepoPkgGlobs <> dependencyGlobs <> [ BuildInfo.buildInfoPath ]
where
-- Here we select the right globs for a monorepo setup with a bunch of packages
projectGlobs = join case depsOnly of
projectGlobs = case depsOnly of
true -> []
false ->
-- We just select all the workspace package globs, because it's (1) intuitive and (2) backwards compatible
map workspacePackageGlob selected
workspacePackageGlob selected

testGlobs = case withTests of
true -> WithTestGlobs
Expand Down
2 changes: 1 addition & 1 deletion src/Spago/Command/Publish.purs
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ publish _args = do
)

-- We then need to check that the dependency graph is accurate. If not, queue the errors
let globs = Build.getBuildGlobs { selected: [ selected ], withTests: false, dependencies, depsOnly: false }
let globs = Build.getBuildGlobs { selected, withTests: false, dependencies, depsOnly: false }
graphCheckErrors <- Graph.runGraphCheck selected globs []
for_ graphCheckErrors addError

Expand Down
2 changes: 1 addition & 1 deletion src/Spago/Command/Repl.purs
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ type ReplEnv a =
, depsOnly :: Boolean
, logOptions :: LogOptions
, pursArgs :: Array String
, selected :: Array WorkspacePackage
, selected :: WorkspacePackage
| a
}

Expand Down
2 changes: 1 addition & 1 deletion src/Spago/Command/Run.purs
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ run = do
, depsOnly: false
-- Here we include tests as well, because we use this code for `spago run` and `spago test`
, withTests: true
, selected: [ selected ]
, selected
}
Purs.graph globs [] >>= case _ of
Left err -> logWarn $ "Could not decode the output of `purs graph`, error: " <> CA.printJsonDecodeError err
Expand Down
29 changes: 28 additions & 1 deletion src/Spago/Config.purs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ module Spago.Config
, getPackageLocation
, fileSystemCharEscape
, getWorkspacePackages
, getToplogicallySortedWorkspacePackages
, module Core
, readWorkspace
, sourceGlob
Expand All @@ -28,6 +29,7 @@ import Data.CodePoint.Unicode as Unicode
import Data.Codec.Argonaut.Record as CAR
import Data.Enum as Enum
import Data.Foldable as Foldable
import Data.Graph as Graph
import Data.HTTP.Method as Method
import Data.Int as Int
import Data.Map as Map
Expand All @@ -47,7 +49,7 @@ import Registry.PackageName as PackageName
import Registry.Range as Range
import Registry.Sha256 as Sha256
import Registry.Version as Version
import Spago.Core.Config as Core
import Spago.Core.Config (BackendConfig, BuildOptionsInput, BundleConfig, BundlePlatform(..), BundleType(..), CensorBuildWarnings(..), Config, Dependencies(..), ExtraPackage(..), GitPackage, LegacyPackageSetEntry, LocalPackage, PackageConfig, PublishConfig, RemotePackage(..), RunConfig, SetAddress(..), ShowSourceCode(..), StatVerbosity(..), TestConfig, WorkspaceConfig, configCodec, dependenciesCodec, extraPackageCodec, gitPackageCodec, legacyPackageSetEntryCodec, localPackageCodec, packageConfigCodec, parseBundleType, parsePlatform, printSpagoRange, readConfig, remotePackageCodec, setAddressCodec, widestRange) as Core
import Spago.FS as FS
import Spago.Git as Git
import Spago.Lock (Lockfile)
Expand Down Expand Up @@ -456,6 +458,31 @@ getWorkspacePackages = Array.mapMaybe extractWorkspacePackage <<< Map.toUnfoldab
Tuple _ (WorkspacePackage p) -> Just p
_ -> Nothing

newtype With a b = With (Tuple a b)

instance Eq b => Eq (With a b) where
eq (With (Tuple _ b1)) (With (Tuple _ b2)) = b1 == b2

instance Ord b => Ord (With a b) where
compare (With (Tuple _ b1)) (With (Tuple _ b2)) = compare b1 b2

derive instance Newtype (With a b) _
derive instance Generic (With a b) _
instance (Show a, Show b) => Show (With a b) where
show (With t) = "(With " <> show t <> ")"

getToplogicallySortedWorkspacePackages :: PackageSet -> Array WorkspacePackage
getToplogicallySortedWorkspacePackages packageSet = do
let
packageMap = Map.fromFoldable $ map (\p -> Tuple p.package.name p) $ getWorkspacePackages packageSet
dependenciesAsList p = Set.toUnfoldable $ Map.keys $ unwrap p.package.dependencies
topSortPkgs =
Array.fromFoldable
$ Graph.topologicalSort
$ Graph.fromMap
$ map (\p -> Tuple p $ dependenciesAsList p) packageMap
Array.mapMaybe (flip Map.lookup packageMap) topSortPkgs

type PackageSetResult = { compiler :: Version, remotePackageSet :: Map PackageName Core.RemotePackage }

packageSetResultCodec :: JsonCodec PackageSetResult
Expand Down