[vcpkg] Add initial versioning documentation (#15565)

* [vcpkg] Improve efficiency and tests of versioning

* [vcpkg] Add initial versioning documentation and rename x-default-baseline to builtin-baseline

* [vcpkg] Enable metrics for builtin-baseline & overrides

* [vcpkg] Address PR comments

* [vcpkg] Add support for  syntax in version>=

* [vcpkg] Remove port-version from dependency syntax

* [vcpkg] Address CR comment

* [vcpkg] Minor docs fixup

docs/users/manifests.md

@@ -64,13 +64,27 @@ Additionally, the `"port-version"` field is used by registries of packages,
 as a way to version "the package gotten from `vcpkg install`" differently from the upstream package version.
 You shouldn't need to worry about this at all.
 
+#### Additional version fields
+
+**Experimental behind the `versions` feature flag**
+
+See [versioning.md](versioning.md#version%20schemes) for additional version types.
+
 ### `"description"`
 
 This is where you describe your project. Give it a good description to help in searching for it!
 This can be a single string, or it can be an array of strings;
 in the latter case, the first string is treated as a summary,
 while the remaining strings are treated as the full description.
 
+### `"builtin-baseline"`
+
+**Experimental behind the `versions` feature flag**
+
+This field indicates the commit of vcpkg which provides global minimum version information for your manifest. It is required for top-level manifest files using versioning.
+
+See also [versioning](versioning.md#builtin-baseline) for more semantic details.
+
 ### `"dependencies"`
 
 This field lists all the dependencies you'll need to build your library (as well as any your dependents might need,
@@ -131,6 +145,16 @@ The common identifiers are:
 
 although one can define their own.
 
+#### `"version>="`
+
+**Experimental behind the `versions` feature flag**
+
+A minimum version constraint on the dependency.
+
+This field specifies the minimum version of the dependency using a '#' suffix to denote port-version if non-zero.
+
+See also [versioning](versioning.md#constraints) for more semantic details.
+
 #### Example:
 
 ```json
@@ -151,6 +175,26 @@ although one can define their own.
 }
 ```
 
+### `"overrides"`
+
+**Experimental behind the `versions` feature flag**
+
+This field enables version resolution to be ignored for certain dependencies and to use specific versions instead.
+
+See also [versioning](versioning.md#overrides) for more semantic details.
+
+#### Example:
+
+```json
+{
+  "overrides": [
+    {
+      "name": "arrow", "version": "1.2.3", "port-version": 7
+    }
+  ]
+}
+```
+
 ### `"supports"`
 
 If your project doesn't support common platforms, you can tell your users this with the `"supports"` field.
@@ -180,17 +224,17 @@ and that's the `"default-features"` field, which is an array of feature names.
   "name": "libdb",
   "description": [
     "An example database library.",
-    "Optionally uses one of CBOR, JSON, or CSV as a backend."
+    "Optionally can build with CBOR, JSON, or CSV as backends."
   ],
+  "$default-features-explanation": "Users using this library transitively will get all backends automatically",
   "default-features": [ "cbor", "csv", "json" ],
   "features": {
     "cbor": {
       "description": "The CBOR backend",
       "dependencies": [
         {
           "$explanation": [
-            "This is currently how you tell vcpkg that the cbor feature depends on the json feature of this package",
-            "We're looking into making this easier"
+            "This is how you tell vcpkg that the cbor feature depends on the json feature of this package"
           ],
           "name": "libdb",
           "default-features": false,

docs/users/versioning.md

@@ -0,0 +1,85 @@
+# Versioning
+
+**This feature is experimental and requires `--feature-flags=versions`**
+
+Versioning allows you to deterministically control the precise revisions of dependencies used by
+your project from within your manifest file.
+
+## Version schemes
+
+### Schemes
+Versions in vcpkg come in four primary flavors:
+
+#### version
+A dot-separated sequence of numbers (1.2.3.4)
+
+#### version-date
+A date (2021-01-01.5)
+
+#### version-semver
+A Semantic Version 2.0 (2.1.0-rc2)
+
+See https://semver.org/ for a full specification.
+
+#### version-string
+An exact, incomparable version (Vista)
+
+### Port Versions
+Each version additionally has a "port-version" which is a nonnegative integer. When rendered as text, the
+port version (if nonzero) is added as a suffix to the primary version text separated by a hash (#).
+Port-versions are sorted lexographically after the primary version text, for example:
+
+    1.0.0 < 1.0.0#1 < 1.0.1 < 1.0.1#5 < 2.0.0
+
+## Constraints
+
+Manifests can place three kinds of constraints upon the versions used:
+
+### builtin-baseline
+The baseline references a commit within the vcpkg repository that
+establishes a minimum version on every dependency in the graph. If
+no other constraints are specified (directly or transitively),
+then the version from the baseline of the top level manifest will
+be used.
+
+You can get the current commit of your vcpkg instance either by adding an empty `"builtin-baseline"` field, installing, and examining the error message or by running `git rev-parse HEAD` in the root of the vcpkg instance.
+
+Baselines provide stability and ease of development for top-level manifest files. They are not considered from ports consumed as a dependency. If a minimum version constraint is required during transitive version resolution, the port should use `version>=`.
+
+### version>=
+Within the "dependencies" field, each dependency can have a
+minimum constraint listed. These minimum constraints will be used
+when transitively depending upon this library. A minimum
+port-version can additionally be specified with a '#' suffix.
+
+This constraint must refer to an existing, valid version (including port-version).
+
+### overrides
+When used as the top-level manifest (such as when running `vcpkg
+install` in the directory), overrides allow a manifest to
+short-circuit dependency resolution and specify exactly the
+version to use. These can be used to handle version conflicts,
+such as with `version-string` dependencies.
+
+Overrides are not considered from ports consumed as a dependency.
+
+## Example top-level manifest:
+```json
+{
+    "name": "example",
+    "version": "1.0",
+    "builtin-baseline": "a14a6bcb27287e3ec138dba1b948a0cdbc337a3a",
+    "dependencies": [
+        { "name": "zlib", "version>=": "1.2.11#8" },
+        "rapidjson"
+    ],
+    "overrides": [
+        { "name": "rapidjson", "version": "2020-09-14" }
+    ]
+}
+```
+See also the [manifest documentation](manifests.md) for more syntax information.
+
+## Original Specification
+
+See also the [original specification](https://github.com/vicroms/vcpkg/blob/versioning-spec/docs/specifications/versioning.md)

scripts/azure-pipelines/end-to-end-tests-dir/registries.ps1

@@ -14,12 +14,13 @@ Throw-IfNotFailed
 Run-Vcpkg install @builtinRegistryArgs --feature-flags=registries 'zlib'
 Throw-IfFailed
 
-# Test git and filesystem registries
+Write-Trace "Test git and filesystem registries"
 Refresh-TestRoot
 $filesystemRegistry = "$TestingRoot/filesystem-registry"
 $gitRegistryUpstream = "$TestingRoot/git-registry-upstream"
 
 # build a filesystem registry
+Write-Trace "build a filesystem registry"
 New-Item -Path $filesystemRegistry -ItemType Directory
 $filesystemRegistry = (Get-Item $filesystemRegistry).FullName
 
@@ -51,6 +52,7 @@ New-Item `
 
 
 # build a git registry
+Write-Trace "build a git registry"
 New-Item -Path $gitRegistryUpstream -ItemType Directory
 $gitRegistryUpstream = (Get-Item $gitRegistryUpstream).FullName
 
@@ -82,6 +84,7 @@ finally
 }
 
 # actually test the registries
+Write-Trace "actually test the registries"
 $vcpkgJson = @{
     "name" = "manifest-test";
     "version-string" = "1.0.0";
@@ -90,37 +93,8 @@ $vcpkgJson = @{
     )
 }
 
-$manifestDir = "$TestingRoot/builtin-registry-test-manifest-dir"
-
-New-Item -Path $manifestDir -ItemType Directory
-$manifestDir = (Get-Item $manifestDir).FullName
-
-Push-Location $manifestDir
-
-try
-{
-    $vcpkgJsonWithBaseline = $vcpkgJson.Clone()
-    $vcpkgJsonWithBaseline['$x-default-baseline'] = 'default'
-
-    New-Item -Path 'vcpkg.json' -ItemType File `
-        -Value (ConvertTo-Json -Depth 5 -InputObject $vcpkgJson)
-
-    Run-Vcpkg install @builtinRegistryArgs '--feature-flags=registries,manifests'
-    Throw-IfNotFailed
-
-    New-Item -Path 'vcpkg.json' -ItemType File -Force `
-        -Value (ConvertTo-Json -Depth 5 -InputObject $vcpkgJsonWithBaseline)
-
-    Run-Vcpkg install @builtinRegistryArgs '--feature-flags=registries,manifests'
-    Throw-IfFailed
-}
-finally
-{
-    Pop-Location
-}
-
-
 # test the filesystem registry
+Write-Trace "test the filesystem registry"
 $manifestDir = "$TestingRoot/filesystem-registry-test-manifest-dir"
 
 New-Item -Path $manifestDir -ItemType Directory
@@ -154,6 +128,7 @@ finally
 }
 
 # test the git registry
+Write-Trace "test the git registry"
 $manifestDir = "$TestingRoot/git-registry-test-manifest-dir"
 
 New-Item -Path $manifestDir -ItemType Directory

scripts/azure-pipelines/end-to-end-tests-dir/versions.ps1

@@ -59,17 +59,40 @@ Throw-IfFailed
 Throw-IfFailed
 
 $CurrentTest = "default baseline"
-$out = ./vcpkg $commonArgs "--feature-flags=versions" install --x-manifest-root=scripts/testing/version-files/default-baseline-1
+$out = ./vcpkg $commonArgs "--feature-flags=versions" install --x-manifest-root=scripts/testing/version-files/default-baseline-1 2>&1 | Out-String
 Throw-IfNotFailed
-# if ($out -notmatch "Error: while checking out baseline" -or $out -notmatch " does not exist in ")
-# {
-#     $out
-#     throw "Expected to fail due to missing baseline"
-# }
+if ($out -notmatch ".*Error: while checking out baseline.*")
+{
+    $out
+    throw "Expected to fail due to missing baseline"
+}
 
 git fetch https://github.com/vicroms/test-registries
-$CurrentTest = "default baseline"
-./vcpkg $commonArgs --debug "--feature-flags=versions" install `
-    "--x-manifest-root=scripts/testing/version-files/default-baseline-2" `
-    "--x-builtin-port-versions-dir=scripts/testing/version-files/default-baseline-2/port_versions"
-Throw-IfFailed
+foreach ($opt_registries in @("",",registries"))
+{
+    Write-Trace "testing baselines: $opt_registries"
+    Refresh-TestRoot
+    $CurrentTest = "without default baseline 2 -- enabling versions should not change behavior"
+    Remove-Item -Recurse $buildtreesRoot/versioning -ErrorAction SilentlyContinue
+    ./vcpkg $commonArgs "--feature-flags=versions$opt_registries" install `
+        "--dry-run" `
+        "--x-manifest-root=scripts/testing/version-files/without-default-baseline-2" `
+        "--x-builtin-port-versions-dir=scripts/testing/version-files/default-baseline-2/port_versions"
+    Throw-IfFailed
+    Require-FileNotExists $buildtreesRoot/versioning
+
+    $CurrentTest = "default baseline 2"
+    ./vcpkg $commonArgs "--feature-flags=versions$opt_registries" install `
+        "--dry-run" `
+        "--x-manifest-root=scripts/testing/version-files/default-baseline-2" `
+        "--x-builtin-port-versions-dir=scripts/testing/version-files/default-baseline-2/port_versions"
+    Throw-IfFailed
+    Require-FileExists $buildtreesRoot/versioning
+
+    $CurrentTest = "using version features fails without flag"
+    ./vcpkg $commonArgs "--feature-flags=-versions$opt_registries" install `
+        "--dry-run" `
+        "--x-manifest-root=scripts/testing/version-files/default-baseline-2" `
+        "--x-builtin-port-versions-dir=scripts/testing/version-files/default-baseline-2/port_versions"
+    Throw-IfNotFailed
+}

scripts/azure-pipelines/end-to-end-tests-prelude.ps1

@@ -63,6 +63,10 @@ function Throw-IfNotFailed {
     }
 }
 
+function Write-Trace ([string]$text) {
+    Write-Host (@($MyInvocation.ScriptName, ":", $MyInvocation.ScriptLineNumber, ": ", $text) -join "")
+}
+
 function Run-Vcpkg {
     Param(
         [Parameter(ValueFromRemainingArguments)]

scripts/generatePortVersionsDb.py

@@ -35,7 +35,7 @@ def generate_port_versions_file(port_name):
         env = os.environ.copy()
         env['GIT_OPTIONAL_LOCKS'] = '0'
         output = subprocess.run(
-            [os.path.join(SCRIPT_DIRECTORY, '../vcpkg.exe'),
+            [os.path.join(SCRIPT_DIRECTORY, '../vcpkg'),
              'x-history', port_name, '--x-json', f'--output={output_file_path}'],
             capture_output=True, encoding='utf-8', env=env)
         if output.returncode != 0:

scripts/testing/version-files/default-baseline-1/vcpkg.json

@@ -1,7 +1,7 @@
 {
     "name": "default-baseline-test",
     "version-string": "0",
-    "$x-default-baseline": "fca18ba3572f8aebe3b8158c359db62a7e26134e",
+    "builtin-baseline": "fca18ba3572f8aebe3b8158c359db62a7e26134e",
     "dependencies": [
         "zlib"
     ]

scripts/testing/version-files/default-baseline-2/vcpkg.json

@@ -1,7 +1,7 @@
 {
-    "name": "default-baseline-test",
+    "name": "default-baseline-test-2",
     "version-string": "0",
-    "$x-default-baseline": "16002d9c2318dec4c69e02d9af8c0e11dca0d4c6",
+    "builtin-baseline": "16002d9c2318dec4c69e02d9af8c0e11dca0d4c6",
     "dependencies": [
         "zlib"
     ]

scripts/testing/version-files/without-default-baseline-2/port_versions/z-/zlib.json

@@ -0,0 +1,14 @@
+{
+    "versions": [
+        {
+            "git-tree": "7bb2b2f3783303a4dd41163553fe4cc103dc9262",
+            "version-string": "1.2.11",
+            "port-version": 9
+        },
+        {
+            "git-tree": "4927735fa9baca564ebddf6e6880de344b20d7a8",
+            "version-string": "1.2.11",
+            "port-version": 8
+        }
+    ]
+}

scripts/testing/version-files/without-default-baseline-2/vcpkg.json

@@ -0,0 +1,7 @@
+{
+    "name": "without-default-baseline-test-2",
+    "version-string": "0",
+    "dependencies": [
+        "zlib"
+    ]
+}