Merge branch main into release-web-2024-11-01 (#32862)

* Bump @azure-tools/typespec-client-generator-core (#32797) Bumps the typespec group with 1 update in the / directory: [@azure-tools/typespec-client-generator-core](https://github.com/Azure/typespec-azure). Updates `@azure-tools/typespec-client-generator-core` from 0.51.2 to 0.51.3 - [Release notes](https://github.com/Azure/typespec-azure/releases) - [Commits](https://github.com/Azure/typespec-azure/compare/@azure-tools/[email protected]...@azure-tools/[email protected]) --- updated-dependencies: - dependency-name: "@azure-tools/typespec-client-generator-core" dependency-type: direct:development update-type: version-update:semver-patch dependency-group: typespec ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Sync eng/common directory with azure-sdk-tools for PR 9902 (#32794) * populate triggeringPaths of each artifactdetails with the owning ci.yml * handle service-directory changes in Get-PrPkgProperties --------- Co-authored-by: Scott Beddall <[email protected]> * [keyvault] Update readme.python.md (#32530) * update oracle linked service (#32750) Co-authored-by: Jingshu918 <[email protected]> * Add Go Data Plane Rules of tspconfig.yaml for TSV (#32779) * manually add * add go dp rules * update --------- Co-authored-by: albertxavier100 <[email protected]> * Update Swagger for New PolicyDefinitions and PolicyTokens API (#32306) * Copy files from stable/2025-01-01 Copied the files in a separate commit. This allows reviewers to easily diff subsequent changes against the previous spec. * Update version to stable/2025-03-01 Updated the API version from stable/2025-01-01 to stable/2025-03-01. * Added tag for 2025-03-01 in readme file * Update readme.md * Add external evaluation enforcement settings contract * Add policy token API contract * Swagger PrettierCheck * Swagger PrettierCheck * Fix JsonSerializationException * Update contract * Swagger LintDiff * Increment to v6 * Revert "Increment to v6" This reverts commit 64631c5. * Update error response * Update duration * Update sdk-suppressions.yaml --------- Co-authored-by: Celina Zhao <[email protected]> Co-authored-by: kazrael2119 <[email protected]> * [Key Vault] docs.microsoft.com -> learn.microsoft.com (#32769) * Add Support for Changed Spec Identification in spec-gen-sdk-runner tool (#32763) * Increase timeout for generation job * Added git op to the wrapper tool * cleaned all files before generation * updated specGenCommand for multiple generation * Used file system access for file search * Combined tsp and readme for same service * Update readme.python.md (#32838) * Update tspconfig.yaml (#32839) * migration accessment, config java sdk (#32840) * Update tspconfig.yaml (#32823) update name of sdk packaging * Update MixedReality API with deprecation message (#32386) * Update MixedReality API * Copy existing API versino spec files * Update new API * Update README with new API version * Add deprecation message for old API versions --------- Co-authored-by: Bianca Tazlauanu <[email protected]> * Add `ListConnectionsInGroup` Api (#31398) * Copy files from stable/2024-01-01 Copied the files in a separate commit. This allows reviewers to easily diff subsequent changes against the previous spec. * Update version to stable/2024-12-01 Updated the API version from stable/2024-01-01 to stable/2024-12-01. * Added tag for 2024-12-01 in readme file * Update API spec and examples to 2024-12-01 * fix pagination according to https://github.com/Azure/azure-api-style-guide/blob/main/openapi-style-guide.md#support-for-pagination * fix(azurearcdata): remove arm-id from sqlServerDatabases (#32825) * fix(azurearcdata): remove arm-id from sqlServerDatabases * SRP 2024-01-01 API Version Swagger - PV2 date-time properties fix (#32835) * Code needs to handle more than one template library (#32853) Co-authored-by: James Suplizio <[email protected]> * Bump @autorest/openapi-to-typespec (#32834) Bumps the openapi-to-typespec group with 1 update in the / directory: [@autorest/openapi-to-typespec](https://github.com/Azure/autorest). Updates `@autorest/openapi-to-typespec` from 0.10.10 to 0.10.11 - [Release notes](https://github.com/Azure/autorest/releases) - [Commits](https://github.com/Azure/autorest/commits) --- updated-dependencies: - dependency-name: "@autorest/openapi-to-typespec" dependency-type: direct:development update-type: version-update:semver-patch dependency-group: openapi-to-typespec ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Dev/mibreset/ccm exports fabric3 (#32444) * Copy files from stable/2024-08-01 Copied the files in a separate commit. This allows reviewers to easily diff subsequent changes against the previous spec. * Update version to preview/2024-10-01-preview Updated the API version from stable/2024-08-01 to preview/2024-10-01-preview. * Added tag for 2024-10-01-preview in readme file * ran prettier on entire directory * made the changes for the fabric version * fixed spelling issue in pricesheet.json * adding 'destination type' as required since Destination is required and needs a required field * Fixed messed up date/time format * Fixed readme file, was missing the list of input files. * Removed required annotation from exportDeliveryDestination. * Include SystemAssigned,UserAssigned (#32826) * [EngSys] npm update (#32857) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Scott Beddall <[email protected]> Co-authored-by: Yuchao Yan <[email protected]> Co-authored-by: Yao Kou <[email protected]> Co-authored-by: Jingshu918 <[email protected]> Co-authored-by: Wanpeng Li <[email protected]> Co-authored-by: albertxavier100 <[email protected]> Co-authored-by: ccz77 <[email protected]> Co-authored-by: Celina Zhao <[email protected]> Co-authored-by: kazrael2119 <[email protected]> Co-authored-by: McCoy Patiño <[email protected]> Co-authored-by: Ray Chen <[email protected]> Co-authored-by: ChenxiJiang333 <[email protected]> Co-authored-by: Weidong Xu <[email protected]> Co-authored-by: Aditya <[email protected]> Co-authored-by: Bianca Tazlauanu <[email protected]> Co-authored-by: Bianca Tazlauanu <[email protected]> Co-authored-by: yzt <[email protected]> Co-authored-by: Sean Wells <[email protected]> Co-authored-by: Mayuri-Gadsing <[email protected]> Co-authored-by: James Suplizio <[email protected]> Co-authored-by: Micah Bresette <[email protected]> Co-authored-by: nasc17 <[email protected]> Co-authored-by: Mike Harder <[email protected]>
Azure · Feb 26, 2025 · afdef51 · afdef51
1 parent 2587598
commit afdef51
Show file tree

Hide file tree

Showing 13 changed files with 2,140 additions and 2,063 deletions.
diff --git a/.github/package-lock.json b/.github/package-lock.json
diff --git a/eng/common/scripts/Package-Properties.ps1 b/eng/common/scripts/Package-Properties.ps1
@@ -91,6 +91,7 @@ class PackageProps {
                 $result = [PSCustomObject]@{
                     ArtifactConfig = [HashTable]$artifactForCurrentPackage
                     ParsedYml = $content
+                    Location = $ymlPath
                 }
 
                 return $result
@@ -126,6 +127,14 @@ class PackageProps {
 
             if ($ciArtifactResult) {
                 $this.ArtifactDetails = [Hashtable]$ciArtifactResult.ArtifactConfig
+
+                if (-not $this.ArtifactDetails["triggeringPaths"]) {
+                    $this.ArtifactDetails["triggeringPaths"] = @()
+                }
+                $RepoRoot = Resolve-Path (Join-Path $PSScriptRoot ".." ".." "..")
+                $relativePath = (Resolve-Path -Path $ciArtifactResult.Location -Relative -RelativeBasePath $RepoRoot).TrimStart(".").Replace("`\", "/")
+                $this.ArtifactDetails["triggeringPaths"] += $relativePath
+
                 $this.CIParameters["CIMatrixConfigs"] = @()
 
                 # if we know this is the matrix for our file, we should now see if there is a custom matrix config for the package
@@ -207,6 +216,7 @@ function Get-PrPkgProperties([string]$InputDiffJson) {
         }
 
         foreach ($file in $targetedFiles) {
+            $pathComponents = $file -split "/"
             $shouldExclude = $false
             foreach ($exclude in $excludePaths) {
                 if ($file.StartsWith($exclude,'CurrentCultureIgnoreCase')) {
@@ -219,12 +229,12 @@ function Get-PrPkgProperties([string]$InputDiffJson) {
             }
             $filePath = (Join-Path $RepoRoot $file)
 
+            # handle direct changes to packages
             $shouldInclude = $filePath -like (Join-Path "$pkgDirectory" "*")
 
-            # this implementation guesses the working directory of the ci.yml
+            # handle changes to files that are RELATED to each package
             foreach($triggerPath in $triggeringPaths) {
                 $resolvedRelativePath = (Join-Path $RepoRoot $triggerPath)
-                # utilize the various trigger paths against the targeted file here
                 if (!$triggerPath.StartsWith("/")){
                     $resolvedRelativePath = (Join-Path $RepoRoot "sdk" "$($pkg.ServiceDirectory)" $triggerPath)
                 }
@@ -237,6 +247,36 @@ function Get-PrPkgProperties([string]$InputDiffJson) {
                     if ($includedForValidation) {
                         $pkg.IncludedForValidation = $true
                     }
+                    break
+                }
+            }
+
+            # handle service-level changes to the ci.yml files
+            # we are using the ci.yml file being added automatically to each artifactdetails as the input
+            # for this task. This is because we can resolve a service directory from the ci.yml, and if
+            # there is a single ci.yml in that directory, we can assume that any file change in that directory
+            # will apply to all packages that exist in that directory.
+            $triggeringCIYmls = $triggeringPaths | Where-Object { $_ -like "*ci*.yml" }
+
+            foreach($yml in $triggeringCIYmls) {
+                # given that this path is coming from the populated triggering paths in the artifact,
+                # we can assume that the path to the ci.yml will successfully resolve.
+                $ciYml = Join-Path $RepoRoot $yml
+                # ensure we terminate the service directory with a /
+                $directory = [System.IO.Path]::GetDirectoryName($ciYml).Replace("`\", "/") + "/"
+                $soleCIYml = (Get-ChildItem -Path $directory -Filter "ci*.yml" -File).Count -eq 1
+
+                if ($soleCIYml -and $filePath.Replace("`\", "/").StartsWith($directory)) {
+                    if (-not $shouldInclude) {
+                        $pkg.IncludedForValidation = $true
+                        $shouldInclude = $true
+                    }
+                    break
+                }
+                else {
+                    # if the ci.yml is not the only file in the directory, we cannot assume that any file changed within the directory that isn't the ci.yml
+                    # should trigger this package
+                    Write-Host "Skipping adding package for file `"$file`" because the ci yml `"$yml`" is not the only file in the service directory `"$directory`""
                 }
             }
 
@@ -280,6 +320,9 @@ function Get-PrPkgProperties([string]$InputDiffJson) {
     # packages. We should never return NO validation.
     if ($packagesWithChanges.Count -eq 0) {
         $packagesWithChanges += ($allPackageProperties | Where-Object { $_.ServiceDirectory -eq "template" })
+        foreach ($package in $packagesWithChanges) {
+            $package.IncludedForValidation = $true
+        }
     }
 
     return $packagesWithChanges

diff --git a/eng/pipelines/templates/stages/archetype-spec-gen-sdk.yml b/eng/pipelines/templates/stages/archetype-spec-gen-sdk.yml
@@ -27,11 +27,13 @@ parameters:
 extends:
   template: /eng/pipelines/templates/stages/1es-redirect.yml
   parameters:
+    Use1ESOfficial: false
     stages:
     - stage: Build
       displayName: 'SDK Generation'
       jobs:
       - job:
+        timeoutInMinutes: 2400
 
         variables:
           - template: /eng/pipelines/templates/variables/image.yml
@@ -56,12 +58,12 @@ extends:
               displayName: Publish SDK artifacts to Pipeline Artifacts
               condition: and(ne(variables['ValidationResult'], ''), eq(variables['HasSDKArtifact'], 'true'))
               artifactName: $(sdkArtifactName)
-              targetPath: "$(System.DefaultWorkingDirectory)/out/generatedSdkArtifacts"
+              targetPath: "$(System.DefaultWorkingDirectory)/out/stagedArtifacts"
             - output: pipelineArtifact
               displayName: Publish API View artifacts to Pipeline Artifacts
               condition: and(ne(variables['ValidationResult'], ''), eq(variables['HasApiViewArtifact'], 'true'))
               artifactName: $(ArtifactName)
-              targetPath: "$(System.DefaultWorkingDirectory)/out/sdkApiViewArtifacts"
+              targetPath: "$(System.DefaultWorkingDirectory)/out/stagedArtifacts"
             - output: pipelineArtifact
               displayName: Publish logs to Pipeline Artifacts
               condition: ne(variables['ValidationResult'], '')
@@ -168,7 +170,7 @@ extends:
               optional_params=""
               sdk_gen_info="sdk generation from Config : "
 
-              if [ "${{ parameters.ConfigType }}" = "TypeSpec" ]; then
+              if [ "$(Build.Reason)" != "PullRequest" ] && [ "${{ parameters.ConfigType }}" = "TypeSpec" ]; then
                 optional_params="$optional_params --tsp-config-relative-path ${{ parameters.ConfigPath }}"
                 sdk_gen_info="$sdk_gen_info '${{ parameters.ConfigPath }}',"
               elif [ "${{ parameters.ConfigType }}" = "OpenAPI" ]; then
@@ -177,7 +179,7 @@ extends:
               fi
 
               if [ "$(Build.Reason)" = "PullRequest" ]; then
-                optional_params="$optional_params --pr-number=$(System.PullRequest.PullRequestNumber)"
+                optional_params="$optional_params --pr-number $(System.PullRequest.PullRequestNumber)"
                 specPrUrl="${{ parameters.SpecRepoUrl }}/pull/$(System.PullRequest.PullRequestNumber)"
                 sdk_gen_info="$sdk_gen_info spec PR: $specPrUrl"
               fi

diff --git a/eng/tools/spec-gen-sdk-runner/cmd/spec-gen-sdk-runner.js b/eng/tools/spec-gen-sdk-runner/cmd/spec-gen-sdk-runner.js
diff --git a/eng/tools/spec-gen-sdk-runner/eslint.config.js b/eng/tools/spec-gen-sdk-runner/eslint.config.js
@@ -68,27 +68,23 @@ const config = tseslint.config(
       "@typescript-eslint/restrict-template-expressions": "off",
       "@typescript-eslint/no-unsafe-assignment": "off",
       "@typescript-eslint/no-unsafe-argument": "off",
+      "@typescript-eslint/consistent-indexed-object-style": "off",
+      "@typescript-eslint/no-unnecessary-condition": "off",
+      "@typescript-eslint/consistent-type-definitions": "off",
+      "@typescript-eslint/no-inferrable-types": "off",
+      "@typescript-eslint/no-dynamic-delete": "off",
 
       // We want more flexibility with file names.
       // https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/filename-case.md
       "unicorn/filename-case": "off",
       "unicorn/prefer-ternary": "off",
+      "unicorn/no-useless-undefined": "off",
+      "unicorn/prevent-abbreviations": "off",
 
       // We prefer to have explicitly import at the top of the file, even if the same element is exported again,
       // which we do in index.ts files.
       // https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/prefer-export-from.md
       "unicorn/prefer-export-from": ["error", { ignoreUsedVariables: true }],
-
-      // We allow some abbreviations that we like.
-      // https://github.com/sindresorhus/eslint-plugin-unicorn/blob/main/docs/rules/prevent-abbreviations.md
-      "unicorn/prevent-abbreviations": [
-        "error",
-        {
-          allowList: {
-            args: true,
-          },
-        },
-      ],
     },
   }
 );

diff --git a/eng/tools/spec-gen-sdk-runner/src/change-files.ts b/eng/tools/spec-gen-sdk-runner/src/change-files.ts
@@ -0,0 +1,184 @@
+import path from "node:path";
+import {
+  getChangedFiles,
+  searchRelatedParentFolders,
+  searchSharedLibrary,
+  searchRelatedTypeSpecProjectBySharedLibrary,
+  groupPathsByService,
+  createCombinedSpecs,
+  type SpecResults,
+  type ChangedSpecs,
+  getLastPathSegment,
+} from "./utils.js";
+import { logMessage } from "./log.js";
+import { SpecGenSdkCmdInput } from "./types.js";
+
+const readmeMdRegex = /^readme.md$/i;
+const typespecProjectRegex = /^tspconfig.yaml$/i;
+const typespecProjectSharedLibraryRegex = /[^/]+\.Shared/;
+
+export function detectChangedSpecConfigFiles(commandInput: SpecGenSdkCmdInput): ChangedSpecs[] {
+  const prChangedFiles: string[] = getChangedFiles(commandInput.localSpecRepoPath) ?? [];
+  if (prChangedFiles.length === 0) {
+    logMessage("No files changed in the PR");
+  }
+  logMessage(`Changed files in the PR: ${prChangedFiles.length}`);
+  for (const file of prChangedFiles) {
+    logMessage(`\t${file}`);
+  }
+  const fileList = prChangedFiles
+    .filter((p) => p.startsWith("specification/"))
+    .filter((p) => !p.includes("/scenarios/"));
+
+  logMessage(`Related readme.md and typespec project list:`);
+  const changedSpecs: ChangedSpecs[] = [];
+
+  const readmeMDResult = searchRelatedParentFolders(fileList, {
+    searchFileRegex: readmeMdRegex,
+    specRepoFolder: commandInput.localSpecRepoPath,
+    stopAtFolder: "specification",
+  });
+
+  const typespecProjectResult = searchRelatedParentFolders(fileList, {
+    searchFileRegex: typespecProjectRegex,
+    specRepoFolder: commandInput.localSpecRepoPath,
+    stopAtFolder: "specification",
+  });
+
+  const typespecProjectSharedLibraries = searchSharedLibrary(fileList, {
+    searchFileRegex: typespecProjectSharedLibraryRegex,
+    specRepoFolder: commandInput.localSpecRepoPath,
+  });
+
+  const typespecProjectResultSearchedBySharedLibrary = searchRelatedTypeSpecProjectBySharedLibrary(
+    typespecProjectSharedLibraries,
+    {
+      searchFileRegex: typespecProjectRegex,
+      specRepoFolder: commandInput.localSpecRepoPath,
+    },
+  );
+
+  // Merge typespec project results
+  for (const folderPath of Object.keys(typespecProjectResultSearchedBySharedLibrary)) {
+    if (typespecProjectResult[folderPath]) {
+      typespecProjectResult[folderPath] = [
+        ...typespecProjectResult[folderPath],
+        ...typespecProjectResultSearchedBySharedLibrary[folderPath],
+      ];
+    } else {
+      typespecProjectResult[folderPath] = typespecProjectResultSearchedBySharedLibrary[folderPath];
+    }
+  }
+
+  // Group paths by service
+  const serviceMap = groupPathsByService(readmeMDResult, typespecProjectResult);
+
+  const results: SpecResults = { readmeMDResult, typespecProjectResult };
+
+  // Process each service
+  for (const [, info] of serviceMap) {
+    // Case: Resource Manager with .Management
+    if (info.managementPaths.length > 0) {
+      if (info.resourceManagerPaths.length === 1) {
+        // Single resource-manager path - match with all Management paths
+        const newSpecs = createCombinedSpecs(
+          info.resourceManagerPaths[0].path,
+          info.managementPaths,
+          results,
+        );
+        changedSpecs.push(...newSpecs);
+        logMessage(
+          `\t readme folders: ${info.resourceManagerPaths[0].path}, tspconfig folders: ${info.managementPaths}`,
+        );
+        for (const p of info.managementPaths) {
+          delete typespecProjectResult[p];
+        }
+        delete readmeMDResult[info.resourceManagerPaths[0].path];
+      } else {
+        // Multiple resource-manager paths - match by subfolder name
+        for (const rmPath of info.resourceManagerPaths) {
+          const matchingManagements = info.managementPaths.filter((mPath) => {
+            const rmSubPath = rmPath.subPath;
+            const managementName = getLastPathSegment(mPath).replace(".Management", "");
+            return rmSubPath && rmSubPath === managementName;
+          });
+          if (matchingManagements.length > 0) {
+            const newSpecs = createCombinedSpecs(rmPath.path, matchingManagements, results);
+            changedSpecs.push(...newSpecs);
+            logMessage(
+              `\t readme folders: ${rmPath.path}, tspconfig folders: ${matchingManagements}`,
+            );
+            for (const p of matchingManagements) {
+              delete typespecProjectResult[p];
+            }
+            delete readmeMDResult[rmPath.path];
+          }
+        }
+      }
+    }
+
+    // Case: Data Plane matching
+    if (info.dataPlanePaths.length > 0 && info.otherTypeSpecPaths.length > 0) {
+      if (info.dataPlanePaths.length === 1) {
+        // Single data-plane path - match with all non-Management TypeSpec paths
+        const newSpecs = createCombinedSpecs(
+          info.dataPlanePaths[0].path,
+          info.otherTypeSpecPaths,
+          results,
+        );
+        changedSpecs.push(...newSpecs);
+        logMessage(
+          `\t readme folders: ${info.dataPlanePaths[0].path}, tspconfig folders: ${info.otherTypeSpecPaths}`,
+        );
+        for (const p of info.otherTypeSpecPaths) {
+          delete typespecProjectResult[p];
+        }
+        delete readmeMDResult[info.dataPlanePaths[0].path];
+      } else {
+        // Multiple data-plane paths - match by subfolder name
+        for (const dpPath of info.dataPlanePaths) {
+          const matchingTypeSpecs = info.otherTypeSpecPaths.filter((tsPath) => {
+            const dpSubFolder = dpPath.subFolder;
+            const tsLastSegment = getLastPathSegment(tsPath);
+            return dpSubFolder && dpSubFolder === tsLastSegment;
+          });
+          if (matchingTypeSpecs.length > 0) {
+            const newSpecs = createCombinedSpecs(dpPath.path, matchingTypeSpecs, results);
+            changedSpecs.push(...newSpecs);
+            logMessage(
+              `\t readme folders: ${dpPath.path}, tspconfig folders: ${matchingTypeSpecs}`,
+            );
+            for (const p of matchingTypeSpecs) {
+              delete typespecProjectResult[p];
+            }
+            delete readmeMDResult[dpPath.path];
+          }
+        }
+      }
+    }
+  }
+
+  // Process remaining unmatched paths
+  for (const folderPath of new Set([
+    ...Object.keys(readmeMDResult),
+    ...Object.keys(typespecProjectResult),
+  ])) {
+    const cs: ChangedSpecs = {
+      specs: [],
+    };
+
+    if (typespecProjectResult[folderPath]) {
+      cs.specs = typespecProjectResult[folderPath];
+      cs.typespecProject = path.join(folderPath, "tspconfig.yaml");
+      logMessage(`\t tspconfig: ${cs.typespecProject}`);
+    } else {
+      cs.readmeMd = path.join(folderPath, "readme.md");
+      cs.specs = readmeMDResult[folderPath];
+      logMessage(`\t readme: ${cs.readmeMd}`);
+    }
+
+    changedSpecs.push(cs);
+  }
+
+  return changedSpecs;
+}