Merge branch 'main' of github.com:tanyasethi-msft/azure-sdk-for-go in…

…to feature/azdatalake/STG78-82

.github/CODEOWNERS

@@ -3,14 +3,14 @@
 
 #############
 # SDK (track2)
-############# 
+#############
 
 # Catch all for non-code project files and unowned files | folders
 /**                           @rickwinter @jhendrixMSFT
 /sdk/                         @rickwinter @chlowell @richardpark-msft
 
 # Samples
-/samples/                     @rickwinter @chlowell @jhendrixMSFT @richardpark-msft
+/sdk/samples/                     @rickwinter @chlowell @jhendrixMSFT @richardpark-msft
 
 # PRLabel: %Azure.Core
 /sdk/azcore/                  @rickwinter @chlowell @jhendrixMSFT @richardpark-msft
@@ -19,7 +19,7 @@
 /sdk/azidentity/              @chlowell @jhendrixMSFT @rickwinter @Azure/azure-sdk-write-identity
 
 # PRLable: %OpenAI
-/sdk/ai                       @richardpark-msft @jhendrixMSFT 
+/sdk/ai                       @richardpark-msft @jhendrixMSFT
 
 # PRLabel: %Internal
 /sdk/internal/                @chlowell @jhendrixMSFT @richardpark-msft @rickwinter

README.md

@@ -16,6 +16,8 @@ To get started with a module, see the README.md file located in the module's pro
 
 > NOTE: Go **1.18** or later is required. You could download and install the latest version of Go from [here](https://go.dev/doc/install). It will replace the existing Go on your machine. If you want to install multiple Go versions on the same machine, you could refer this [doc](https://go.dev/doc/manage-install).
 
+> NOTE: The [root azure-sdk-for-go Go module](https://godoc.org/github.com/Azure/azure-sdk-for-go) which contains subpaths of `/services/**/mgmt/**` (also known as track 1) is [deprecated and no longer recieving support](https://azure.github.io/azure-sdk/releases/deprecated/go.html). See [the migration guide](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/MIGRATION_GUIDE.md) to learn how to migrate to the current version.
+
 ## Packages available
 
 Each service can have both 'client' and 'management' modules. 'Client' modules are used to consume the service, whereas 'management' modules are used to configure and manage the service.

documentation/MIGRATION_GUIDE.md

@@ -1,6 +1,6 @@
 # Guide for migrating to `sdk/resourcemanager/**/arm**` from `services/**/mgmt/**`
 
-This document is intended for users that are familiar with the previous version of the Azure SDK For Go for management modules (`services/**/mgmt/**`) and wish to migrate their application to the next version of Azure resource management libraries (`sdk/resourcemanager/**/arm**`)
+This document is intended for users that are familiar with the older version of Azure SDKs of management libraries for Go language (package name starts with `services/**/mgmt/**`, also known as track1) and pursue to migrate their application code on to the latest version of Azure SDKs for Go that conform to our [current Azure SDK guidelines](https://azure.github.io/azure-sdk/general_introduction.html) (SDK package name starts with `sdk/resourcemanager/**/arm**`, also known as track2). The older Golang libraries that do not conform to [Azure SDK guidelines](https://azure.github.io/azure-sdk/general_introduction.html) are deprecated by 30 September 2023. Although the older libraries can still be used beyond deprecation, they will no longer receive official support and updates from Microsoft after that date. 
 
 **For users new to the Azure SDK For Go for resource management modules, please see the [README for 'sdk/azcore`](https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/azcore) and the README for every individual package.**
 

documentation/developer_setup.md

@@ -5,6 +5,7 @@
 * [Module Skeleton](#create-module-skeleton)
 * [Create SDK](#create-your-sdk)
 * [Write Tests](#write-tests)
+* [Write Examples](#write-examples)
 
 ## Installing Go
 
@@ -366,6 +367,20 @@ test-proxy push -a <path-to-assets.json>
 
 On completion of the push, a newly created tag will be stamped into the `assets.json` file. This new tag must be committed and pushed to your package directory along with any other changes.
 
+## Write Examples
+
+Examples are built into the Go toolchain by way of [testable examples][testable_examples]. By convention, examples are placed in a file named `example_test.go` and
+may be spread across multiple files, grouped by feature (e.g. `example_<feature>_test.go`). Since testable examples are by definition tests, the file(s) must have the `_test.go` suffix.
+
+Examples **should** be succinct allowing for copy/paste usage and **must** be clearly commented so they're easy to understand.
+
+Examples **must** be provided as testable examples, not as markdown blocks in README files (code snippets are ok but should be used sparingly as they tend to rot over time).
+This ensures that examples actually compile (and work!) and remain current as a SDK evolves. It also allows the doc tooling to automatically link API docs to their examples.
+
+Please consult the canonical documentation on [testable examples][testable_examples] for instructions on how to create/name testable examples and enabling testable example execution.
+
+All SDKs **must** include, at minimum, examples for their champion scenarios.
+
 ## Create Pipelines
 
 When you create the first PR for your library you will want to create this PR against a `track2-<package>` library. Submitting PRs to the `main` branch should only be done once your package is close to being released. Treating `track2-<package>` as your main development branch will allow nightly CI and live pipeline runs to pick up issues as soon as they are introduced. After creating this PR add a comment with the following:
@@ -398,3 +413,4 @@ This creates the pipelines that will verify future PRs. The `azure-sdk-for-go` i
 [autorest_directives]: https://github.com/Azure/autorest/blob/main/docs/generate/directives.md
 [test_resources]: https://github.com/Azure/azure-sdk-tools/tree/main/eng/common/TestResources
 [recording_package]: https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/internal/recording
+[testable_examples]: https://go.dev/blog/examples

eng/common/TestResources/New-TestResources.ps1

@@ -290,7 +290,14 @@ function BuildDeploymentOutputs([string]$serviceName, [object]$azContext, [objec
         }
     }
 
-    return $deploymentOutputs
+    # Force capitalization of all keys to avoid Azure Pipelines confusion with
+    # variable auto-capitalization and OS env var capitalization differences
+    $capitalized = @{}
+    foreach ($item in $deploymentOutputs.GetEnumerator()) {
+        $capitalized[$item.Name.ToUpperInvariant()] = $item.Value
+    }
+
+    return $capitalized
 }
 
 function SetDeploymentOutputs(

eng/common/pipelines/codeowners-linter.yml

@@ -30,7 +30,7 @@ stages:
       vmImage: ubuntu-22.04
 
     variables:
-      CodeownersLinterVersion: '1.0.0-dev.20231107.2'
+      CodeownersLinterVersion: '1.0.0-dev.20231120.3'
       DotNetDevOpsFeed: "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json"
       RepoLabelUri: "https://azuresdkartifacts.blob.core.windows.net/azure-sdk-write-teams/repository-labels-blob"
       TeamUserUri: "https://azuresdkartifacts.blob.core.windows.net/azure-sdk-write-teams/azure-sdk-write-teams-blob"

eng/common/scripts/Cadl-Project-Generate.ps1

@@ -6,7 +6,7 @@ param (
     [ValidateNotNullOrEmpty()]
     [string] $ProjectDirectory,
     [Parameter(Position=1)]
-    [string] $CadlAdditionalOptions ## addtional cadl emitter options, separated by semicolon if more than one, e.g. option1=value1;option2=value2
+    [string] $CadlAdditionalOptions ## additional cadl emitter options, separated by semicolon if more than one, e.g. option1=value1;option2=value2
 )
 
 $ErrorActionPreference = "Stop"
@@ -98,4 +98,4 @@ finally {
 $shouldCleanUp = $configuration["cleanup"] ?? $true
 if ($shouldCleanUp) {
     Remove-Item $tempFolder -Recurse -Force
-}
+}

eng/common/scripts/Helpers/DevOps-WorkItem-Helpers.ps1

@@ -135,7 +135,7 @@ function BuildHashKey()
 }
 
 $parentWorkItems = @{}
-function FindParentWorkItem($serviceName, $packageDisplayName, $outputCommand = $false, $ignoreReleasePlannerTests = $true)
+function FindParentWorkItem($serviceName, $packageDisplayName, $outputCommand = $false, $ignoreReleasePlannerTests = $true, $tag = $null)
 {
   $key = BuildHashKey $serviceName $packageDisplayName
   if ($key -and $parentWorkItems.ContainsKey($key)) {
@@ -154,6 +154,9 @@ function FindParentWorkItem($serviceName, $packageDisplayName, $outputCommand =
   else {
     $serviceCondition = "[ServiceName] <> ''"
   }
+  if ($tag) {
+    $serviceCondition += " AND [Tags] CONTAINS '${tag}'"
+  }
   if($ignoreReleasePlannerTests){
     $serviceCondition += " AND [Tags] NOT CONTAINS 'Release Planner App Test'"
   }
@@ -185,10 +188,10 @@ function FindParentWorkItem($serviceName, $packageDisplayName, $outputCommand =
 $packageWorkItems = @{}
 $packageWorkItemWithoutKeyFields = @{}
 
-function FindLatestPackageWorkItem($lang, $packageName, $outputCommand = $true, $ignoreReleasePlannerTests = $true)
+function FindLatestPackageWorkItem($lang, $packageName, $outputCommand = $true, $ignoreReleasePlannerTests = $true, $tag = $null)
 {
   # Cache all the versions of this package and language work items
-  $null = FindPackageWorkItem $lang $packageName -includeClosed $true -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests
+  $null = FindPackageWorkItem $lang $packageName -includeClosed $true -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests -tag $tag
 
   $latestWI = $null
   foreach ($wi in $packageWorkItems.Values)
@@ -208,7 +211,7 @@ function FindLatestPackageWorkItem($lang, $packageName, $outputCommand = $true,
   return $latestWI
 }
 
-function FindPackageWorkItem($lang, $packageName, $version, $outputCommand = $true, $includeClosed = $false, $ignoreReleasePlannerTests = $true)
+function FindPackageWorkItem($lang, $packageName, $version, $outputCommand = $true, $includeClosed = $false, $ignoreReleasePlannerTests = $true, $tag = $null)
 {
   $key = BuildHashKeyNoNull $lang $packageName $version
   if ($key -and $packageWorkItems.ContainsKey($key)) {
@@ -254,6 +257,9 @@ function FindPackageWorkItem($lang, $packageName, $version, $outputCommand = $tr
   if ($version) {
     $query += " AND [PackageVersionMajorMinor] = '${version}'"
   }
+  if ($tag) {
+    $query += " AND [Tags]  CONTAINS '${tag}'"
+  }
   if($ignoreReleasePlannerTests){
     $query += " AND [Tags] NOT CONTAINS 'Release Planner App Test'"
   }
@@ -331,7 +337,8 @@ function CreateWorkItemParent($id, $parentId, $oldParentId, $outputCommand = $tr
 
   Invoke-AzBoardsCmd "work-item relation add" $parameters $outputCommand | Out-Null
 }
-function CreateWorkItem($title, $type, $iteration, $area, $fields, $assignedTo, $parentId, $outputCommand = $true)
+
+function CreateWorkItem($title, $type, $iteration, $area, $fields, $assignedTo, $parentId, $relatedId = $null, $outputCommand = $true, $tag = $null)
 {
   $parameters = $ReleaseDevOpsCommonParametersWithProject
   $parameters += "--title", "`"${title}`""
@@ -341,25 +348,51 @@ function CreateWorkItem($title, $type, $iteration, $area, $fields, $assignedTo,
   if ($assignedTo) {
     $parameters += "--assigned-to", "`"${assignedTo}`""
   }
+  if ($tag)
+  {
+    if ($fields)
+    {
+      $fields += "`"System.Tags=${tag}`""
+    }
+    else
+    {
+      $parameters += "--fields"
+      $parameters += "`"System.Tags=${tag}`""
+    }
+  }
   if ($fields) {
     $parameters += "--fields"
     $parameters += $fields
   }
 
+  Write-Host "Creating work item"
   $workItem = Invoke-AzBoardsCmd "work-item create" $parameters $outputCommand
-
-  if ($parentId) {
-    $parameters = $ReleaseDevOpsCommonParameters
-    $parameters += "--id", $workItem.id
-    $parameters += "--relation-type", "parent"
-    $parameters += "--target-id", $parentId
-
-    Invoke-AzBoardsCmd "work-item relation add" $parameters $outputCommand | Out-Null
+  Write-Host $workItem
+  $workItemId = $workItem.id
+  Write-Host "Created work item [$workItemId]."
+  if ($parentId)
+  {
+    CreateWorkItemRelation $workItemId $parentId "parent" $outputCommand
+  }
+
+  # Add a work item as related if given.
+  if ($relatedId)
+  {
+    CreateWorkItemRelation $workItemId $relatedId "Related" $outputCommand
   }
-
   return $workItem
 }
 
+function CreateWorkItemRelation($id, $relatedId, $relationType, $outputCommand = $true)
+{
+  $parameters = $ReleaseDevOpsCommonParameters
+  $parameters += "--id", $id
+  $parameters += "--relation-type", $relationType
+  $parameters += "--target-id", $relatedId
+  Write-Host "Updating work item [$relatedId] as [$relationType] of [$id]."
+  Invoke-AzBoardsCmd "work-item relation add" $parameters $outputCommand | Out-Null
+}
+
 function UpdateWorkItem($id, $fields, $title, $state, $assignedTo, $outputCommand = $true)
 {
   $parameters = $ReleaseDevOpsCommonParameters
@@ -387,12 +420,12 @@ function UpdatePackageWorkItemReleaseState($id, $state, $releaseType, $outputCom
   return UpdateWorkItem -id $id -state $state -fields $fields -outputCommand $outputCommand
 }
 
-function FindOrCreateClonePackageWorkItem($lang, $pkg, $verMajorMinor, $allowPrompt = $false, $outputCommand = $false)
+function FindOrCreateClonePackageWorkItem($lang, $pkg, $verMajorMinor, $allowPrompt = $false, $outputCommand = $false, $relatedId = $null, $tag= $null, $ignoreReleasePlannerTests = $true)
 {
-  $workItem = FindPackageWorkItem -lang $lang -packageName $pkg.Package -version $verMajorMinor -includeClosed $true -outputCommand $outputCommand
+  $workItem = FindPackageWorkItem -lang $lang -packageName $pkg.Package -version $verMajorMinor -includeClosed $true -outputCommand $outputCommand -tag $tag -ignoreReleasePlannerTests $ignoreReleasePlannerTests
 
   if (!$workItem) {
-    $latestVersionItem = FindLatestPackageWorkItem -lang $lang -packageName $pkg.Package -outputCommand $outputCommand
+    $latestVersionItem = FindLatestPackageWorkItem -lang $lang -packageName $pkg.Package -outputCommand $outputCommand -tag $tag -ignoreReleasePlannerTests $ignoreReleasePlannerTests
     $assignedTo = "me"
     $extraFields = @()
     if ($latestVersionItem) {
@@ -428,15 +461,13 @@ function FindOrCreateClonePackageWorkItem($lang, $pkg, $verMajorMinor, $allowPro
         $packageInfo.ServiceName = $readInput
       }
     }
-
-
-    $workItem = CreateOrUpdatePackageWorkItem $lang $pkg $verMajorMinor -existingItem $null -assignedTo $assignedTo -extraFields $extraFields -outputCommand $outputCommand
+    $workItem = CreateOrUpdatePackageWorkItem $lang $pkg $verMajorMinor -existingItem $null -assignedTo $assignedTo -extraFields $extraFields -outputCommand $outputCommand -relatedId $relatedId -tag $tag -ignoreReleasePlannerTests $ignoreReleasePlannerTests
   }
 
   return $workItem
 }
 
-function CreateOrUpdatePackageWorkItem($lang, $pkg, $verMajorMinor, $existingItem, $assignedTo = $null, $extraFields = $null, $outputCommand = $true)
+function CreateOrUpdatePackageWorkItem($lang, $pkg, $verMajorMinor, $existingItem, $assignedTo = $null, $extraFields = $null, $outputCommand = $true, $relatedId = $null, $tag = $null, $ignoreReleasePlannerTests = $true)
 {
   if (!$lang -or !$pkg -or !$verMajorMinor) {
     Write-Host "Cannot create or update because one of lang, pkg or verMajorMinor aren't set. [$lang|$($pkg.Package)|$verMajorMinor]"
@@ -495,23 +526,24 @@ function CreateOrUpdatePackageWorkItem($lang, $pkg, $verMajorMinor, $existingIte
       }
     }
 
-    $newparentItem = FindOrCreatePackageGroupParent $serviceName $pkgDisplayName -outputCommand $false
+    $newparentItem = FindOrCreatePackageGroupParent $serviceName $pkgDisplayName -outputCommand $false -tag $tag -ignoreReleasePlannerTests $ignoreReleasePlannerTests
     UpdateWorkItemParent $existingItem $newParentItem -outputCommand $outputCommand
     return $existingItem
   }
 
-  $parentItem = FindOrCreatePackageGroupParent $serviceName $pkgDisplayName -outputCommand $false
-  $workItem = CreateWorkItem $title "Package" "Release" "Release" $fields $assignedTo $parentItem.id -outputCommand $outputCommand
+  $parentItem = FindOrCreatePackageGroupParent $serviceName $pkgDisplayName -outputCommand $false -tag $tag -ignoreReleasePlannerTests $ignoreReleasePlannerTests
+  Write-Host "Found product work item [$($parentItem.id)]. Creating package work item."
+  $workItem = CreateWorkItem $title "Package" "Release" "Release" $fields $assignedTo $parentItem.id -outputCommand $outputCommand -relatedId $relatedId -tag $tag
   Write-Host "[$($workItem.id)]$lang - $pkgName($verMajorMinor) - Created"
   return $workItem
 }
 
-function FindOrCreatePackageGroupParent($serviceName, $packageDisplayName, $outputCommand = $true, $ignoreReleasePlannerTests = $true)
+function FindOrCreatePackageGroupParent($serviceName, $packageDisplayName, $outputCommand = $true, $ignoreReleasePlannerTests = $true, $tag = $null)
 {
-  $existingItem = FindParentWorkItem $serviceName $packageDisplayName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests
+  $existingItem = FindParentWorkItem $serviceName $packageDisplayName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests -tag $tag
   if ($existingItem) {
     Write-Host "Found existing product work item [$($existingItem.id)]"
-    $newparentItem = FindOrCreateServiceParent $serviceName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests
+    $newparentItem = FindOrCreateServiceParent $serviceName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests -tag $tag
     UpdateWorkItemParent $existingItem $newParentItem
     return $existingItem
   }
@@ -520,18 +552,18 @@ function FindOrCreatePackageGroupParent($serviceName, $packageDisplayName, $outp
   $fields += "`"PackageDisplayName=${packageDisplayName}`""
   $fields += "`"ServiceName=${serviceName}`""
   $fields += "`"Custom.EpicType=Product`""
-  $serviceParentItem = FindOrCreateServiceParent $serviceName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests
-  $workItem = CreateWorkItem $packageDisplayName "Epic" "Release" "Release" $fields $null $serviceParentItem.id
+  $serviceParentItem = FindOrCreateServiceParent $serviceName -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests -tag $tag
+  $workItem = CreateWorkItem $packageDisplayName "Epic" "Release" "Release" $fields $null $serviceParentItem.id -tag $tag
 
   $localKey = BuildHashKey $serviceName $packageDisplayName
   Write-Host "[$($workItem.id)]$localKey - Created Parent"
   $parentWorkItems[$localKey] = $workItem
   return $workItem
 }
 
-function FindOrCreateServiceParent($serviceName, $outputCommand = $true, $ignoreReleasePlannerTests = $true)
+function FindOrCreateServiceParent($serviceName, $outputCommand = $true, $ignoreReleasePlannerTests = $true, $tag = $null)
 {
-  $serviceParent = FindParentWorkItem $serviceName -packageDisplayName $null -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests
+  $serviceParent = FindParentWorkItem $serviceName -packageDisplayName $null -outputCommand $outputCommand -ignoreReleasePlannerTests $ignoreReleasePlannerTests -tag $tag
   if ($serviceParent) {
     Write-Host "Found existing service work item [$($serviceParent.id)]"
     return $serviceParent
@@ -542,7 +574,7 @@ function FindOrCreateServiceParent($serviceName, $outputCommand = $true, $ignore
   $fields += "`"ServiceName=${serviceName}`""
   $fields += "`"Custom.EpicType=Service`""
   $parentId = $null
-  $workItem = CreateWorkItem $serviceName "Epic" "Release" "Release" $fields $null $parentId -outputCommand $outputCommand
+  $workItem = CreateWorkItem $serviceName "Epic" "Release" "Release" $fields $null $parentId -outputCommand $outputCommand -tag $tag
 
   $localKey = BuildHashKey $serviceName
   Write-Host "[$($workItem.id)]$localKey - Created service work item"

eng/common/scripts/Test-SampleMetadata.ps1

@@ -104,6 +104,7 @@ begin {
         "azure-active-directory-b2c",
         "azure-active-directory-domain",
         "azure-advisor",
+        "azure-ai-content-safety",
         "azure-analysis-services",
         "azure-anomaly-detector",
         "azure-api-apps",