Skip to content

Commit

Permalink
Fix integration comment
Browse files Browse the repository at this point in the history
Upgrade non windows native builds to C++17 (DataDog/dd-trace-dotnet#1543)

* Upgrade macos build to cpp 17

* Fixes linux build

Change native format to Microsoft style (DataDog/dd-trace-dotnet#1544)

* clang-format with Microsoft style

* Improve formatting

Add small CI fix and docker optimization (DataDog/dd-trace-dotnet#1551)

* Use a build image that is cached on the hosted windows agents, as defined here: https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#cached-docker-images

* Fix additional errors with newline character endings

Ducktype - Explicit interface method implementation support (DataDog/dd-trace-dotnet#1555)

* Adds explicit interface implementation support to DuckTyping

* Add support for wildcard.

Update ci provider extractor according to specs (DataDog/dd-trace-dotnet#1554)

* Update ci provider extractor according to specs

* fixes

Replace calli call with a normal call if the target method is a generic method. (DataDog/dd-trace-dotnet#1557)

Change native format to custom style (DataDog/dd-trace-dotnet#1553)

* Change format to custom

* changes

* changes

* Fix for windows

Modify CustomSerilogLogProvider to use older LogContext.PushProperties API when LogContext.Push API is not present (DataDog/dd-trace-dotnet#1560)

This PR fixes a gap we have today: when the application's Serilog library is old enough that it does not have the LogContext.Push API, we fail to set up our logs injection. The solution involves the following changes:
  1. Override the IsLoggerAvailable check to only return true when either the LogContext.Push API is available or the LogContext.PushProperties API is available. Otherwise, this returns false and the built-in LibLog SerilogLogProvider will be used.
  2. Update the SerilogEnricher to optionally pass in the enricher object by itself or an array containing the enricher object, based on the Serilog API used for LogContext updates.

Enable calltarget by default on .NET 4.6+ runtimes (DataDog/dd-trace-dotnet#1542)

* Enable calltarget by default on .NET 4.6+ runtimes

Merge hotfix 1.27.1 (DataDog/dd-trace-dotnet#1547)

* Bump version to 1.27.1 (DataDog/dd-trace-dotnet#1538)

* Custom native IsAlreadyLoaded() implementation for .NET 4.5.x (open-telemetry#1528) (DataDog/dd-trace-dotnet#1539)

* Split IsAlreadyLoaded() implementation depending of the .NET Framework version (<4.6)

* change comment.

(cherry picked from commit 13cdc48)

Co-authored-by: Tony Redondo <[email protected]>

* copy pipeline triggers from `master` and add `release/*` to ultimate-pipeline (DataDog/dd-trace-dotnet#1545)

* run pipelines in hotfix branches (DataDog/dd-trace-dotnet#1546)

* fix bad merge conflict resolution

* update version in build pipeline

Co-authored-by: Tony Redondo <[email protected]>

Add support for NestedTypes and GenericParameters to EnsureTypeVisibility (DataDog/dd-trace-dotnet#1561)

* Add more ducktyping tests for explict interfaces

* Add a duck typing test reproducing the specific ILogger issue

* Improves EnsuresTypeVisibility with support for NestedTypes and GenericParameters.

Co-authored-by: Tony Redondo <[email protected]>

Initialize performance counters asynchronously (DataDog/dd-trace-dotnet#1564)

Remove callsite benchmarks and set iteration time back to 2 seconds (DataDog/dd-trace-dotnet#1511)

Nuke build: overwrite files when copying trace home directory (DataDog/dd-trace-dotnet#1567)

Wait 10 more seconds on runtime metrics tests (DataDog/dd-trace-dotnet#1566)

hide warnings for EOL .NET Core targets (DataDog/dd-trace-dotnet#1569)

Use a UUID for runtime-id instead of container id (DataDog/dd-trace-dotnet#1548)

* Use a UUID for runtime-id instead of container id

Fix x86 builds in consolidated pipeline (DataDog/dd-trace-dotnet#1563)

* Rename platform -> buildPlatform

Nuke uses the Platform variable to switch between x86/x64
We could also explicitly add `-Platform $(buildPlatform)`; the current approach sets it implicitly

* Add missing build configs for WebForms project

* Update Nuke integration test runs to use x86 SDK when set to x86

Attempting to load the x86 compiled libs into the x64 bit process causes BadImageFormatExceptions

* Add templates for installing the 32 bit .NET SDKs

Install the x86 SDKs in Windows integration tests

* Add check for using install-dotnet-sdk-32bit wrong

Fix race condition in PerformanceCountersListenerTests (DataDog/dd-trace-dotnet#1573)

Revert the order in which the log providers are resolved (DataDog/dd-trace-dotnet#1578)

Update README (DataDog/dd-trace-dotnet#1576)

Reduce dependencies between build tools and helper projects (DataDog/dd-trace-dotnet#1568)

* reduce dependencies by using EnvironmentTools.GetSolutionDirectory() instead of EnvironmentHelpers

* add TracerVersion to SetAllVersions

* remove unused code

* remove dependency on Datadog.Core.Tools

* add xml-doc comments

* make SetAllVersions non-static to remove the static properties and pass path as ctor argument

CI tweaks (DataDog/dd-trace-dotnet#1570)

* Extract ARM64 jobs from linux

The linux hosts have different availability to the linux jobs, so reduces unnecessary blocking if one or the others are not available.
As a side effect of splitting these out, it reduces the size of the strategy matrix variables too.

* Increase startup timeout on CosmosDB emulator

* Don't use Console.Writeline in xUnit tests

Adds cruft to otherwise successful test runs

* "simplify" the benchmarks/throughput test variables

Previously we had two variables to control whether to run the benchmarks, but as there's already a UI for selecting which stages to run, this seems redundant. We are likely going to start running other stages on a scheduled build too, so exposing that directly makes more sense I think

* Add a step to the pipeline to upload to the Azure feed

Pulls down all our "publish" artefacts and publishes them all as a universal package

* Add missing Arm64 dependencies

* Fix version extraction

* Create a bundled "release" artifact

* Fix version number in artifact

* Set Platform when publishing

This only seems to be necessary locally, but I'm not entirely sure why!

* Fix dd-tracer-home

Delete old unused pipelines and files (DataDog/dd-trace-dotnet#1513)

* Move iis dockerfile to live with others

* Remove other pipelines (all handled by ultimate-pipeline)

* Remove no-longer-necessary docker-compose targets

* Delete unused dockerfiles and scripts

* Remove sample targets in docker-compose.yml

Somewhat opinionated, but as services are exposed on ports anyway, just as easy to run samples locally instead (Also, there's a nuke target for that)

* Start annotating which proj targets we actually use

* Remove unused targets from Datadog.Trace.proj

* This isn't the YAML you're looking for...

* Remove deleted files from solution folder

* Remove deleted file from SetAllVersions

* Delete unused Samples-iis.sln

Fixes writing the environment values in the logs on non windows platforms. (DataDog/dd-trace-dotnet#1581)

fix "PrepareRelease msi" command (DataDog/dd-trace-dotnet#1583)

Add support for Microsoft.Data.SqlClient 3.* (DataDog/dd-trace-dotnet#1579)

* Add support for Microsoft.Data.SqlClient 3.*

Fix flaky Kafka test (DataDog/dd-trace-dotnet#1585)

In Confluent.Kafka 1.5.3, attempting to consume from a non-existent topic throws an exception.
Catching the exception isn't enough, as it causes the consumer to no longer consume, even if the topic does exist later.

In the sample app, there was a race condition, whereby if the topic hadn't completely finished being created, the consumer would fail, causing the test to fail.

As a workaround, we use a separate consumer for the initial "manual" consumes, to give time for the topic to be ready. To ensure the fix works, we also _explicitly_ consume against a non-existent topic, to ensure we are expecting those errors.

> Note, I tried converting this test to snapshot testing, but the actual number of spans is indeterminate, as we can get multiple consume spans for a single message, which makes snapshot testing difficult.

Small ASP.NET Snapshot refactoring (DataDog/dd-trace-dotnet#1580)

* Update VerifyHelper to reorder tags in snapshots

Obviously, being a dictionary, the enumeration order is technically indeterminate, but in practice, this reordering seems to work, and is necessary where the order tags are added to the span at the source may change per invocation

* Move snapshots into a Snaphots sub folder

Lose the benefits of code folding in VS/Rider, but there's so many, it's too unwieldy within GitHub/filesystem currently

* Move ASP.NET Snapshots to sub folder

* Update aspnet snapshots to reorder tags and add missing runtime-id

* Fix directory char in samples directory

Uses correct character regardless of OS
This was causing some issues with IisExpress for me locally

* Don't pass executable AND applicationPath to StartProcess

I personally found this really confusing, especially when it comes to aspnetcore, and has clearly evolved a lot - there's two paths passed to it, sometimes one is executed, sometimes the other etc. Plus double negatives.
Instead, I pass the executable, add applicationPath to args if necessary in the callee, and explicitly allow passing the process to profile (optional). I think that makes things a bit clearer, but it could do with more refactoring, and reducing of duplication tbh.

* Allow specifying how many web server spans are expected in TestHelper

Revert "Delete old unused pipelines and files (open-telemetry#1513)" (DataDog/dd-trace-dotnet#1588)

This reverts commit 3d3269c.

Don't create an azure artifact (DataDog/dd-trace-dotnet#1589)

It has the potential to cause issues if storage fills up, and was only created to make programmatic consumption easier. We can reevaluate at a later date
  • Loading branch information
tonyredondo authored and pjanotti committed Sep 23, 2021
1 parent b17975f commit 6cc6a22
Show file tree
Hide file tree
Showing 1,060 changed files with 23,834 additions and 21,170 deletions.
28 changes: 22 additions & 6 deletions .azure-pipelines/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,36 @@ pr: none

variables:
buildConfiguration: release
dotnetCoreSdkVersion: 5.0.103
ddApiKey: $(DD_API_KEY)
publishOutput: $(Build.SourcesDirectory)/src/bin/managed-publish

# Declare the datadog agent as a resource to be used as a pipeline service
resources:
containers:
- container: dd_agent
image: datadog/agent
ports:
- 8126:8126
env:
DD_API_KEY: $(ddApiKey)
DD_INSIDE_CI: true

stages:
- stage: build
jobs:

#### Windows
#### Windows

- job: Windows
pool: Benchmarks

workspace:
clean: all

# Enable the Datadog Agent service for this job
services:
dd_agent: dd_agent
variables:
tracerHomeName: windows-tracer-home
tracerHome: $(System.DefaultWorkingDirectory)/src/bin/$(tracerHomeName)
Expand All @@ -29,7 +45,7 @@ stages:
displayName: install dotnet core sdk
inputs:
packageType: sdk
version: 5.0.x
version: $(dotnetCoreSdkVersion)

- task: NuGetToolInstaller@1
displayName: install nuget
Expand All @@ -52,10 +68,10 @@ stages:
inputs:
command: 'run'
projects: '$(System.DefaultWorkingDirectory)/test/benchmarks/Benchmarks.Trace/Benchmarks.Trace.csproj'
arguments: '-c Release -f netcoreapp3.1 -- -r net472 netcoreapp3.1 -m -f * --iterationTime 1500'
arguments: '-c Release -f netcoreapp3.1 -- -r net472 netcoreapp3.1 -m -f * --iterationTime 2000'
env:
OTEL_ENV: CI
OTEL_SERVICE: dd-trace-dotnet
DD_ENV: CI
DD_SERVICE: dd-trace-dotnet

- task: PowerShell@2
inputs:
Expand Down
3 changes: 1 addition & 2 deletions .azure-pipelines/crank.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ pr: none

jobs:


- job: windows_profiler
pool:
vmImage: windows-2019
Expand Down Expand Up @@ -188,4 +187,4 @@ jobs:
chmod +x ./run.sh
./run.sh
env:
OTEL_SERVICE: dd-trace-dotnet
DD_SERVICE: dd-trace-dotnet
111 changes: 102 additions & 9 deletions .azure-pipelines/integration-tests.yml
Original file line number Diff line number Diff line change
@@ -1,16 +1,19 @@
trigger:
branches:
include:
- main
- master
- release/*
- hotfix/*
paths:
exclude:
- docs/*
- .github/*
pr:
branches:
include:
- main
- master
- release/*
- hotfix/*
paths:
exclude:
- docs/*
Expand All @@ -22,6 +25,7 @@ pool:
variables:
buildConfiguration: Release
publishOutput: $(System.DefaultWorkingDirectory)/src/bin/managed-publish
dotnetCoreSdk5Version: 5.0.103

jobs:

Expand All @@ -38,7 +42,8 @@ jobs:
displayName: install dotnet core sdk 5
inputs:
packageType: sdk
version: 5.0.x
version: $(dotnetCoreSdk5Version)
includePreviewVersions: true

- task: DotNetCoreCLI@2
displayName: dotnet build Datadog.Trace.ClrProfiler.Managed.Loader
Expand Down Expand Up @@ -66,7 +71,7 @@ jobs:
zipAfterPublish: false
projects: src/Datadog.Trace.ClrProfiler.Managed/Datadog.Trace.ClrProfiler.Managed.csproj
arguments: --configuration $(buildConfiguration) --framework netcoreapp3.1 --output $(publishOutput)/netcoreapp3.1

- task: DockerCompose@0
displayName: docker-compose run Profiler
inputs:
Expand All @@ -89,7 +94,7 @@ jobs:
publishTargetFramework: netcoreapp3.1
net5_0:
publishTargetFramework: net5.0

variables:
TestAllPackageVersions: true

Expand Down Expand Up @@ -138,7 +143,8 @@ jobs:
displayName: install dotnet core sdk 5
inputs:
packageType: sdk
version: 5.0.x
version: $(dotnetCoreSdk5Version)
includePreviewVersions: true

- task: DotNetCoreCLI@2
displayName: dotnet build Datadog.Trace.ClrProfiler.Managed.Loader
Expand Down Expand Up @@ -369,7 +375,7 @@ jobs:
containerregistrytype: Container Registry
dockerComposeCommand: down
condition: succeededOrFailed()

- job: Windows_IIS
timeoutInMinutes: 100
strategy:
Expand Down Expand Up @@ -408,7 +414,8 @@ jobs:
displayName: install dotnet core sdk 5
inputs:
packageType: sdk
version: 5.0.x
version: $(dotnetCoreSdk5Version)
includePreviewVersions: true

- task: NuGetToolInstaller@1
displayName: install nuget
Expand Down Expand Up @@ -489,7 +496,7 @@ jobs:
[System.Reflection.Assembly]::Load("System.EnterpriseServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a")
$publish = New-Object System.EnterpriseServices.Internal.Publish
Get-ChildItem $(publishOutput)/net45 -Filter *.dll | Foreach-Object { $publish.GacInstall($_.FullName) }
displayName: Add net45 OpenTelemetry.AutoInstrumentation.ClrProfiler.Managed assets to the GAC
displayName: Add net45 Datadog.Trace.ClrProfiler.Managed assets to the GAC
- task: DotNetCoreCLI@2
displayName: dotnet test --filter LoadFromGAC=True
Expand All @@ -504,3 +511,89 @@ jobs:
containerregistrytype: Container Registry
dockerComposeCommand: down
condition: succeededOrFailed()

- job: build_linux_profiler_arm64
pool: Arm64
workspace:
clean: all

steps:

- task: DotNetCoreCLI@2
displayName: dotnet build Datadog.Trace.ClrProfiler.Managed.Loader
inputs:
command: build
projects: src/Datadog.Trace.ClrProfiler.Managed.Loader/Datadog.Trace.ClrProfiler.Managed.Loader.csproj
arguments: --configuration $(buildConfiguration) /nowarn:netsdk1138

- task: DotNetCoreCLI@2
displayName: dotnet publish Datadog.Trace.ClrProfiler.Managed 2.0
inputs:
command: publish
publishWebProjects: false
modifyOutputPath: false
zipAfterPublish: false
projects: src/Datadog.Trace.ClrProfiler.Managed/Datadog.Trace.ClrProfiler.Managed.csproj
arguments: --configuration $(buildConfiguration) --framework netstandard2.0 --output $(publishOutput)/netstandard2.0

- task: DotNetCoreCLI@2
displayName: dotnet publish Datadog.Trace.ClrProfiler.Managed 3.1
inputs:
command: publish
publishWebProjects: false
modifyOutputPath: false
zipAfterPublish: false
projects: src/Datadog.Trace.ClrProfiler.Managed/Datadog.Trace.ClrProfiler.Managed.csproj
arguments: --configuration $(buildConfiguration) --framework netcoreapp3.1 --output $(publishOutput)/netcoreapp3.1

- task: DockerCompose@0
displayName: docker-compose run Profiler
inputs:
containerregistrytype: Container Registry
dockerComposeCommand: run -e buildConfiguration=$(buildConfiguration) Profiler

- publish: $(System.DefaultWorkingDirectory)/src/Datadog.Trace.ClrProfiler.Native/bin/$(buildConfiguration)/x64
artifact: linux-tracer-home_arm

- job: Linux_arm64
pool: Arm64
workspace:
clean: all

dependsOn: build_linux_profiler_arm64
condition: succeeded()

variables:
TestAllPackageVersions: true

steps:
- download: current
artifact: linux-tracer-home_arm

- task: CopyFiles@2
inputs:
sourceFolder: $(Pipeline.Workspace)/linux-tracer-home_arm
targetFolder: $(System.DefaultWorkingDirectory)/src/Datadog.Trace.ClrProfiler.Native/bin/$(buildConfiguration)/arm64

- task: DockerCompose@0
displayName: docker-compose run build.arm64
inputs:
containerregistrytype: Container Registry
dockerComposeCommand: run -e TestAllPackageVersions=true -e buildConfiguration=$(buildConfiguration) -e publishTargetFramework=net5.0 build.arm64

- task: DockerCompose@0
displayName: docker-compose run IntegrationTests.ARM64.Core50
inputs:
containerregistrytype: Container Registry
dockerComposeCommand: run -e TestAllPackageVersions=true -e buildConfiguration=$(buildConfiguration) IntegrationTests.ARM64.Core50

- publish: build_data
artifact: $(Agent.JobName)_profiler-logs-$(System.JobAttempt)
condition: succeededOrFailed()

- task: PublishTestResults@2
displayName: publish test results
inputs:
testResultsFormat: VSTest
testResultsFiles: test/**/*.trx
condition: succeededOrFailed()
13 changes: 7 additions & 6 deletions .azure-pipelines/packages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ pr: none

variables:
buildConfiguration: release
dotnetCoreSdkVersion: 5.0.103
publishOutput: $(Build.SourcesDirectory)/src/bin/managed-publish

stages:
Expand All @@ -26,7 +27,7 @@ stages:
displayName: install dotnet core sdk 5.0
inputs:
packageType: sdk
version: 5.0.x
version: $(dotnetCoreSdkVersion)

- task: NuGetToolInstaller@1
displayName: install nuget
Expand Down Expand Up @@ -82,7 +83,7 @@ stages:
- task: UseDotNet@2
displayName: install dotnet core sdk
inputs:
version: 5.0.x
version: $(dotnetCoreSdkVersion)

- task: DotNetCoreCLI@2
displayName: dotnet build Datadog.Trace.ClrProfiler.Managed.Loader
Expand Down Expand Up @@ -136,7 +137,7 @@ stages:
- task: UseDotNet@2
displayName: install dotnet core sdk
inputs:
version: 5.0.x
version: $(dotnetCoreSdkVersion)

- task: DotNetCoreCLI@2
displayName: dotnet build Datadog.Trace.ClrProfiler.Managed.Loader
Expand Down Expand Up @@ -299,11 +300,11 @@ stages:
env:
SECRET: $(AWS_SECRET_ACCESS_KEY)

# by default, run this step on main branch only.
# by default, run this step on master branch only.
# use "push_artifacts_to_s3" to override:
# "true": run this step
# "false": do NOT run this step
# else: run this stage if branch is main
# else: run this stage if branch is master

- script: aws s3 cp s3_upload s3://datadog-reliability-env/dotnet/ --recursive
displayName: Upload deb, MSI, index.txt to s3
Expand All @@ -313,6 +314,6 @@ stages:
ne(variables['push_artifacts_to_s3'], 'false'),
or(
eq(variables['push_artifacts_to_s3'], 'true'),
eq(variables['Build.SourceBranch'], 'refs/heads/main')
eq(variables['Build.SourceBranch'], 'refs/heads/master')
)
)
Loading

0 comments on commit 6cc6a22

Please sign in to comment.