Show actual failure reason in TeamCity build failure summary

[bronumski]

Description

Prior to using Fake 5 if the msbuild task or other tasks failed it was possible to see the failure reason in TeamCity's build failure summary for the broken build. Now with version 5 I am only seeing the general failure information.

The information is in the logs but you have to go and dig for it. Whilst this is okay for someone comfortable with Fake and TeamCity I am trying to onboard a team to use Fake as their build tool over using configuration in TeamCity. This will be one of the push backs.

Is there something in how I am either running Fake or setting up the task that would prevent this information from propagating into the failure summary for TeamCity?

Repro steps

Using this to run the build script in TeamCity command line runner:

%env.BashPath% build.sh run build.fsx

Bash script as per the getting started guide:

#!/usr/bin/env bash

set -eu
set -o pipefail

# liberated from https://stackoverflow.com/a/18443300/433393
realpath() {
  OURPWD=$PWD
  cd "$(dirname "$1")"
  LINK=$(readlink "$(basename "$1")")
  while [ "$LINK" ]; do
    cd "$(dirname "$LINK")"
    LINK=$(readlink "$(basename "$1")")
  done
  REALPATH="$PWD/$(basename "$1")"
  cd "$OURPWD"
  echo "$REALPATH"
}

TOOL_PATH=$(realpath .fake)
FAKE="$TOOL_PATH"/fake

if ! [ -e "$FAKE" ]
then
  dotnet tool install fake-cli --tool-path $TOOL_PATH --version 5.*
fi
"$FAKE" "$@"

MSbuild task setup in fake script:

Target.create "Build" (fun _ ->
  solutionFile
  |> MSBuild.build (fun p ->
        { p with
            ToolsVersion = Some "15.0"
            Verbosity = Some(Quiet)
            Targets = ["Build"]
            Properties = ["Optimize", "True"
                          "DebugSymbols", "True"
                          "Configuration", "Release"
                          "RunCodeAnalysis", "True"
                          "CodeAnalysisGenerateSuccessFile", "False"]
         }))

Expected behavior

Detail compilation (or other failure reason) to be shown in TeamCity build failure summary.

Actual behavior

As per the screen shot above.

[matthid]

Any idea what the criteria is for text to appear there in teamcity?
IIRC we removed some special logger when going from 4 to 5, the problem with the thing was that it lead to error after error especially when msbuild versions were changing, so a portable way to do it would be nice.

Maybe we just need to "forward" standard error to standard error. Problem with this approach is that the build will probably "fail" by default for any warning, so people will open issues for that again.

[bronumski]

I think you can with the ##teamcity[... messages but I would have to look into it. Now that I know that there is nothing specific I can see if I can Jerry-Rig something together. If that works I will share it here.

[matthid]

Note that that is only half of the problem because next problem is:
which msbuild output to redirect? As we redirect messages we loose coloring and other stuff and I don't really want to analyze msbuild output.

We could decide to write the last couple of lines to the build server, but if you have parallel builds that might not be enough/useful at all

[bronumski]

So I did some quick testing:

Reporting Compilation Messages

https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingCompilationMessages

This is not enough to fail the build or output anything to the build failure summary although it does appear nice and pretty in the build log:

Trace.tracefn "##teamcity[message text='%s' status='ERROR']" "This is a test error"

However this does cause a failure:

Trace.trace "##teamcity[compilationStarted compiler='DotNet']"
Trace.tracefn "##teamcity[message text='%s' status='ERROR']" "This is a test error"
Trace.trace "##teamcity[compilationFinished compiler='DotNet']"

Other Options

Reporting Build Status

https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingBuildStatus

This was used prior to TeamCity 7.1 to fail the build (backwards compatablility)

Reporting Build Problems

https://confluence.jetbrains.com/display/TCD18/Build+Script+Interaction+with+TeamCity#BuildScriptInteractionwithTeamCity-ReportingBuildProblems

For TeamCity 7.1 and above this is another way to fail the build.

Using the following:

Trace.trace "##teamcity[buildProblem description='It broke' identity='xyz']"

Resulted in:

My Two Cents

Both of the two options result in an over all build failure and they make the text included in the message available in the summary. I don't know which I prefer, they both update the summary. The compilation error is more specific but you can always add the additional detail into the problem build message.

Another thing to note is that neither of these messages prevented the build from completing so they only effect the overall state of the build which would support the parallel builds.

[bronumski]

If you can give me some pointers as to where I can hook into the msbuild results I could wrap my task with the messages and do a proof of concept. I could try and work it out but it might take me a bit longer. At any rate we could have a work around that others can use until something can be added and properly tested.

[matthid]

The relevant parts are on one hand the msbuild module (Fake.DotNet.MSBuild in src/app) and the TeamCity module

[BlythMeister]

In theory, the TeamCity trace listener should be handling this...https://github.com/fsharp/FAKE/blob/release/next/src/app/Fake.BuildServer.TeamCity/TeamCity.fs#L367

The close tag blocks should check for error and then send the build status error message...I think.

[bronumski]

Thanks @BlythMeister however looking at the listener I don't see anything in there that opens or closes the blocks when we start compilation. All the messages sent seem to be around tests and importing data. Seeing the failure reason when tests fail has not been a problem, TeamCity reports that tests have failed and the failing tests are shown in the failure summary so that all seems to work correctly.

In order to get the specific compilation failure reason we would need to implement either of the following messages:

This wraps the compilation in a block. Any errors reported here will fail the build and the text in the message will appear in the build failure summary.

# Not sure if the compiler name needs to be unique for builds run in parallel
"##teamcity[compilationStarted compiler='DotNet']"
...
"##teamcity[message text='Some error' status='ERROR']"
...
"##teamcity[compilationFinished compiler='DotNet']"

Or just sending the following would result in an over all failure and what ever is sent in the description would appear in the build failure summary.

"##teamcity[buildProblem description='It broke' identity='xyz']"

[BlythMeister]

It's the TraceData.CloseTag and TraceData.OpenTag

These are used for each target start and stop
So if the target fails, you can handle that and send the right TeamCity message.
I wrote my own custom listener with different colours which handled targets ending with a failure (not for TeamCity mind, but the concept is the same). See here https://github.com/15below/Build.Tools/blob/feature/FAKE5/Logging.fsx

[matthid]

@bronumski What is it that you are after:

• Reporting just "Compilation failed" instead of "Process exited with code 1"
• Reporting the actual compilation errors

The first one should be rather "easy" to achieve by just reporting the failed target in the Listener implementation as @BlythMeister proposed.

I was talking about the second one, and that is much more difficult to achieve properly.

[BlythMeister]

Granted I'm using msbuild, but TeamCity is able to pull the errors from the msbuild output in the console...

[bronumski]

@matthid & @BlythMeister - I want to know why the compilation failed. This would be the same as we had in Fake 4 or if you where just using raw TeamCity build steps. The actual error is in the logs but you have to dig down several nodes to get to it. This is going to make buy in harder with the current team which are used to doing the builds with the individual TeamCity steps.

I will check out the customer logger when I get a chance and report back.

[matthid]

one approach (not sure about compatibility and potential problems) might be to record a binlog and then analyse it via http://msbuildlog.com/#api

it introduces a dependency on msbuild nuget stuff but it shouldn't have the same problem we previously had with custom msbuild loggers

(The problem was that because of different msbuild versions and target frameworks we need to compile and bundle our logger multiple times)

[matthid]

this approach obviously only works for msbuild versions supporting the binlog, I believe that excludes <=msbuild14 (vs2015) which would be fine by me

[matthid]

@bronumski Can you test if 5.6.2-alpha.491 improves the situation for teamcity (or breaks something)?

[bronumski]

@matthid I will try it out today. Just out of interest, was this change only for msbuild or will it also work for builds using dotnet core. I know I specifically mentioned MSBuild but the answer will depend on how and where I test it. Thanks

[matthid]

@bronumski Yes it should work everywhere now (msbuild and all dotnet cli commands). Basically all dotnet commands (like dotnet pack) are just msbuild commands with a nicer cli (like dotnet msbuild /t:pack).

I just released some bug in 5.7.2 (#2102), because I couldn't wait for this as it is really nice (at least on all the other CIs I tested it, we have no TeamCity yet) :)

[matthid]

And by the way: Next thing I'm probably working on is getting a TeamCity instance hosted by Jetbrains back to life so integrations might improve from there :)

[BlythMeister]

@matthid TeamCity free is pretty good these days...worst case, can host in something like Azure...but there is cost there I guess.

[matthid]

@BlythMeister In reality I don't have time and money to host stuff myself just to do more work ;)