Differences in executable line detection between CI jobs

[sloede]

I have noticed something odd when looking at the coverage lcov.info files from the individual jobs in https://github.com/trixi-framework/Trixi.jl/runs/3512265649: For example, I compared the files lcov-tree_part1-ubuntu-latest-1.6-x64.info and lcov-p4est_part1-ubuntu-latest-1.6-x64.info for the file src/equations/compressible_euler_equations_1d.jl, for the lines 259ff (the flux function).

The flux function is defined from lines 259 to 268 (inclusive), where the first line and the last lines are the function flux(...) signature and the end statement, respectively. In the first LCOV file lcov-tree_part1-ubuntu-latest-1.6-x64.info , we get

...
DA:254,100                                                                                          
DA:260,12680238                                                                                     
DA:261,12680238                                                                                     
DA:262,12680238                                                                                     
DA:264,12680238                                                                                     
DA:265,12680238                                                                                     
DA:266,12680238                                                                                     
DA:267,12680238                                                                                     
DA:288,67168
...

where DA means "line with executable code" and the numbers are "line number, execution count". That is, both the function flux(...) line (line 259) and the end line (line 268) are not counted as "executable", and are thus not instrumented (see https://manpages.debian.org/stretch/lcov/geninfo.1.en.html#FILES or https://twiki.cern.ch/twiki/bin/view/SPI/PrincipleOfGCOV for more details on DA and the LCOV file structure), while the function body was executed 12,680,238 times.

The second LCOV file, lcov-p4est_part1-ubuntu-latest-1.6-x64.info, is for P4est.jl tests, thus the 1D Euler equations will never be used. As expected, the respective line execution counts in the LCOV file are all zero:

DA:254,0                                                                                            
DA:259,0                                                                                            
DA:260,0                                                                                            
DA:261,0                                                                                            
DA:262,0                                                                                            
DA:264,0                                                                                            
DA:265,0                                                                                            
DA:266,0                                                                                            
DA:267,0                                                                                            
DA:286,0

However, what's clearly different is that now lines 259 and 267 are also marked as executable, although with count zero (as they should be).

My strong suspicion now is that this is what causes the weird (and wrong) effect in the final Coverage report:

Somehow, while merging the two reports in Coverage.jl, the zero-count from the P4est tests for line 259 is registered as "hey, this line is executable but wasn't", while the Tree tests do not consider this line as executable and thus do not report any execution count at all. The end statement is obviously successfully discarded, but the function statement isn't - which causes the final report to contain a zero-count for the function statement.

Finally, the culprit seems to lie either with the parallel Coverage.jl execution, which extracts the line counts from the gcov files and does a bad job in filtering non-executable lines. Alternatively, the error is in the merging routine which should have discarded the line that is not reported at all by one of the measurements as executable.

Now, what to do with this? One possibility is to try to comb through the Coverage.jl code; alternatively we could open an issue there and ask about this behavior. Any suggestions?

[ranocha]

The reason for this seems to be related to the workaround described in https://github.com/JuliaCI/Coverage.jl#a-note-for-advanced-users.

[ranocha]

Running some simple coverage test locally (julia --code-coverage=user -e 'using Trixi; trixi_include(joinpath(get_examples(), "tree_1d_dgsem", "elixir_euler_source_terms.jl"))'), I get the following coverage report (*.cov) from Julia of lines 259-268 of src/equations/compressible_euler_1d.jl:

        - @inline function flux(u, orientation::Integer, equations::CompressibleEulerEquations1D)
   109648   rho, rho_v1, rho_e = u
   109648   v1 = rho_v1 / rho
   109648   p = (equations.gamma - 1) * (rho_e - 0.5 * rho_v1 * v1)
        -   # Ignore orientation since it is always "1" in 1D
   109648   f1 = rho_v1
   109648   f2 = rho_v1 * v1 + p
   109648   f3 = (rho_e + p) * v1
   109648   return SVector(f1, f2, f3)
        - end

Thus, the Julia code coverage generation mechanism doesn't seem to count function ... lines and corresponding end lines as executable code. However, the coverage fix described in https://github.com/JuliaCI/Coverage.jl#a-note-for-advanced-users seems to do so.

[ranocha]

It's JuliaCI/CoverageTools.jl#48

[ranocha]

Hopefully fixed by JuliaLang/julia#42170

[ranocha]

Closed by #985