Skip to content

Commit

Permalink
Check the specific Windows GIX_TEST_IGNORE_ARCHIVES=1 failures
Browse files Browse the repository at this point in the history
This modifies the `test-fixtures-windows` job that tests on Windows
with `GIX_TEST_IGNORE_ARCHIVES=1` so that, instead of checking that
no more than 14 failures occur, it checks that the failing tests
are exactly those that are documented in GitoxideLabs#1358 as expected to fail.

The initial check that no tests have *error* status is preserved,
with only stylistic changes, and kept separate from the subsequent
logic so that the output is clearer.

The new steps are no longer conditional on `nextest` having exited
with a failure status, since (a) that was probably unnecessary
before and definitely unnecessary now, (b) at last for now, the
comparison is precise, so it would be strange to pass if the diff
were to have changes on *all* lines, and (c) this makes it slightly
less likely that GitoxideLabs#1358 will accidentally stay open even once fixed.

The current approach is to actually retrieve the list of tests
expected to fail on Windows with `GIX_TEST_IGNORE_ARCHIVES=1` from
the GitoxideLabs#1358 issue body. This has the advantage that it automatically
keeps up to date with changes made to that issue description, but
this is of course not the only possible approach for populating the
expected value.

Two changes should be made before this is ready:

- As noted in the "FIXME" comment, the job should currently fail
  becuase the performance test reported to fail in GitoxideLabs#1358 is not
  being filtered out from the expected failures list. It's left in
  as of this commit, to verify that the job is capable of failing.

  (After that, the performance test should either be filtered out
  or removed from the list in GitoxideLabs#1358, but the former approach is
  currently preferable because I have not done diverse enough
  testing to check if the failure on my main Windows system is due
  to that system being too slow rather than a performance bug.)

- The scratchwork file should be removed once no longer needed.
  • Loading branch information
EliahKagan committed Nov 8, 2024
1 parent cf0c7ee commit d6388c2
Show file tree
Hide file tree
Showing 2 changed files with 77 additions and 8 deletions.
42 changes: 34 additions & 8 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,15 +95,41 @@ jobs:
GIX_TEST_IGNORE_ARCHIVES: 1
run: cargo nextest --profile=with-xml run --workspace --no-fail-fast
continue-on-error: true
- name: Check how many tests failed
if: steps.nextest.outcome == 'failure'
env:
# See https://github.com/GitoxideLabs/gitoxide/issues/1358.
EXPECTED_FAILURE_COUNT: 14
- name: Check for errors
run: |
[xml]$junit_xml = Get-Content -Path 'target/nextest/with-xml/junit.xml'
if ($junit_xml.testsuites.errors -ne 0) { exit 1 }
- name: Collect actual failures
run: |
[xml]$junit_xml = Get-Content 'target/nextest/with-xml/junit.xml'
$actual_failures = $junit_xml.SelectNodes("//testcase[failure]") |
ForEach-Object { "$($_.classname) $($_.name)" } |
Sort-Object
Write-Output $actual_failures
Set-Content -Path 'actual-failures.txt' -Value $actual_failures
- name: Collect expected failures
run: |
$issue = 1358 # https://github.com/GitoxideLabs/gitoxide/issues/1358
$match_info = gh issue --repo GitoxideLabs/gitoxide view $issue --json body --jq .body |
Out-String |
Select-String -Pattern '(?s)```text\r?\n(.*?)```'
# FIXME: Check that the diff can fail, then filter out performance tests in Where-Object.
$expected_failures = $match_info.Matches.Groups[1].Value -split "`n" |
Where-Object { $_ -match '^\s*FAIL \[' } |
ForEach-Object { $_ -replace '^\s*FAIL \[\s*\d+\.\d+s\]\s*', '' -replace '\s+$', '' } |
Sort-Object
Write-Output $expected_failures
Set-Content -Path 'expected-failures.txt' -Value $expected_failures
- name: Compare expected and actual failures
run: |
[xml]$junit = Get-Content -Path 'target/nextest/with-xml/junit.xml'
if ($junit.testsuites.errors -ne 0) { exit 1 }
if ($junit.testsuites.failures -gt $env:EXPECTED_FAILURE_COUNT) { exit 1 }
# Fail the check if there are any differences, even unexpectedly passing tests, so they can be
# investigated. (If this check is made blocking for PRs, this exact check may need to be changed.)
git --no-pager diff --no-index --exit-code -U1000000 -- expected-failures.txt actual-failures.txt
test-32bit:
runs-on: ubuntu-latest
Expand Down
43 changes: 43 additions & 0 deletions scratchwork.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Omit from all script steps, becuase GHA prepends it.
$ErrorActionPreference = 'stop'


Write-Output '====== Output of scratchwork for "Collect actual failures" step: ======'

[xml]$junit_xml = Get-Content 'target/nextest/with-xml/junit.xml'

$actual_failures = $junit_xml.SelectNodes("//testcase[failure]") |
ForEach-Object { "$($_.classname) $($_.name)" } |
Sort-Object

Write-Output $actual_failures
Set-Content -Path 'actual-failures.txt' -Value $actual_failures


Write-Output '====== Output of scratchwork for "Collect expected failures" step: ======'

$issue = 1358 # https://github.com/GitoxideLabs/gitoxide/issues/1358

$match_info = gh issue --repo GitoxideLabs/gitoxide view $issue --json body --jq .body |
Out-String |
Select-String -Pattern '(?s)```text\r?\n(.*?)```'

# FIXME: Check that the diff can fail, then filter out performance tests in Where-Object.
$expected_failures = $match_info.Matches.Groups[1].Value -split "`n" |
Where-Object { $_ -match '^\s*FAIL \[' } |
ForEach-Object { $_ -replace '^\s*FAIL \[\s*\d+\.\d+s\]\s*', '' -replace '\s+$', '' } |
Sort-Object

Write-Output $expected_failures
Set-Content -Path 'expected-failures.txt' -Value $expected_failures


Write-Output '====== Output of scratchwork for "Compare expected and actual failures" step: ======'

# Fail the check if there are any differences, even unexpectedly passing tests, so they can be
# investigated. (If this check is made blocking for PRs, this exact check may need to be changed.)
git --no-pager diff --no-index --exit-code -U1000000 -- expected-failures.txt actual-failures.txt


# Omit from script steps, because GHA appends it.
if ((Test-Path -LiteralPath variable:\LASTEXITCODE)) { exit $LASTEXITCODE }

0 comments on commit d6388c2

Please sign in to comment.