Add basic ATF A11y checks via AccessibilityCheckAfterTestStrategy and checkRoboAccessibility

[yschimke]

Run ATF after test completes to check for A11y issues.

Future possible changes

• Collect results and annotate screenshots with failures
• demonstrate in CI showing results

 @get:Rule
 val roborazziRule = RoborazziRule(
   composeRule = composeTestRule,
   captureRoot = composeTestRule.onRoot(),
   options = Options(
     roborazziAccessibilityOptions = RoborazziATFAccessibilityCheckOptions(
       RoborazziATFAccessibilityChecker(
         preset = AccessibilityCheckPreset.LATEST,
         suppressions = matchesElements(withTestTag("suppress")),
       ),
       failureLevel = RoborazziATFAccessibilityChecker.CheckLevel.Warning,
     ),
     accessibilityCheckStrategy = AccessibilityCheckAfterTestStrategy()
   )
 )

or

   composeTestRule.onNodeWithTag("contactcard").checkRoboAccessibility(
     roborazziATFAccessibilityCheckOptions = RoborazziATFAccessibilityCheckOptions(
       failureLevel = RoborazziATFAccessibilityChecker.CheckLevel.Warning
     )
   )

[github-actions]

Snapshot diff report

File name	Image
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.smallC lickable_compare.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.faintT ext_compare.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.clicka bleBox_compare.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.supres sionsTakeEffect_comp are.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.clicka bleWithoutSemantics_ compare.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.boxWit hEmptyContentDescrip tion_compare.png
com.github.takahirom .roborazzi.sample.Co mposeA11yTest.normal Text_compare.png

[takahirom]

Let me update this branch to check the screenshot differences, as Robolectric has been updated in the main branch.

[takahirom]

I don't fully understand what has been done in this pull request yet. I'll take a look tomorrow.🙏

[yschimke]

There are multiple ways to achieve this, depending on the end goals.

1. Apps call enableAccessibilityChecks from compose ui-test
2. Roborazzi does that
3. Roborazzi just calls AccessibilityValidator (ATF) similar to ui-test
4. Hook in to the ATF APIs that AccessibilityValidator is a thin wrapper on

I started at 2, then moved to 3, then ended up at 4.

Firstly, I think we should aim to make this part of most developers toolkit.

But with 2 or 3 it's hard to override the checks, suppressions, how screenshots are taken.

I think it ended up quite clean but also flexible.

[yschimke]

I think robolectric 4.14 produces a lot of small diffs. Improvements, but also differences in font handling.

[yschimke]

One small example that shows the benefit of this. The stdout for each test shows the errors and warnings notices for checks. So we could have a Log mode and raise awareness in existing tests

[takahirom]

📝

Error: [AccessibilityViewCheckResult check=AccessibilityHierarchyCheckResult ERROR SpeakableTextPresentCheck 4 [ViewHierarchyElement class=android.view.View bounds=Rect(474, 1074 - 606, 1206)] null num_answers:0 view=null]
Warning: [AccessibilityViewCheckResult check=AccessibilityHierarchyCheckResult WARNING TextContrastCheck 11 [ViewHierarchyElement class=android.widget.TextView text=Something hard to read bounds=Rect(403, 1002 - 678, 1092)] {KEY_BACKGROUND_COLOR=-12303292, KEY_CONTRAST_RATIO=1.02, KEY_FOREGROUND_COLOR=-12369085, KEY_REQUIRED_CONTRAST_RATIO=3.0} num_answers:0 view=null]
Error: [AccessibilityViewCheckResult check=AccessibilityHierarchyCheckResult ERROR SpeakableTextPresentCheck 4 [ViewHierarchyElement class=android.view.View bounds=Rect(474, 1074 - 606, 1206)] null num_answers:0 view=null]

[yschimke]

Looks like a good compromise.

Ideally this would be on for everyone. This is a great way to make it accessible (pun intended).

I'd like to make the errors visual in future, but this can presumably be done with a similar approach of an optional overlay for screenshots?

[yschimke]

Slight refactor and added a README

[github-actions]

⚠️ There are differences in library dependencies. See details here.

[takahirom]

I've added View support by using perform(). 👀

[takahirom]

I've added a test for Android View. 👀

[takahirom]

Looks great! Thank you for your contribution. I have a quick question.

[yschimke]

@takahirom looks great! I think I successfully nerdsniped you into building the feature I wanted but didn't know how to build. :)

[takahirom]

Thanks! Accessibility is important, and I noticed you wrote a test using a custom check. It made me think that we could also make our apps more accessible for AI agents to further improve their quality, such as by adding content descriptions for image buttons or other elements that agents might need to interact with. I believe this could also be good preparation for AI agent testing.

[yschimke]

and I noticed you wrote a test using a custom check

The intent was to prove it's possible and demystify this.