[NTV-648] Agnostic Snapshot Testing Between M1 and Intel

[msadoon]

📲 What

We were having trouble using FBSnapshotTestCase to record agnostically between Intel and M1 builds. This is an issue because we cannot have a cross platform build between these two types of machines and have snapshots on one machine pass on another.

Example: A screenshot recorded on M1 will pass on M1 builds but not on Intel machines.

So a developer creating a page on Intel will not be able to push up a screenshot test that will pass on an M1 machine.

Also CI currently supports Intel based machines.

Even more technical detail is noted in this older PR.

🤔 Why

The thinking here is that we cannot use the existing screenshots because they are in a different directory and were recorded on Intel machines with a different dependency.

It's not even a matter of perceptualPrecision* as was noted in the older PR above. Some screenshots will pass with 0.98 perceptual precision, (image cannot be considered different by the human eye at that percentage), but others are different by more than 50%. So it's hard to reuse some pre-recorded images from FBSnapshotTestCase with others from SnapshotTesting.

It makes sense to re-record everything in SnapshotTesting as it provides agnostic screenshot testing across Intel and M1 and smartly puts screenshots in the folder of the view that is recorded.

SnapshotTesting is also very flexible in that it allows us to take snapshots of anything.

Another benefit is cleaning out our old _64 screenshots folder, which had a lot of outdated, unused screenshots.

Yet another benefit is putting the screenshots in a directory local to the view they record, making it easier to access them via XCode (Find in Folder), but not included in the .pbx file which would bloat the project and slow down XCode.

*As screens were verified on Intel for recordings on M1 we found that some snapshots did require a perceptualPrecision of 0.98. That is completely acceptable as you can see from this post, the human eye cannot tell the difference between two images at that match rate.
Also, this post by one of the creators of SnapshotTesting does a great job plainly explaining the difference between regular per-pixel matching and perceptual precision.

FBSnapshotTestCase only did pixel by pixel matching, not perceptual precision matching. SnapshotTesting does both.

🛠 How

Manual labour.

Found every FBSnapshotVerifyView function call and replace with assertSnapshot.

Also imported SnapshotTesting into every test file.

Ran the tests, they got recorded.

Validated some snapshots that were iffy because they recorded vc.view which failed to run the assertSnapshot, instead used parent.view or vc with those tests to record with assertSnapshot.

🏎 How do we write tests now?

We still record on iPhone 8, min iOS 14.5, this is enforced in the preferredSimulatorCheck that is setup in TestCase.
Tests will fail to record and run if the schema for Kickstarter-Framework-iOS is set to anything other than iPhone 8.

We're now using something like:

        assertSnapshot(
          matching: parent.view,
          as: .image(perceptualPrecision: 0.98),
          named: "lang_\(language)_device_\(device)"
        )

Important thing to remember is perceptualPrecision which is set to 0.98 to account for differences in SnapshotTesting recording between Intel and M1.

Everything else should work as before.

🏎 Performance

New SnapshotTesting library is definitely slower (built first, then run test suite after pre-build.)
Intel test run time with FBSnapshotTest

Intel test run time with SnapshotTesting

So oddly, this new library comes with a significant performance cost on Intel.

However the reverse is true on M1.

SnapshotTesting

FBSnapshotTest (granted some tests did fail, unlike the other runs, but the entire suite still ran with no early exits.)

Looks like a significant improvement in running on M1

✅ Acceptance criteria

• Ensure recordings of snapshots on M1 are passing on Intel
• Ensure recordings of snapshots on Intel pass on M1

⏰ TODO

• Record tests for SetYourPasswordViewController and ResetYourFacebookPasswordViewController as they are missing.
• Remove FBSnapshotTestCase completely from project
  • remove SPM package
  • embeds in targets,
  • schema variables like FB_REFERENCE_NAME_DIR,
  • _64 director
  • Left in preferred simulator check and used in setup for XCTestCase subclass.
  • Check if codecov includes screenshots, it’ll change after above steps if it does.

[nativeksr]

	1 Warning
⚠️	Big PR

Generated by 🚫 Danger

[codecov]

Codecov Report

Merging #1794 (98135d5) into main (40d368f) will increase coverage by 0.23%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1794      +/-   ##
==========================================
+ Coverage   85.23%   85.47%   +0.23%     
==========================================
  Files        1281     1282       +1     
  Lines      116753   117246     +493     
  Branches    30823    31037     +214     
==========================================
+ Hits        99519   100211     +692     
+ Misses      16160    15960     -200     
- Partials     1074     1075       +1     

Impacted Files	Coverage Δ
...ies/Controller/ActivitiesViewControllerTests.swift	100.00% <100.00%> (ø)
...r/BackerDashboardProjectsViewControllerTests.swift	100.00% <100.00%> (ø)
...ools/Controller/BetaToolsViewControllerTests.swift	100.00% <100.00%> (ø)
...e/Controller/CancelPledgeViewControllerTests.swift	100.00% <100.00%> (ø)
...troller/CategorySelectionViewControllerTests.swift	100.00% <100.00%> (ø)
...il/Controller/ChangeEmailViewControllerTests.swift	95.45% <100.00%> (ø)
...Controller/ChangePasswordViewControllerTests.swift	87.50% <100.00%> (ø)
...ontrollers/CommentRepliesViewControllerTests.swift	100.00% <100.00%> (ø)
...ents/Controllers/CommentsViewControllerTests.swift	100.00% <100.00%> (ø)
...OS/Features/Signup/SignupViewControllerTests.swift	100.00% <0.00%> (ø)
... and 59 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[msadoon]

@scottkicks suggest opening this one locally on xcode, the number of files will break the page (at least on mine). It's a huge number of file changes, so lmk if you have any questions with the way tests work now.

[scottkicks]

Passed all tests on my M1!
Nice work. Really appreciate the thourough explanation