iOSSnapshotTestCase doesn't work on Bitrise.

[durul]

Hi Guys,

I am happy to see iOSSnapshotTestCase still alive. Also, I have seen you fixed a lot of issues. But There is a problem continues integrity system. For example, our team uses Bitrise. Without Bitrise all test methods work fine but with Bitrise I am getting an error from iOSSnapshotTestCase methods . I share below. Rest of the methods work fine with Bitrise. Feel free to share fix step thanks a lot.

PMITests.LoginViewControllerSnapTest
testDefault, failed - Snapshot comparison failed: Optional(Error Domain=FBSnapshotTestControllerErrorDomain Code=1 "Unable to load reference image." UserInfo={NSLocalizedFailureReason=Reference image not found. You need to run the test in record mode, NSLocalizedDescription=Unable to load reference image., FBReferenceImageFilePathKey=/Users/vagrant/Library/Developer/Xcode/DerivedData/Acadia-aashxoxksbzmztdoledkxuavkfly/Build/Products/Integration-iphonesimulator/Vibrent.app/PlugIns/PMITests.xctest/ReferenceImages_64/PMITests.LoginViewControllerSnapTest/[email protected]})
/Users/vagrant/git/Acadia/PMITests/SnapshotTestCases/LoginViewControllerSnapTest.swift:32

        let vc = LoginViewController()
        FBSnapshotVerifyView(vc.view)

testDefault, failed - Snapshot comparison failed: Optional(Error Domain=FBSnapshotTestControllerErrorDomain Code=1 "Unable to load reference image." UserInfo={NSLocalizedFailureReason=Reference image not found. You need to run the test in record mode, NSLocalizedDescription=Unable to load reference image., FBReferenceImageFilePathKey=/Users/vagrant/Library/Developer/Xcode/DerivedData/Acadia-aashxoxksbzmztdoledkxuavkfly/Build/Products/Integration-iphonesimulator/Vibrent.app/PlugIns/PMITests.xctest/ReferenceImages_32/PMITests.LoginViewControllerSnapTest/[email protected]})
/Users/vagrant/git/Acadia/PMITests/SnapshotTestCases/LoginViewControllerSnapTest.swift:32

        let vc = LoginViewController()
        FBSnapshotVerifyView(vc.view)

 Executed 24 tests, with 2 failures (0 unexpected) in 0.534 (0.619) seconds

[alanzeino]

I don't know what Bitrise is, but looking at those logs it's kinda saying 'I can't find the reference image and I'm looking in DerivedData'. Are you setting a reference images directory? If so, are you using the environment variable or the scheme setting?

[durul]

Hi, thank you for sharing comment. Yes, I set reference images directory and I use environment variable.

[alanzeino]

My guess is that doesn't work with Bitrise, but I would say the best course of action is to figure out how that service passes environment variables. A quick google search found me this:

https://discuss.bitrise.io/t/how-to-pass-extra-build-flags-to-xcodebuild/1689/2

But again I don't use Bitrise so I can't really help you here other than to generally guide you in the right direction which I believe is to make sure the environment variable is being passed to the xcodebuild test ... command that Bitrise runs during CI jobs.

[damirstuhec]

@alanzeino I'm seeing the same issues but on CircleCI. All of my snapshot tests pass with exception of one with the same error message posted by @durul above. There are reference images in _64 but for some reason not getting found for this single test.

[alanzeino]

I can't really help with CI issues, I don't know why those services don't work correctly. We have targets that we run in our CI job on travis CI and they work just fine.

Just to clarify my earlier comment; it only falls back to looking for the images in DerivedData when no environment variable or preprocessor definition has been set for FB_REFERENCE_IMAGE_DIR. You can see that here:

- (NSString *)getReferenceImageDirectoryWithDefault:(NSString *)dir
{
    NSString *envReferenceImageDirectory = [NSProcessInfo processInfo].environment[@"FB_REFERENCE_IMAGE_DIR"];
    if (envReferenceImageDirectory) {
        return envReferenceImageDirectory;
    }
    if (dir && dir.length > 0) {
        return dir;
    }
    return [[NSBundle bundleForClass:self.class].resourcePath stringByAppendingPathComponent:@"ReferenceImages"];
}

[obuseme]

I'm having the same problem as @damirstuhec
My saved reference images are in ReferenceImages_64/ but running tests in Xcode, it's failing because it is looking for them in ReferenceImages_32/

In my scheme FB_REFERENCE_IMAGE_DIR is defined as $(SOURCE_ROOT)/$(PROJECT_NAME)Tests/ReferenceImages

[damirstuhec]

@alanzeino it's failing with the same error now locally as well. Even though the reference images are in ReferenceImages_64 it still fails and looks for them in ReferenceImages_32.
I also have the directory specified like @obuseme above. Any ideas what could be causing this?

[alanzeino]

It's not possible for me to debug your issue remotely. But you can set a breakpoint in this code and figure out what isn't working for you:

NSOrderedSet *FBSnapshotTestCaseDefaultSuffixes(void)
{
    NSMutableOrderedSet *suffixesSet = [[NSMutableOrderedSet alloc] init];
    [suffixesSet addObject:@"_32"];
    [suffixesSet addObject:@"_64"];
    if (FBSnapshotTestCaseIs64Bit()) {
        return [suffixesSet reversedOrderedSet];
    }
    return [suffixesSet copy];
}

That's what determines what suffix to apply at runtime.

[rdgborges]

@durul @damirstuhec Any updates on that issue? I'm facing exactly the same problem on TeamCity.

[durul]

I decided not using a long time ago. I am not following anymore.

[alanzeino]

I can only ask for people to meet me halfway and do a little more than the bare minimum to at least look at the code and determine why it isn't working as expected.

Bug reports that are in the vein of 'this thing doesn't work' are always going to go nowhere unless you help me out by at least doing some investigation on your end to find out what it might be. I'm not going to set up an account on every CI service that exists just to fix this, and I don't think any of you would either if you were in my shoes.

So if you're here from a google search and you're having this issue, try setting the breakpoint as I asked earlier above and let me know what you find.