Add plugin/hook ability to allow for extending Fork operations to accommodate domain specific functionality.

[sean-kenny]

After trying out Fork, we realized a need to extend some of the Fork core/runner functionality in a couple of ways to support our internal use cases and requirements.

It seems like Fork is missing the ability to customize in areas that we needed. In light of this it would be very useful if Fork could provide additional customization hooks...the alternative is to maintain a separate fork with additional features (obviously not ideal), but a plugin/hook system could be useful for many other organizations/developers as well..

To be specific. We required additional hooks into the following areas of the system:

1. Additional test method Annotation data extraction. Which can be added to TestCaseEvent data and used during TestRun execution. For example: perhaps there is a @CleanPolicy annotation and you want to extract information on how to clean the app/device between test runs using arguments within the annotation.

2. Customize RemoteAndroidTestRunner. For example: perhaps you want to add additional custom runner arguments.

3. Add custom ITestRunListener. This is pretty simple, just any additional functionality/reporting you want to do could be achieved via a simple ability to add custom ITestRunListeners.

Those are just 3 specific examples. I have developed a couple simple interfaces that can be utilized to achieve these tasks:

First, add fork configuration option to register plugins forkPlugins:

fork {
            testPackage = "com.myorg"
            fallbackToScreenshots = true

            poolingStrategy {
                computed {
                    characteristic = "sw"
                    groups {
                        allDevices = 0
                    }
                }
            }

            // custom plugin implements ForkPlugin interface
            forkPlugins = [new CustomForkPlugin(logger)]
}

A Fork plugin has the following capabilities:

/**
 * Allows for participating in and extending Fork functionality.
 */
public interface ForkPlugin {
    /**
     * Called during execution of {@link com.shazam.fork.suite.TestSuiteLoader#convertToTestCaseEvent(ClassDefItem, AnnotationDirectoryItem, AnnotationDirectoryItem.MethodAnnotation)}.
     * It is called with all annotations available for the test method currently being inspected for information.
     * This hook allows for returning a {@link Map} of data that is stored within {@link com.shazam.fork.model.TestCaseEvent#metadata}.
     * This stored metadata can later be retrieved via {@link TestCaseEvent#getMetadata()}.
     * @param annotations All annotations for the test method currently being inspected for information.
     * @return Metadata information that can later be retrieved via {@link TestCaseEvent#getMetadata()}
     */
    @Nonnull
    Map<String, String> extractTestCaseEventMetadata(@Nonnull AnnotationItem[] annotations);

    /**
     * Called when a TestRun is created from a {@link TestCaseEvent}
     * This hook allows for creating a {@link ForkTestRunHook} that will be called during TestRun execution.
     * Using this hook allows for participating in and extending Fork TestRun functionality.
     * @return A (possibly null) {@link ForkTestRunHook} that will be added to a TestRun and called during execution.
     */
    @Nullable
    ForkTestRunHook createTestRunHook();
}

A Fork plugin can also create ForkTestRunHooks that allow for interacting with the system during TestRun execution.

/**
 * Allow for selectively modifying and listening to a TestRun and it's {@link RemoteAndroidTestRunner}.
 */
public interface ForkTestRunHook {
    /**
     * Called before TestRun invokes {@link RemoteAndroidTestRunner#run(Collection)} during execution.
     * This hook allows for using {@link TestCaseEvent} to make decisions on possible ways to configure
     * the {@link RemoteAndroidTestRunner} in a customized way.
     * @param testCaseEvent The currently executing test case information
     * @param remoteAndroidTestRunner The test runner that is about to have run invoked
     */
    void onExecute(@Nonnull TestCaseEvent testCaseEvent,
                   @Nonnull RemoteAndroidTestRunner remoteAndroidTestRunner);

    /**
     * Called before TestRun invokes {@link RemoteAndroidTestRunner#run(Collection)} during execution.
     * This hook allows for adding a custom listener to respond to events during a test run lifecycle.
     * @return An (possible null) {@link ITestRunListener} that will be handed to {@link RemoteAndroidTestRunner#run(Collection)}
     */
    @Nullable
    ITestRunListener createTestRunListener();
}

Would love to discuss this further (perhaps there is a slack or some other channel) and get an implementation mainlined into Fork. Let me know your thoughts..