Occasional Missing Value with CachedObservable

[benjchristensen]

A bug report from https://groups.google.com/forum/#!topic/hystrixoss/sBjgpcMWch4 revealed an issue that I have tracked down to interaction between request caching and RxJava ReplaySubject.

This will reveal the bug non-deterministically. It started in RxJava 0.17. This bug does not express itself if I remove use of Observable.cache().

Thank you @chrismathews for the bug report and unit test to demonstrate it.

import static org.junit.Assert.assertEquals;

import java.util.*;
import java.util.concurrent.*;

import org.junit.Test;
import org.slf4j.*;

import com.netflix.hystrix.*;

public class SimpleHystrixCommandTest {
    private static final Logger logger = LoggerFactory.getLogger(SimpleHystrixCommandTest.class);

    @Test
    public void testSimpleHystrixCommand() throws Exception {
        int executionCounter = 0;
        int successCounter = 0;
        int fallbackCounter = 0;
        int nullCounter = 0;

        final int executorThreadPoolSize = 10;
        final int executionCount = 500000;
        ExecutorService executor = null;
        try {
            executor = Executors.newFixedThreadPool(executorThreadPoolSize);

            final List<Future<Boolean>> futures = new ArrayList<Future<Boolean>>();
            for (int i = 1; i <= executionCount; i++) {
                final int c = i;
                futures.add(executor.submit(new Callable<Boolean>() {

                    @Override
                    public Boolean call() throws Exception {
                        SimpleHystrixCommand cmd = new SimpleHystrixCommand(Boolean.TRUE);
                        Boolean b = cmd.execute();
                        if (b == null) {
                            System.err.println(c + " NULL events: " + cmd.getExecutionEvents() + " " + cmd.getExecutionTimeInMilliseconds());
                        }
                        return b;
                    }

                }));
            }
            for (final Future<Boolean> future : futures) {
                final Boolean response = future.get();
                executionCounter++;
                if (response == null) {
                    System.err.println("Null Response received on Execution=[" + executionCounter + "]");
                    nullCounter++;
                } else {
                    if (response) {
                        successCounter++;
                    } else {
                        fallbackCounter++;
                    }
                }
            }
        } finally {
            if (executor != null) {
                executor.shutdown();
            }
        }
        System.err.println("Success Count= " + successCounter);
        System.err.println("Fallback Count= " + fallbackCounter);
        System.err.println("Null Count= " + nullCounter);
        assertEquals(0, nullCounter);
    }

    private static class SimpleHystrixCommand extends HystrixCommand<Boolean> {
        private final Boolean response;

        public SimpleHystrixCommand(final Boolean response) {
            super(HystrixCommandGroupKey.Factory.asKey("SimpleHystrixCommand"));
            this.response = response;
        }

        @Override
        protected Boolean run() throws Exception {
            return response;
        }

        @Override
        protected Boolean getFallback() {
            return Boolean.FALSE;
        }
    }
}

[benjchristensen]

Looks like we have a fix done in ReactiveX/RxJava#1147. This will be released in RxJava 0.18.2 and I'll release Hystrix 1.3.16 with that version, along with the optimization to not use the cache when getCacheKey() is not implemented.

[benjchristensen]

Fixed in release 1.3.16 https://github.com/Netflix/Hystrix/releases/tag/1.3.16