Datastore emulator support

[myashchenko]

Please add a possibility to set emulator host for datastore (as it's implemented for PubSub). Also, if emulator host is set, instantiate NoCredentialsProvider instead of DefaultCredentialsProvider

[dzou]

Hi, just created #1445 to solve this problem.

Regarding the CredentialsProvider fix, this may not be possible. I made a few attempts:

• I tried to provide NoCredentialsProvider when the emulator is used, however it doesn't appear to quite work. You will get a NullPointerException when the Datastore object (aka. the Google client library object to interface with Datastore) is being created; it seems like the Datastore factory class actually attempts to use the credentials.
• I also tried to instantiate a CredentialsProvider which returned empty credentials, however this also seemed to fail as you will get error about how Oauth credential needs to be refreshed.

Given this, it looks like part of using the emulator is to also verify that your credentials are correct; this behavior may actually be useful. So I think the most prudent choice here is to just keep it DefaultCredentialsProvider; this still keeps the option open to allow users to provide their own CredentialsProvider bean to override if they wish to do so.

[myashchenko]

@dzou as of now, I use the following custom configuration along with Datastore emulator:

@Configuration
@ConditionalOnProperty(name = "spring.cloud.gcp.emulator-enabled", havingValue = "true")
public class GcpConfiguration {

    @Bean
    public CredentialsProvider credentialsProvider() {
        return NoCredentials::getInstance;
    }
}

Works like a charm.

I think you might use it as well

[dzou]

Nice one, that works for me. OK, I would be open to including this in our configuration.

I will check in with our team to confirm the approach.

@meltsufin, @artembilan: What are your thoughts about including a NoCredentials provider bean if an emulator is installed? The main question here is that if we override the CredentialsProvider bean, this will affect usage of entire Spring Cloud GCP library. Is there risk of user wanting to use emulator with real GCP services in test situation? (such as if user wanted to use datastore emulator + real stackdriver logging, etc.)

[artembilan]

this will affect usage of entire Spring Cloud GCP library

Well, we are here in the opinionated auto-configuration and having @ConditionalOnBean we always leave a room for end-users to override our NoCredentials for his/own purpose in the additional @Configuration for the target test.
On the other hand we may introduce a configuration option like spring.cloud.gcp.emulator.no-credentials=true/false.

[meltsufin]

In PubSubAutoConfiguration constructor, we have this logic:

		if (gcpPubSubProperties.getEmulatorHost() == null
				|| "false".equals(gcpPubSubProperties.getEmulatorHost())) {
			this.finalCredentialsProvider = gcpPubSubProperties.getCredentials().hasKey()
					? new DefaultCredentialsProvider(gcpPubSubProperties)
					: credentialsProvider;
		}
		else {
			// Since we cannot create a general NoCredentialsProvider if the emulator host is enabled
			// (because it would also be used for the other components), we have to create one here
			// for this particular case.
			this.finalCredentialsProvider = NoCredentialsProvider.create();
		}

We should do the same for Datastore. It will not override the global credentials provider.

[dzou]

OK, sounds good. I have revised my PR to adopt this style.

[lucasoares]

Hello. I have a question and I don't know if I should open another ticket.

Google libraries have a good class to start the emulator through java. For datastore this class is LocalDatastoreHelper.java.

With this new support for datastore-emulator the host must have to be defined statically in application.properties. The problem is the emulator helper starts search for a random free port to start the emulator and the datastore connection for spring data cannot connect to the datastore instance.

Do you guys have any kind of helper similar to LocalDatastoreHelper.java that uses the spring.cloud.gcp.datastore.emulator-host properties?

[dzou]

@lucasoares

Hey thanks for posting the question. Is it possible to startup your emulator with a predetermined port via. --port flag?

I.e. in an ideal scenario you could set your emulator and application.properties to the same port.

However, if this solution doesn't work for you, please feel free to file a new issue on our github just so we don't forget to look deeper into it; often times comments on older issues will get missed.

Thanks.

[meltsufin]

@lucasoares Using the port flag that @dzou suggested should solve your issue.
However, maybe you're also suggesting that we should have a configuration property like spring.cloud.gcp.datastore.emulator=true that will then automatically start-up the emulator and configure the port. It's definitely an idea we can pursue and utilize the mentioned LocalDatastoreHelper to accomplish it. If you think that would be useful, we can open an issue for it.

[lucasoares]

@meltsufin yes, exactly that..

@dzou unfortunately the LocalDatastoreHelper have a hardcoded way to determine the port itself. It can not be changed outside the class. Will be impossible to start the emulator with a custom fixed port:
LocalDatastoreHelper port usage
LocalDatastoreHelper port initialization
BaseEmulatorHelper port finder

I will open a new issue and also create a PR to google-cloud-java lib to make the emulator accept a custom port to make this new configuration possible.

Thank you guys.

[meltsufin]

LocalDatastoreHelper should definitely be changed to accept a user-defined port. Please link to the issue you create for it here. I will also create an issue in our repo to track auto-configuration of the Datastore emulator.

[meltsufin]

@lucasoares I created the issue #1642. Contributions are always welcomed! 😉

[lucasoares]

Google's clients now accepts a custom port. We just need to wait for their release haha

After that I can create a PR here to auto configure the datastore emulator after using a custom properties.