Add tests for ssl/tls #162

lburgazzoli · 2017-07-12T06:19:12Z

No description provided.

fanminshi · 2017-07-12T23:41:56Z

https://github.com/grpc/grpc-java/blob/master/interop-testing/src/test/java/io/grpc/testing/integration/TlsTest.java

there is a example of TlsTest from grpc. Maybe we can get some idea for that.

yangliucheng · 2017-09-07T03:37:09Z

I use the example create a HTTPS client for JETCD,but it did not work very well:
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.lang.IllegalArgumentException: File does not contain valid private key: D:\kubernetes\server_key_decode.pem
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)
at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:222)
at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:54)
at com.yangliucheng.com.dilimiter.cache.DiEtcd.(DiEtcd.java:31)
at com.yangliucheng.com.dilimiter.cache.DiEtcd.create(DiEtcd.java:59)

DamianCummins · 2017-09-12T09:02:21Z

@lburgazzoli
@fanminshi

I'm trying to migrate from etcd v2 to v3 using the jetcd library but struggling without examples for SslContext configuration.

When I attempt to connect using the SSL certificate (caCert in the code below) I hit the following error:

com.coreos.jetcd.exception.EtcdException: Channel closed while performing protocol negotiation
at com.coreos.jetcd.exception.EtcdExceptionFactory.newEtcdException(EtcdExceptionFactory.java:34)
at com.coreos.jetcd.exception.EtcdExceptionFactory.fromStatus(EtcdExceptionFactory.java:82)
at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:78)
at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:73)
at com.coreos.jetcd.internal.impl.ClientConnectionManager.generateToken(ClientConnectionManager.java:236)
at com.coreos.jetcd.internal.impl.ClientConnectionManager.getToken(ClientConnectionManager.java:112)
at com.coreos.jetcd.internal.impl.ClientConnectionManager.access$100(ClientConnectionManager.java:56)
at com.coreos.jetcd.internal.impl.ClientConnectionManager$AuthTokenInterceptor$1.start(ClientConnectionManager.java:255)
at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:261)
at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:237)
at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:171)
at com.coreos.jetcd.api.LeaseGrpc$LeaseFutureStub.leaseGrant(LeaseGrpc.java:327)
at com.coreos.jetcd.internal.impl.LeaseImpl.lambda$grant$0(LeaseImpl.java:107)
at com.coreos.jetcd.internal.impl.LeaseImpl$$Lambda$16.000000000F6154F0.get(Unknown Source)
at com.coreos.jetcd.internal.impl.Util.toCompletableFutureWithRetry(Util.java:104)
at com.coreos.jetcd.internal.impl.LeaseImpl.grant(LeaseImpl.java:106)
...

If I pass in InsecureTrustManagerFactory.INSTANCE then it is successful.

FWIW this succeeds via etcdctl.

Please see the code snippet below. Any guidance here would be much appreciated.

try {
   // Passing in the certificate causes an connection error
   final byte[] sslBytes = Base64.getDecoder().decode(etcdSSL_.getBytes("UTF-8"));
   final InputStream is = new ByteArrayInputStream(sslBytes);
   final CertificateFactory cf = CertificateFactory.getInstance("X.509");
   final X509Certificate caCert = (X509Certificate) cf.generateCertificate(is);

   final String[] usernamepassword = etcdURL_.substring(8, etcdURL_.lastIndexOf("@")).split(":");
   final String url = etcdURL_.replace(usernamepassword[0] + ":" + usernamepassword[1] + "@", "");
   final ApplicationProtocolConfig apc = new ApplicationProtocolConfig(Protocol.ALPN, SelectorFailureBehavior.NO_ADVERTISE, SelectedListenerFailureBehavior.ACCEPT, ApplicationProtocolNames.HTTP_2);

   // TODO Pass in InsecureTrustManagerFactory.INSTANCE instead of caCert for
   // the moment
   final SslContext sslC = GrpcSslContexts.forClient().sslProvider(SslProvider.OPENSSL).ciphers(Http2SecurityUtil.CIPHERS, SupportedCipherSuiteFilter.INSTANCE).trustManager(InsecureTrustManagerFactory.INSTANCE).applicationProtocolConfig(apc).build();

   final Client client = Client.builder().endpoints(url).user(ByteSequence.fromString(usernamepassword[0])).password(ByteSequence.fromString(usernamepassword[1])).sslContext(sslC).lazyInitialization(false).build();   

}
catch (final Exception e) {
   ...
}

fanminshi · 2017-09-12T17:25:19Z

@yangliucheng @DamianCummins I am looking into this.

davissp14 · 2017-09-20T18:21:08Z

@fanminshi Any updates?

DamianCummins · 2017-09-28T11:31:19Z

@fanminshi Some further investigation on the above and it appears there are two issues:

I forked this repo and added the above ssl configuration to the setUp() method of the `/jetcd-core/src/test/java/com/coreos/jetcd/internal/impl/KVTest.java" class.

FWIW I'm using the following:

        <dependency>
          <groupId>io.netty</groupId>
          <artifactId>netty-tcnative-boringssl-static</artifactId>
          <version>2.0.5.Final</version>
          <classifier>windows-x86_64</classifier>
          <scope>test</scope>
        </dependency>

If I run mvn -Dtest=com.coreos.jetcd.internal.impl.KVTest.java package then I hit the following problem:
Issue 1:

com.coreos.jetcd.exception.EtcdException
        at com.coreos.jetcd.exception.EtcdExceptionFactory.newEtcdException(EtcdExceptionFactory.java:34)
        at com.coreos.jetcd.exception.EtcdExceptionFactory.fromStatus(EtcdExceptionFactory.java:82)
        at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:78)
        at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:73)
        at com.coreos.jetcd.internal.impl.ClientConnectionManager.generateToken(ClientConnectionManager.java:236)
        at com.coreos.jetcd.internal.impl.ClientConnectionManager.getToken(ClientConnectionManager.java:112)
        at com.coreos.jetcd.internal.impl.ClientConnectionManager.access$100(ClientConnectionManager.java:56)
        at com.coreos.jetcd.internal.impl.ClientConnectionManager$AuthTokenInterceptor$1.start(ClientConnectionManager.java:255)
        at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:259)
        at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:235)
        at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:169)
        at com.coreos.jetcd.api.KVGrpc$KVFutureStub.put(KVGrpc.java:417)
        at com.coreos.jetcd.internal.impl.KVImpl.lambda$put$0(KVImpl.java:77)
        at com.coreos.jetcd.internal.impl.KVImpl$$Lambda$11.000000001249F4E0.get(Unknown Source)
        at com.coreos.jetcd.internal.impl.Util.toCompletableFutureWithRetry(Util.java:104)
        at com.coreos.jetcd.internal.impl.KVImpl.put(KVImpl.java:76)
        at com.coreos.jetcd.internal.impl.KVImpl.put(KVImpl.java:59)
        at com.coreos.jetcd.internal.impl.KVTest.testPut(KVTest.java:110)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:508)
        at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
        at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
        at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
        at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
        at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
        at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
        at org.testng.TestRunner.privateRun(TestRunner.java:744)
        at org.testng.TestRunner.run(TestRunner.java:602)
        at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
        at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
        at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
        at org.testng.SuiteRunner.run(SuiteRunner.java:289)
        at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
        at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
        at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
        at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
        at org.testng.TestNG.runSuites(TestNG.java:1144)
        at org.testng.TestNG.run(TestNG.java:1115)
        at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
        at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
        at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
        at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
        at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:647)
        at io.netty.internal.tcnative.SSL.readFromSSL(Native Method)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.readPlaintextData(ReferenceCountedOpenSslEngine.java:481)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1019)
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1126)
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:210)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:785)
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching etcd found.
        at com.ibm.jsse2.util.b.c(b.java:108)
        at com.ibm.jsse2.util.b.a(b.java:19)
        at com.ibm.jsse2.aA.a(aA.java:98)
        at com.ibm.jsse2.aA.a(aA.java:148)
        at com.ibm.jsse2.aA.checkServerTrusted(aA.java:51)
        at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:221)
        at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:643)
        ... 26 more

At this stage I'll point out that I am using an instance of etcd managed by IBM Compose, however seeing as etcdctl connects successfully, this is not an issue with what Compose are providing.

I worked around this by changing the hardcoded authority from "etcd" to my etcd hostname in the SmartNameResolver:

public SmartNameResolver(List<URI> uris) {
    this.lock = new Object();
    this.authority = "etcd"; //<--
    this.uris = uris;
    this.resolvers = new ArrayList<>();

Links I looked at : https://github.com/grpc/grpc/blob/master/doc/naming.md , https://groups.google.com/forum/#!topic/grpc-io/xtwACYjZ2p8

So, at this point, if I re run mvn -Dtest=com.coreos.jetcd.internal.impl.KVTest.java package then the KVTests run successfully!

However, if I then run the same tests in eclipse using the TestNG plugin, I get this error (same as the one reported in the previous comment):

com.coreos.jetcd.exception.EtcdException: Channel closed while performing protocol negotiation
	at com.coreos.jetcd.exception.EtcdExceptionFactory.newEtcdException(EtcdExceptionFactory.java:34)
	at com.coreos.jetcd.exception.EtcdExceptionFactory.fromStatus(EtcdExceptionFactory.java:82)
	at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:78)
	at com.coreos.jetcd.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:73)
	at com.coreos.jetcd.internal.impl.ClientConnectionManager.generateToken(ClientConnectionManager.java:236)
	at com.coreos.jetcd.internal.impl.ClientConnectionManager.getToken(ClientConnectionManager.java:112)
	at com.coreos.jetcd.internal.impl.ClientConnectionManager.access$100(ClientConnectionManager.java:56)
	at com.coreos.jetcd.internal.impl.ClientConnectionManager$AuthTokenInterceptor$1.start(ClientConnectionManager.java:255)
	at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:261)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:237)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:171)
	at com.coreos.jetcd.api.KVGrpc$KVFutureStub.put(KVGrpc.java:417)
	at com.coreos.jetcd.internal.impl.KVImpl.lambda$put$0(KVImpl.java:77)
	at com.coreos.jetcd.internal.impl.KVImpl$$Lambda$11.0000000011A0BBE0.get(Unknown Source)
	at com.coreos.jetcd.internal.impl.Util.toCompletableFutureWithRetry(Util.java:104)
	at com.coreos.jetcd.internal.impl.KVImpl.put(KVImpl.java:76)
	at com.coreos.jetcd.internal.impl.KVImpl.put(KVImpl.java:59)
	at com.coreos.jetcd.internal.impl.KVTest.testPut(KVTest.java:110)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:108)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:661)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:869)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1193)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:126)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:744)
	at org.testng.TestRunner.run(TestRunner.java:602)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:380)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:375)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:340)
	at org.testng.SuiteRunner.run(SuiteRunner.java:289)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1301)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1226)
	at org.testng.TestNG.runSuites(TestNG.java:1144)
	at org.testng.TestNG.run(TestNG.java:1115)
	at org.testng.remote.AbstractRemoteTestNG.run(AbstractRemoteTestNG.java:114)
	at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:251)
	at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:77)

So there is different behaviour between mvn command line and running in eclipse?

Is this perhaps due to the OS specific netty library?

Other notable environment settings:
I have MINGW64 installed.
I added System.out.println(OpenSsl.supportsKeyManagerFactory()); to the setup and this consistently returns false whether run through maven command line or TestNG in eclipse.

lburgazzoli · 2017-09-28T15:02:24Z

I guess the first thing we need to do is to add the authority as builder option, make sense ?

lburgazzoli · 2017-09-29T06:51:14Z

@DamianCummins do you have by chance any time to send a PR ?

DamianCummins · 2017-11-13T12:30:36Z

Hi @fanminshi - do you have an eta on when this will make it into a dev snapshot / release?

DamianCummins · 2017-11-20T15:23:59Z

Any update on a dev snapshot / release? @fanminshi cc: @xiang90 - thanks

davissp14 · 2017-11-21T22:34:04Z

@xiang90 @fanminshi Hey guys, this is a big one for us as well. Any information on when we can get this out would be super helpful! Appreciate it!

fanminshi · 2017-11-21T22:38:16Z

@DamianCummins @DamianCummins Sorry for the delay. I'll try to get this into the dev snapshot this week. For the release, I'll think there are few tasks needs to be done for 0.02 release. I'll evaluate those and get you back later.

davissp14 · 2017-11-22T17:00:40Z

@fanminshi @xiang90

Would you consider cutting a patch release for this specific issue and rolling the remaining issues in https://github.com/coreos/jetcd/milestone/2 into the next release 0.0.3. The remaining issues appear non-critical and given the client is in a non-working state for V3 users looking to leverage TLS, it seems like it would probably be worth expediting.

Thoughts?

gtudan · 2017-11-23T07:00:38Z

I took a look at the newly added authority option in the client build, but I'm not sure how to use it in combination with the multiple endpoints.

If we're using a single endpoint, I can just pass the host name as authority and everything is fine. But what if I have more than one endpoint? Should I issue a certificate with an additional SAN like "etcd"? That's probably close to impossible with "official" CAs.

To be honest, I don't quiet get why it is needed at all. The authority is used for Server Name Indication - is that really necessary, as etcd only serves a single certificate? Why not just connect, get the certificate and validate it against the endpoint uri using the defaut trust manager? I looked at the grpc-example here https://grpc.io/docs/guides/auth.html and that's pretty much what google recommends. Or is the authority somewhat related to smart name resolving?

lburgazzoli · 2017-11-23T09:18:27Z

For single endpoint, yes the issue is the SmartNameResolver so I will improve it but for multiple endpoint the issue is that the NameResolver can't provide a an authority per sub channel, see the following discussion: grpc/grpc-java#2662

I don't have much time to investigate the issue further but I'll do my best, any hel is very appreciated.

lburgazzoli mentioned this issue Jul 12, 2017

Make it easy to configure ssl #161

Merged

fanminshi assigned lburgazzoli Jul 31, 2017

lburgazzoli added a commit to lburgazzoli/etcd-io-jetcd that referenced this issue Sep 29, 2017

test(ssl): add tests for ssl/tls etcd-io#162

49f79f7

lburgazzoli mentioned this issue Sep 29, 2017

test(ssl): add tests for ssl/tls #162 #245

Merged

lburgazzoli added a commit to lburgazzoli/etcd-io-jetcd that referenced this issue Sep 30, 2017

test(ssl): add tests for ssl/tls etcd-io#162

769f51f

lburgazzoli added a commit to lburgazzoli/etcd-io-jetcd that referenced this issue Oct 5, 2017

test(ssl): add tests for ssl/tls etcd-io#162

2296d59

fanminshi closed this as completed in #245 Oct 5, 2017

fanminshi pushed a commit that referenced this issue Oct 5, 2017

test(ssl): add tests for ssl/tls #162

1906f20

This was referenced Oct 5, 2017

How can I get the sample code contain the HTTPS #232

Closed

client: add doc on how to set up tls #164

Closed

test java client #87

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tests for ssl/tls #162

Add tests for ssl/tls #162

lburgazzoli commented Jul 12, 2017

fanminshi commented Jul 12, 2017

yangliucheng commented Sep 7, 2017

DamianCummins commented Sep 12, 2017 •

edited

Loading

fanminshi commented Sep 12, 2017

davissp14 commented Sep 20, 2017

DamianCummins commented Sep 28, 2017 •

edited

Loading

lburgazzoli commented Sep 28, 2017

lburgazzoli commented Sep 29, 2017

DamianCummins commented Nov 13, 2017

DamianCummins commented Nov 20, 2017

davissp14 commented Nov 21, 2017 •

edited

Loading

fanminshi commented Nov 21, 2017

davissp14 commented Nov 22, 2017 •

edited

Loading

gtudan commented Nov 23, 2017

lburgazzoli commented Nov 23, 2017

Add tests for ssl/tls #162

Add tests for ssl/tls #162

Comments

lburgazzoli commented Jul 12, 2017

fanminshi commented Jul 12, 2017

yangliucheng commented Sep 7, 2017

DamianCummins commented Sep 12, 2017 • edited Loading

fanminshi commented Sep 12, 2017

davissp14 commented Sep 20, 2017

DamianCummins commented Sep 28, 2017 • edited Loading

lburgazzoli commented Sep 28, 2017

lburgazzoli commented Sep 29, 2017

DamianCummins commented Nov 13, 2017

DamianCummins commented Nov 20, 2017

davissp14 commented Nov 21, 2017 • edited Loading

fanminshi commented Nov 21, 2017

davissp14 commented Nov 22, 2017 • edited Loading

gtudan commented Nov 23, 2017

lburgazzoli commented Nov 23, 2017

DamianCummins commented Sep 12, 2017 •

edited

Loading

DamianCummins commented Sep 28, 2017 •

edited

Loading

davissp14 commented Nov 21, 2017 •

edited

Loading

davissp14 commented Nov 22, 2017 •

edited

Loading