Upgrade GCS client to 2.2.8 #3507

elonazoulay · 2020-04-22T12:03:10Z

This fixes the issue with performance cache: GoogleCloudDataproc/hadoop-connectors#342

medb · 2020-04-30T18:06:16Z

pom.xml

@@ -1010,6 +1030,20 @@
                <groupId>com.google.cloud.bigdataoss</groupId>
                <artifactId>gcsio</artifactId>


If you are using GCS connector only through HDFS/HCFS API you can depend on shaded JAR instead (<classifier>shaded</classifier>). This way you will have only one dependency instead of 4 dependencies now.

electrum · 2020-05-12T23:26:30Z

pom.xml

@@ -51,7 +51,7 @@
        <dep.jdbi3.version>3.4.0</dep.jdbi3.version>
        <dep.drift.version>1.14</dep.drift.version>
        <dep.tempto.version>174</dep.tempto.version>
-        <dep.gcs.version>2.0.0</dep.gcs.version>
+        <dep.gcs.version>2.1.1</dep.gcs.version>


2.1.3 is the latest, should we use that instead?

Yep, thanks!

pom.xml

electrum · 2020-05-12T23:52:35Z

pom.xml

+            <dependency>
+                <groupId>com.google.http-client</groupId>
+                <artifactId>google-http-client</artifactId>
+                <version>1.34.2</version>


Latest is 1.35.0

No longer needed with the shaded gcs-connector jar.

pom.xml

medb · 2020-05-14T22:49:25Z

pom.xml

+            <dependency>
+                <groupId>com.google.cloud.bigdataoss</groupId>
+                <artifactId>util-hadoop</artifactId>
+                <version>hadoop2-${dep.gcs.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>com.google.cloud.bigdataoss</groupId>
+                <artifactId>gcsio</artifactId>
+                <version>${dep.gcs.version}</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.httpcomponents</groupId>
+                <artifactId>httpclient</artifactId>
+                <version>4.5.11</version>
+            </dependency>
+
+            <dependency>
+                <groupId>org.apache.httpcomponents</groupId>
+                <artifactId>httpcore</artifactId>
+                <version>4.4.13</version>
+            </dependency>
+
+            <dependency>
+                <groupId>com.google.code.gson</groupId>
+                <artifactId>gson</artifactId>
+                <version>2.8.6</version>
+            </dependency>
+


All these dependencies should be redundant if you are using shaded GCS connector.

They are there due to enforcer errors. Without those dependencies amazon-aws-sdk and a few other artifacts complain about upper bounds.

If the versions for gson, httpclient and httpcore dependencies are not explicitly defined in dependency management, maven-enforcer fails with the following upper bound dependency violations:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireUpperBoundDeps failed with message: Failed while enforcing RequireUpperBoundDeps. The error(s) are [ Require upper bound dependencies error for org.apache.httpcomponents:httpclient:4.5.9 paths to dependency are: +-io.prestosql:presto-hive:334-SNAPSHOT +-com.amazonaws:aws-java-sdk-core:1.11.749 +-org.apache.httpcomponents:httpclient:4.5.9 and +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.1.3 +-com.google.oauth-client:google-oauth-client:1.30.6 +-com.google.http-client:google-http-client:1.34.2 +-org.apache.httpcomponents:httpclient:4.5.11 , Require upper bound dependencies error for org.apache.httpcomponents:httpcore:4.4.11 paths to dependency are: +-io.prestosql:presto-hive:334-SNAPSHOT +-com.amazonaws:aws-java-sdk-core:1.11.749 +-org.apache.httpcomponents:httpclient:4.5.9 +-org.apache.httpcomponents:httpcore:4.4.11 and +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.1.3 +-com.google.oauth-client:google-oauth-client:1.30.6 +-com.google.http-client:google-http-client:1.34.2 +-org.apache.httpcomponents:httpcore:4.4.13 , Require upper bound dependencies error for com.google.code.gson:gson:2.2.4 paths to dependency are: +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcsio:2.1.3 +-com.google.code.gson:gson:2.2.4 and +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcsio:2.1.3 +-com.google.protobuf:protobuf-java-util:3.11.4 +-com.google.code.gson:gson:2.8.6 and +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcsio:2.1.3 +-io.grpc:grpc-alts:1.29.0 +-io.grpc:grpc-core:1.29.0 +-com.google.code.gson:gson:2.8.6

If the gcsio version is not explicitly defined in dependencyManagement, maven-enforcer fails with the following upper bound dependency violations:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireUpperBoundDeps failed with message: Failed while enforcing RequireUpperBoundDeps. The error(s) are [ Require upper bound dependencies error for io.grpc:grpc-context:1.22.1 paths to dependency are: +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.1.3 +-com.google.oauth-client:google-oauth-client:1.30.6 +-com.google.http-client:google-http-client:1.34.2 +-io.opencensus:opencensus-api:0.24.0 +-io.grpc:grpc-context:1.22.1 and +-io.prestosql:presto-hive:334-SNAPSHOT +-com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.1.3 +-com.google.cloud.bigdataoss:gcsio:2.1.3 +-io.grpc:grpc-auth:1.29.0 +-io.grpc:grpc-api:1.29.0 +-io.grpc:grpc-context:1.29.0

If util-hadoop is not explicitly defined then presto-hive fails with an used but undeclared dependency error.

I think it depends on next Presto release date, if it scheduled after new GCS connector release in 2 weeks or so, then it makes sense to wait and update straight to shaded GCS connector 2.1.4. But if next Presto release is earlier, then we can go ahead with unshaded artifacts for now, I think.

I'll update the pr with non-shaded artifacts and save this one, can always go back to it - we'll see which one happens first:)

Thanks @medb!

GCS connector 2.1.4 was just released, so you can update this PR to use it now.

elonazoulay · 2020-08-05T19:36:12Z

Hi @medb, update: tried 2.1.4 locally, will post issues - one of them is that gcsio.GoogleCloudStorageFileSystem is not repackaged in com.google.cloud.hadoop.fs.gcs so it still needs to be referred to as com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

medb · 2020-08-05T21:15:43Z

Hi @medb, update: tried 2.1.4 locally, will post issues - one of them is that gcsio.GoogleCloudStorageFileSystem is not repackaged in com.google.cloud.hadoop.fs.gcs so it still needs to be referred to as com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem

Yep, if you directly depend on gcsio package then you should not use shaded GCS connector. I thought that Presto depends on HCFS/HDFS for interface in GCS connector for GCS access.

Why there a need to directly use classes in gcsio? Usually if you use classes in gcsio then you should use only gcsio library, not GCS connector that implements HDFS/HCFS API on top of it (i.e. you should depend directly only on either gcs-connector or gcsio library, not on both of them).

elonazoulay · 2020-08-06T14:06:08Z

It's only. used to get the scheme in GcsConfigurationProvider, i.e. GoogleCloudStorageFileSystem.SCHEME.

Also, I noticed that if I select with a limit then I see the following stack trace in the logs, but the query succeeds, is there some config that would avoid the stack trace?

Here it is:

java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
	at java.base/java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:390)
	at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:385)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.read(GoogleHadoopFSInputStream.java:131)
	at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
	at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.read(GoogleHadoopFSInputStream.java:161)
	at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
	at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
	at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
	at io.prestosql.plugin.hive.parquet.HdfsParquetDataSource.readFully(HdfsParquetDataSource.java:114)
	at io.prestosql.plugin.hive.parquet.HdfsParquetDataSource$ReferenceCountedReader.read(HdfsParquetDataSource.java:290)
	at io.prestosql.plugin.hive.parquet.HdfsParquetDataSource$1.read(HdfsParquetDataSource.java:187)
	at io.prestosql.parquet.reader.ParquetReader.readPrimitive(ParquetReader.java:249)
	at io.prestosql.parquet.reader.ParquetReader.readColumnChunk(ParquetReader.java:310)
	at io.prestosql.parquet.reader.ParquetReader.readBlock(ParquetReader.java:293)
	at io.prestosql.plugin.hive.parquet.ParquetPageSource$ParquetBlockLoader.load(ParquetPageSource.java:164)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
	at io.prestosql.spi.Page.getLoadedPage(Page.java:273)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:304)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_unknown____20200806_135605_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

elonazoulay · 2020-08-06T19:35:18Z

Looks like the stack trace is ok, it does this for the current version as well. It definitely doesn't result in query failures.
@medb, @ebyhr when you have a chance let me know if this is ok.

medb · 2020-08-07T19:54:23Z

Yeah, we don't have SCHEME constant in GCS connector directly, added it in GoogleCloudDataproc/hadoop-connectors#426, so it will be in 2.1.5 release.

presto-hive/src/main/java/io/prestosql/plugin/hive/gcs/GoogleGcsConfigurationInitializer.java

elonazoulay · 2020-09-14T17:39:13Z

Looks like 2.1.5 was just released, will update shortly.

ebyhr · 2022-09-02T02:40:04Z

Sent #13969 within the repository.

elonazoulay · 2022-09-15T07:23:49Z

Hey @ebyhr @brokenjacobs @medb - I did some testing and cross cloud (from trino in azure -> gcs) there was a drastic latency increase in gcs calls due to keep alive cache. I added the Connection:close custom header and the latency disappeared but http requests would fail and queries were unreliable, tried setting http retries to higher (20k) values but it was still unreliable.
Here's a thread dump - all threads but 1 are blocked due to trying to put the client back in the keep alive cache:
The keep alive cache uses a synchronized method to put the client back in the cache.

 java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.net.www.http.KeepAliveCache.put([email protected]/KeepAliveCache.java:89)
        - waiting to lock <0x00007fa656a70650> (a sun.net.www.http.KeepAliveCache)
        at sun.net.www.protocol.https.HttpsClient.putInKeepAliveCache([email protected]/HttpsClient.java:663)
        at sun.net.www.http.HttpClient.finished([email protected]/HttpClient.java:440)
        at sun.net.www.http.KeepAliveStream.close([email protected]/KeepAliveStream.java:99)
        at sun.net.www.MeteredStream.justRead([email protected]/MeteredStream.java:93)
        at sun.net.www.MeteredStream.read([email protected]/MeteredStream.java:135)
        - locked <0x00007fb6533639e0> (a sun.net.www.http.KeepAliveStream)
        at java.io.FilterInputStream.read([email protected]/FilterInputStream.java:133)
        at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read([email protected]/HttpURLConnection.java:3525)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.cacheFooter(GoogleCloudStorageReadChannel.java:802)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openStream(GoogleCloudStorageReadChannel.java:980)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.openContentChannel(GoogleCloudStorageReadChannel.java:712)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.performLazySeek(GoogleCloudStorageReadChannel.java:703)
        at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageReadChannel.read(GoogleCloudStorageReadChannel.java:306)
        at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream.read(GoogleHadoopFSInputStream.java:118)

Would you know if there is a way to disable the keepalive cache - found this - but I tried 2.1.6 and it had similar behavior.

Anyone else familiar with this issue?

This affected metadata calls most, not reads - and the use case was for reading tons of tiny files that make up hudi timelines (on the coordinator). For reading data there was no degradation.

brokenjacobs · 2022-09-15T12:57:53Z

Is it http or tcp keepalives causing your issue? That link seems to be referring to tcp keepalives and the sslsocketfactory not http keepalives. (Http 1.1 feature). It seems odd that http keepalives (implied by the custom header) would be causing delays?

elonazoulay · 2022-09-15T14:57:01Z

By disabling http keepalive (custom header) the connection is closed after 1 request. This causes the next request to fail and the code will create a new http client which eliminates the latency but is unreliable. Keep alive cache is causing locking which results in delays (specifically for the hudi use case w tons of tiny metadata files).

elonazoulay · 2022-09-15T15:06:12Z

So main issue is that all threads which use the http client are blocked except one due to the keep alive cache. The custom headers, etc. was just for testing but showed that the latency is due to the synchronized method which puts connections back in the keep alive cache

elonazoulay · 2022-09-15T15:16:04Z

Also, to be clear: since the behavior seems to be in older versions also, it is not a regresssion wtih 2.2.7.

alexjo2144 · 2022-09-15T15:52:01Z

Also, to be clear: since the behavior seems to be in older versions also, it is not a regresssion wtih 2.2.7.

Maybe we split this conversation into a separate GH Issue then?

elonazoulay · 2022-09-15T18:04:00Z

@alexjo2144 absolutely, I will create an issue for it. Might not be something people encounter outside of this specific use case (tons of tiny files, firewall, cross cloud, etc.).

lukasz-walkiewicz · 2022-09-16T12:56:30Z

pom.xml

@@ -60,7 +60,7 @@
        <dep.oracle.version>19.3.0.0</dep.oracle.version>
        <dep.drift.version>1.14</dep.drift.version>
        <dep.tempto.version>189</dep.tempto.version>
-        <dep.gcs.version>2.0.0</dep.gcs.version>
+        <dep.gcs.version>2.2.7</dep.gcs.version>


2.2.8 is already out

kokosing · 2022-09-19T11:08:05Z

pom.xml

@@ -1062,6 +1062,7 @@
                <groupId>com.google.cloud.bigdataoss</groupId>
                <artifactId>gcs-connector</artifactId>
                <version>hadoop2-${dep.gcs.version}</version>
+                <classifier>shaded</classifier>


I think it is because we can have issues with other dependencies without shading.

kokosing · 2022-09-19T11:09:50Z

pom.xml

@@ -1952,6 +1973,44 @@
                        <useNativeGit>true</useNativeGit>
                    </configuration>
                </plugin>
+                <plugin>
+                    <groupId>org.basepom.maven</groupId>
+                    <artifactId>duplicate-finder-maven-plugin</artifactId>


What it takes to fix issues instead of ignoring them? Can you please list issues that got without this?

Here are the issues:

[WARNING] Found duplicate (but equal) resources in [com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.8:jar:shaded, org.apache.httpcomponents:httpclient:4.5.13]: [WARNING] mozilla/public-suffix-list.txt [WARNING] Found duplicate classes/resources in compile classpath. [WARNING] Found duplicate (but equal) resources in [com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.8:jar:shaded, org.apache.httpcomponents:httpclient:4.5.13]: [WARNING] mozilla/public-suffix-list.txt [WARNING] Found duplicate classes/resources in runtime classpath. [WARNING] Found duplicate (but equal) resources in [com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.8:jar:shaded, org.apache.httpcomponents:httpclient:4.5.13]: [WARNING] mozilla/public-suffix-list.txt [WARNING] Found duplicate classes/resources in test classpath.

Second warning is from the trino-faulttolerant-tests module:

[WARNING] Found duplicate and different resources in [com.google.api:gax:2.17.0, com.google.cloud.bigdataoss:gcs-connector:hadoop2-2.2.8:jar:shaded]: [WARNING] dependencies.properties [WARNING] Found duplicate classes/resources in test classpath.

Thank you for the update.

kokosing · 2022-09-19T11:11:04Z

@elonazoulay Are you working on this actively or would like me or somebody else to take it over?

electrum · 2022-09-20T20:11:42Z

pom.xml

@@ -1124,33 +1125,53 @@
                        <groupId>org.slf4j</groupId>
                        <artifactId>slf4j-log4j12</artifactId>
                    </exclusion>


Can we use a wildcard exclusion here?

@kokosing either way works: I think the changes in 2.2.8 will help our use case and we were able to deploy 2.2.7 (this pr) recently and it worked as good or better than previous version we were using (2.2.2). Either way I will be testing 2.2.8 in the next day or so - lmk what works best for you. If you want to use the other pr I can test that as well.

Updated. I will remove the duplicate finder and paste the results as well and update after we deploy it. And if you want to use the other pr I will test it also.

kokosing · 2022-09-22T07:08:06Z

CI hit: #14248, retrying to verify if is truly flaky. EDIT it is truly flaky.

kokosing · 2022-09-22T08:15:58Z

Merged, thanks!

cla-bot bot added the cla-signed label Apr 22, 2020

elonazoulay requested review from ebyhr, electrum and findepi April 22, 2020 12:18

findepi removed their request for review April 25, 2020 21:37

medb reviewed Apr 30, 2020

View reviewed changes

electrum reviewed May 12, 2020

View reviewed changes

electrum reviewed May 13, 2020

View reviewed changes

elonazoulay force-pushed the update_gcs211 branch 2 times, most recently from 7092b42 to 29fb8ce Compare May 14, 2020 10:12

medb reviewed May 14, 2020

View reviewed changes

elonazoulay force-pushed the update_gcs211 branch 5 times, most recently from 7bfe57c to d398a5a Compare May 20, 2020 14:58

elonazoulay force-pushed the update_gcs211 branch from d398a5a to d8b2693 Compare May 23, 2020 16:45

elonazoulay force-pushed the update_gcs211 branch from d8b2693 to 4ce8a43 Compare August 6, 2020 14:01

elonazoulay changed the title ~~Upgrade gcs to 2.1.1~~ Upgrade gcs to 2.1.4 Aug 6, 2020

elonazoulay force-pushed the update_gcs211 branch from 4ce8a43 to 9510f5b Compare August 6, 2020 19:48

medb approved these changes Aug 7, 2020

View reviewed changes

presto-hive/src/main/java/io/prestosql/plugin/hive/gcs/GoogleGcsConfigurationInitializer.java Outdated Show resolved Hide resolved

presto-hive/src/main/java/io/prestosql/plugin/hive/gcs/GoogleGcsConfigurationInitializer.java Outdated Show resolved Hide resolved

elonazoulay force-pushed the update_gcs211 branch from 9510f5b to c13d592 Compare August 8, 2020 18:32

elonazoulay force-pushed the update_gcs211 branch from c13d592 to b82079f Compare August 21, 2020 20:30

elonazoulay force-pushed the update_gcs211 branch from b82079f to 453fff8 Compare November 3, 2020 02:20

elonazoulay force-pushed the update_gcs211 branch from 516fc87 to 4bba8f1 Compare August 31, 2022 18:56

ebyhr mentioned this pull request Sep 2, 2022

Test: Upgrade GCS client to 2.2.7 #13969

Closed

alexjo2144 mentioned this pull request Sep 15, 2022

Improve Iceberg Glue table caching #13875

Merged

findepi requested review from electrum and removed request for electrum September 16, 2022 12:23

kokosing self-requested a review September 16, 2022 12:47

lukasz-walkiewicz approved these changes Sep 16, 2022

View reviewed changes

kokosing reviewed Sep 19, 2022

View reviewed changes

electrum reviewed Sep 20, 2022

View reviewed changes

electrum approved these changes Sep 20, 2022

View reviewed changes

lukasz-walkiewicz mentioned this pull request Sep 21, 2022

Upgrade gcs to 2.2.8 #14232

Closed

Upgrade gcs to 2.2.8

2e2b668

elonazoulay force-pushed the update_gcs211 branch from 4bba8f1 to 2e2b668 Compare September 21, 2022 22:27

elonazoulay changed the title ~~Upgrade GCS client to 2.2.7~~ Upgrade GCS client to 2.2.8 Sep 21, 2022

kokosing approved these changes Sep 22, 2022

View reviewed changes

kokosing merged commit 34984f1 into trinodb:master Sep 22, 2022

findinpath mentioned this pull request Sep 22, 2022

Add Delta synchronizer for GCS #12523

Merged

github-actions bot added this to the 398 milestone Sep 22, 2022

colebow mentioned this pull request Sep 27, 2022

Add Trino 398 release notes #14319

Merged

		@@ -1010,6 +1030,20 @@
		<groupId>com.google.cloud.bigdataoss</groupId>
		<artifactId>gcsio</artifactId>

Upgrade GCS client to 2.2.8 #3507

Upgrade GCS client to 2.2.8 #3507

Conversation

elonazoulay commented Apr 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elonazoulay commented Aug 5, 2020 • edited Loading

medb commented Aug 5, 2020 • edited Loading

elonazoulay commented Aug 6, 2020 • edited Loading

elonazoulay commented Aug 6, 2020 • edited Loading

medb commented Aug 7, 2020

elonazoulay commented Sep 14, 2020

ebyhr commented Sep 2, 2022

elonazoulay commented Sep 15, 2022 • edited Loading

brokenjacobs commented Sep 15, 2022

elonazoulay commented Sep 15, 2022

elonazoulay commented Sep 15, 2022

elonazoulay commented Sep 15, 2022

alexjo2144 commented Sep 15, 2022

elonazoulay commented Sep 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elonazoulay Sep 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kokosing commented Sep 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kokosing commented Sep 22, 2022 • edited Loading

kokosing commented Sep 22, 2022

elonazoulay commented Aug 5, 2020 •

edited

Loading

medb commented Aug 5, 2020 •

edited

Loading

elonazoulay commented Aug 6, 2020 •

edited

Loading

elonazoulay commented Aug 6, 2020 •

edited

Loading

elonazoulay commented Sep 15, 2022 •

edited

Loading

elonazoulay Sep 22, 2022 •

edited

Loading

kokosing commented Sep 22, 2022 •

edited

Loading