42.2.8: Infinite loops with message "Ignoring request to process copy results, already processing" #1592

hyunkshinft · 2019-10-25T20:22:52Z

I'm submitting a ...

bug report
feature request

Describe the issue
A clear and concise description of what the issue is.

The JDBC driver falls into infinite loops during the streaming. One thread attempt abort streaming by closing the stream while another tread is still reading it. It occasionally loops while printing "Ignoring request to process copy results, already processing" indefinitely.

This looks like related to the PR #1467

Driver Version?
42.2.8
Java Version?
java version "1.8.0_131"

OS Version?
Linux ip-10-47-10-161 4.9.51-10.52.amzn1.x86_64 #1 SMP Fri Sep 29 01:16:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

PostgreSQL Version?
10.7
To Reproduce
Steps to reproduce the behaviour:

thread 1:

stream.read(); or stream.readPending();

thread 2:

stream.close();
replicationConnection.close()

Expected behaviour
A clear and concise description of what you expected to happen.

thread 1 throws an exception.

And what actually happens

Infinite loops printing the message "Ignoring request to process copy results, already processing"

Logs
If possible PostgreSQL logs surrounding the occurrence of the issue
Additionally logs from the driver can be obtained adding

loggerLevel=TRACE&loggerFile=pgjdbc-trace.log

to the connection string

The text was updated successfully, but these errors were encountered:

… another thread closes the connection

davecramer · 2019-10-29T14:02:52Z

@hyunkshinft can you test the PR above ?

hyunkshinft · 2019-10-29T16:09:09Z

@davecramer thanks for the quick fix. I will try this as soon as I can and let you know of the result.

hyunkshinft · 2019-10-30T15:42:03Z

@davecramer unfortunately this did not fix the problem. FYI the following is the piece of code that closed the stream.

        try {
            if (context.stream != null && !context.stream.isClosed()) context.stream.close();
            if (context.replicationConnection != null) context.replicationConnection.close();
        } catch (SQLException tolerable) {
        }

davecramer · 2019-10-30T15:47:27Z

hmmm strange. I would think that should cause an exception to be thrown.
FYI, there's no thread safety here. In reality you should not be closing it in another thread.

hyunkshinft · 2019-10-30T15:55:53Z

I thought so too. Your fix seems very obvious to me but somehow it passed connection close check.

davecramer · 2019-10-30T16:37:04Z

Something else must be going on. Any chance you can debug or provide a stand alone test case ?

hyunkshinft · 2019-10-30T16:59:16Z

It does not happen always and might not duplicate under debugger but I will try. I am sorry but I don't know exact scenario yet.

hyunkshinft · 2019-10-30T23:18:03Z

@davecramer one thing that I want to remark about this particular connection is the WAL records are usually as big as 48MB and the end to end connection has very long pipe with at least two ssh tunneling.

davecramer · 2019-11-01T13:35:55Z

@hyunkshinft OK, I have some cycles to look at this. Any update before I do ?

hyunkshinft · 2019-11-01T17:33:12Z

@davecramer Now I don't close the stream so I don't hit "Ignoring request ..." message anymore. I even added an exception to fail immediately as below in processCopyResults in order to avoid excessive logging in case of this error.

    if ( processingCopyResults.compareAndSet(false,true) == false ) {
      PSQLException error = new PSQLException(GT
          .tr("Ignoring request to process copy results, already processing"),
          PSQLState.UNEXPECTED_ERROR);
      LOGGER.log(Level.INFO, "Ignoring request to process copy results, already processing", error);
      throw error;
//      return null;
    }

I know you have implemented this thread protection code to address the issue with "Unexpected packet type during copy" error. But I am still getting "Unexpected packet type during copy" error with out hitting above Exception code.

class java.io.IOException: Unexpected packet type during copy: 109
org.postgresql.core.v3.QueryExecutorImpl::processCopyResults(QueryExecutorImpl.java::1293)
org.postgresql.core.v3.QueryExecutorImpl::readFromCopy(QueryExecutorImpl.java::1064)
org.postgresql.core.v3.CopyDualImpl::readFromCopy(CopyDualImpl.java::37)
org.postgresql.core.v3.replication.V3PGReplicationStream::receiveNextData(V3PGReplicationStream.java::158)
org.postgresql.core.v3.replication.V3PGReplicationStream::readInternal(V3PGReplicationStream.java::123)

My code has retry algorithms (close existing stream and connection then reopen) in case of PSQLException and it works for a while after retry then hit the same error again and again.

class java.io.IOException: Unexpected packet type during copy: 32
org.postgresql.core.v3.QueryExecutorImpl::processCopyResults(QueryExecutorImpl.java::1293)
org.postgresql.core.v3.QueryExecutorImpl::readFromCopy(QueryExecutorImpl.java::1064)
org.postgresql.core.v3.CopyDualImpl::readFromCopy(CopyDualImpl.java::37)
org.postgresql.core.v3.replication.V3PGReplicationStream::receiveNextData(V3PGReplicationStream.java::158)
org.postgresql.core.v3.replication.V3PGReplicationStream::readInternal(V3PGReplicationStream.java::123)

Working and fail goes on and on until it hits OOM when that "unexpected packet type" is among 'E' or 'N' probably coincidently. If it gets big elen or nlen then the process dies with OOM as below

class java.lang.OutOfMemoryError: Java heap space
org.postgresql.core.VisibleBufferedInputStream::doubleBuffer(VisibleBufferedInputStream.java::152)
org.postgresql.core.VisibleBufferedInputStream::readMore(VisibleBufferedInputStream.java::136)
org.postgresql.core.VisibleBufferedInputStream::ensureBytes(VisibleBufferedInputStream.java::109)
org.postgresql.core.PGStream::receiveErrorString(PGStream.java::383)
org.postgresql.core.v3.QueryExecutorImpl::receiveErrorResponse(QueryExecutorImpl.java::2463)
org.postgresql.core.v3.QueryExecutorImpl::processCopyResults(QueryExecutorImpl.java::1144)
org.postgresql.core.v3.QueryExecutorImpl::writeToCopy(QueryExecutorImpl.java::1026)
org.postgresql.core.v3.CopyDualImpl::writeToCopy(CopyDualImpl.java::19)

I added the check in receiveErrorResponse() and receiveNoticeResponse() and return error when there is invalid elen or nlen as below.

    if (elen <= 4 || elen >= 100 * 1024) { // error message greater tahn 100K? Something wrong.

        return new PSQLException(GT.tr("Protocol error.  Invalid error message length " + elen), PSQLState.PROTOCOL_VIOLATION);
    }

And

    if (nlen <= 4 || nlen >= 100 * 1024) { // error message greater tahn 100K? Something wrong.

        throw new IOException("Protocol error.  Invalid error message length " + nlen);
    }

And now it fails at different place.

class java.lang.OutOfMemoryError: Java heap space
org.postgresql.core.PGStream::receive(PGStream.java::473)
org.postgresql.core.v3.QueryExecutorImpl::processCopyResults(QueryExecutorImpl.java::1247)
org.postgresql.core.v3.QueryExecutorImpl::readFromCopy(QueryExecutorImpl.java::1064)
org.postgresql.core.v3.CopyDualImpl::readFromCopy(CopyDualImpl.java::37)
org.postgresql.core.v3.replication.V3PGReplicationStream::receiveNextData(V3PGReplicationStream.java::158)
org.postgresql.core.v3.replication.V3PGReplicationStream::readInternal(V3PGReplicationStream.java::123)
org.postgresql.core.v3.replication.V3PGReplicationStream::readPending(V3PGReplicationStream.java::80)

I have never seen a message "Ignoring request to process copy results, already processing" if I don't stop stream and still getting above errors. Which means that "Unexpected .." error might not be related to thread racing condition in the first place?

hyunkshinft · 2019-11-01T18:38:18Z

@davecramer I will describe our code design here a bit in order to help you understand what is going on.

I have two threads.
The first is a producer which keeps reading from the stream and adding WAL data(ByteBuffer) to a ConcurrentLinkedQueue.
The 2nd is the consumer which reads WAL(ByteBuffer) from the queue and processes it. this thread will never access stream (previously it closed the stream when it decided to exit, but now it does not).

There is a flow control in place through the concurrent queue between the producer and the consumer.

I suppose the stream API allocates a new ByteBuffer every time stream.read() or stream.readPending() is called so the consumer completely owns the buffer once it is read and returned by stream.read().

davecramer · 2019-11-01T19:12:36Z

@hyunkshinft your code must have more than one reader. How would it be possible to get around the lock that was put in to avoid the Unexpected char message ? This might also explain the first symptom as well.

hyunkshinft · 2019-11-01T19:21:26Z

@davecramer I am pretty sure that we have one reader thread. that is why I have never seen log message nor exception caused by this code.

    if ( processingCopyResults.compareAndSet(false,true) == false ) {
      PSQLException error = new PSQLException(GT
          .tr("Ignoring request to process copy results, already processing"),
          PSQLState.UNEXPECTED_ERROR);
      LOGGER.log(Level.INFO, "Ignoring request to process copy results, already processing", error);
      throw error;
//      return null;
    }

davecramer · 2019-11-04T13:05:16Z

OK, let me try to write a test case to test this.

hyunkshinft · 2019-11-08T18:23:51Z

@davecramer I think I found a cause of the problems for "Unexpected packet type ..." as well as OOM, but I am not sure if it has any thing to do with the original infinite loop issue reported here. As far as I don't close the stream I haven't encountered this infinite loop. What is so bad about this infinite loop is that it produces so many logs especially in the production it increases cloud logging cost dramatically without being noticed as it is just an INFO message.

davecramer · 2019-11-08T18:24:51Z

@hyunkshinft I'd love to know the cause. are you able to share?

hyunkshinft · 2019-11-08T18:34:44Z

Here is a proposed fix for the "class java.io.IOException: Unexpected packet type during copy: " and OOM. The root cause of these problems is that when you hit SocketTimeout in the middle of CopyData you should not fail but just wait as far as the connection is still connected. Otherwise, your next processCopyResuts will start at random position and ends up with Unexpected ... if the random character is unknown one and if the random character just happens to be the valid command then the next 4 bytes are taken as message length which is completely random. So you end up with unpredictable error including OOM thereafter. SocketTimeout can happen by the various reasons. Slow server, network latency, etc.

diff -Naur 42.2.6/pgjdbc/pgjdbc/src/main/java/org/postgresql/core/VisibleBufferedInputStream.java 42.2.6-patched/pgjdbc/pgjdbc/src/main/java/org/postgresql/core/VisibleBufferedInputStream.java
--- 42.2.6/pgjdbc/pgjdbc/src/main/java/org/postgresql/core/VisibleBufferedInputStream.java	2019-11-08 09:27:07.000000000 -0800
+++ 42.2.6-patched/pgjdbc/pgjdbc/src/main/java/org/postgresql/core/VisibleBufferedInputStream.java	2019-11-08 00:58:19.000000000 -0800
@@ -8,6 +8,7 @@
 import java.io.EOFException;
 import java.io.IOException;
 import java.io.InputStream;
+import java.net.SocketTimeoutException;
 
 /**
  * A faster version of BufferedInputStream. Does no synchronisation and allows direct access to the
@@ -138,7 +139,7 @@
       canFit = buffer.length - endIndex;
     }
     int read = wrapped.read(buffer, endIndex, canFit);
-    if (read < 0) {
+    if (read <= 0) {
       return false;
     }
     endIndex += read;
@@ -211,7 +212,12 @@
 
     // then directly from wrapped stream
     do {
-      int r = wrapped.read(to, off, len);
+      int r;
+      try {
+        r = wrapped.read(to, off, len);
+      } catch (SocketTimeoutException e) {
+        return read;
+      }
       if (r <= 0) {
         return (read == 0) ? r : read;
       }

hyunkshinft · 2019-11-08T18:38:50Z

@davecramer let me know what you think.

davecramer · 2019-11-08T18:41:24Z

Were you able to confirm that this fixes things for you? I'd like to remove the code I put in and try this with the test case with very large copy. Seems like this might be the underlying reason

hyunkshinft · 2019-11-08T18:43:21Z

Yes, I am running this code without problem for about a day now. It works for both 42.2.6 and 42.2.8 but as I said 42.2.8 has other problems so yes, you may consider revert your change.

hyunkshinft · 2019-11-08T18:45:36Z

If you can briefly pause the communication during CopyData then you can duplicate this problem easily.

davecramer · 2019-11-08T18:46:29Z

thx, I'll look into it. Dave Cramer

…

On Fri, 8 Nov 2019 at 13:45, hyunkshinft ***@***.***> wrote: If you can briefly pause the communication during CopyData then you can duplicate this problem easily. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1592?email_source=notifications&email_token=AADDH5VOVMIH7XX4AR7APYDQSWXVDA5CNFSM4JFIXQ72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDTAF3A#issuecomment-551944940>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDH5XYOJ62W2SSI3TBWADQSWXVDANCNFSM4JFIXQ7Q> .

davecramer · 2019-11-08T21:09:18Z

OK, I removed the lock and put your code in and I can still get the error with unexpected character pretty easily.

hyunkshinft · 2019-11-08T23:01:58Z

Are you using multiple readers of the stream? My change is for fixing the issue with the input stream processing in a single threaded scenario. Your change may still be needed to cover multi threads scenario but the danger of that fix is the infinite loop in certain scenario.

davecramer · 2019-11-09T01:17:38Z

No not using threaded readers. As I said JDBC is not thread-safe.
#1466 scroll to the end. The guys from braintree that originally found the problem are testing and have not been able to repeat it. Need to look at this further next week

hyunkshinft · 2019-11-09T01:30:05Z

Which means you can still duplicate the problem with new code while the issuer of #1466 cannot?

davecramer · 2019-11-09T01:31:16Z

That is the short version, yes. But I am using their code to duplicate it. So at the moment I don't have an answer

hyunkshinft · 2019-11-09T01:47:27Z

It could be that there is more than SocketTimeoutException that causes the processCopyResults to fail prematurely. If you can easily duplicate then I suggest you to catch all Throwable instead of just SocketTimeoutException and see what else is being captured.

davecramer · 2019-11-09T23:36:15Z

I caught everything. Still failed.

hyunkshinft · 2019-12-24T21:36:27Z

@davecramer I noticed that my proposed fix didn't make to the official release so I modified the test created by #1466 in order to duplicate the error quickly and reliably. The test reproduced "Unexpected packet type error" using the version 42.2.9. I also included the test with my proposed fix. Please find modified test case here.

https://github.com/hyunkshinft/streamProblems#proposed-fix.

Hopefully you incorporate my proposed fix in the next official release.

davecramer · 2019-12-26T11:33:47Z

@hyunkshinft I did try this and was able to reproduce the error quicker I think I may have to revisit this when I come back in the new year. Thanks for reminding me

hyunkshinft · 2019-12-27T18:28:51Z

@davecramer please note that the proposed fix has been slightly changed.

davecramer · 2020-01-03T13:27:59Z

OK, I've removed the "hack" and applied your patch to the latest code and this seems to work. I'd like a few others to test it.

I will push a fix so everyone can test it

davecramer · 2020-01-03T19:07:02Z

Thanks see #1665

davecramer · 2020-01-03T19:25:50Z

so it appears that catching and ignoring the socket timeout exception causes other issues with getting async notifies. We use the socket timeout and since it is caught and ignored we never timeout.

davecramer · 2020-01-06T13:20:49Z

@hyunkshinft ya, while this fixes one problem it causes others. Need a rethink. Honestly I think we really need to look at reading data from the backend in a thread.

hyunkshinft · 2020-01-10T18:04:10Z

@davecramer Sorry for the late response. I have been on vacation. I agree that it could cause an issue with non-blocking read. I will take a look at next week and propose you a quick solution. I think we should handle socket timeout differently when we are fresh with expecting a new message and when we have read partial data already.

davecramer · 2020-01-10T18:05:05Z

@hyunkshinft no worries. As I said I think we need to have reads running in a background thread.

hyunkshinft · 2020-01-13T23:22:33Z

@davecramer is there an easy way to setup a regression test environment locally so that I can test my fix?

davecramer · 2020-01-14T00:31:16Z

I was able to just use the tests that were in the PR that you had. The problem is a decent sized disk. I am travelling this week so I couldn't test it but I can look at it to see if we can find anything before hand Dave Cramer

…

On Mon, 13 Jan 2020 at 18:22, hyunkshinft ***@***.***> wrote: @davecramer <https://github.com/davecramer> is there an easy way to setup a regression test environment locally so that I can test my fix? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1592?email_source=notifications&email_token=AADDH5QELYJHFZV6RZSTUVDQ5TZTTA5CNFSM4JFIXQ72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI2VR4Y#issuecomment-573921523>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDH5RVDIUFPA7UKFMINLLQ5TZTTANCNFSM4JFIXQ7Q> .

hyunkshinft · 2020-01-14T19:03:43Z

@davecramer I have added additional fix that addressed the problem with non-blocking IO. Hopefully this fixes the test failure you saw. The idea of this fix is that the socket timeout applies only at the message boundary. Which means socket timeout is taken into account until you saw the first character of message in processNotifies and processCopyResults. Once you have the first character of a new message, any socket timeout thereafter will be ignored until the full message is received in order to preserve the message boundary. The fix has been implemented on top of 42.2.6. Please take a look when you have a chance.

https://github.com/hyunkshinft/streamProblems#proposed-fix

davecramer · 2020-01-25T11:11:52Z

@hyunkshinft ok, better but now failing setNetworkTImeout Test see #1679

hyunkshinft · 2020-01-27T17:21:37Z

@davecramer thanks for checking this out. I am going to address the test failure here. #1681

hyunkshinft · 2020-01-27T19:17:12Z

@davecramer I have fixed all the unit test failures in #1681. Would you please review it from that PR or review and incorporate the latest changes into yours for inclusion in the next release?

…er thread closes the connection (#1594) * fix: Fixes issue #1592 where one thread is reading the copy and another thread closes the connection

* origin/master: (427 commits) refactor: make PSQLState enum consts for integrity constraint violations (pgjdbc#1699) [maven-release-plugin] prepare for next development iteration [maven-release-plugin] prepare release REL42.2.10 pass gpg key through arguments add passphrase to release mvn task chore: update signing key Metadata queries improvment (pgjdbc#1694) WIP release notes for 42.2.10 (pgjdbc#1688) chore(deps): bump checkstyle from 8.28 to 8.29 in /pgjdbc (pgjdbc#1691) Cleanup PGProperty, sort values, and add some missing to docs (pgjdbc#1686) fix: Fixes issue pgjdbc#1592 where one thread is reading the copy and another thread closes the connection (pgjdbc#1594) Fixing LocalTime rounding (losing precision) (pgjdbc#1570) sync error message value with tested value (pgjdbc#1664) add DatabaseMetaDataCacheTest to test suite to run it (pgjdbc#1685) Fix Network Performance of PgDatabaseMetaData.getTypeInfo() method (pgjdbc#1668) fix: Issue pgjdbc#1680 updating a boolean field requires special handling to set it to t or f instead of true or false (pgjdbc#1682) fix testSetNetworkTimeoutEnforcement test failure (pgjdbc#1681) minor: fix checkstyle violation of unused import (pgjdbc#1683) fix: pgjdbc#1677 NumberFormatException when fetching PGInterval with small value (pgjdbc#1678) fix: actually use milliseconds instead of microseconds for timeouts (pgjdbc#1653) ...

ejboy · 2022-05-31T23:05:19Z

@hyunkshinft @davecramer I wonder if the change in https://github.com/pgjdbc/pgjdbc/pull/1681/files#diff-1c9bbe3aa0c1f2d0d7a6ecef10d981c2be064a526e8f91e2b47b04823fd50dedR142 is intentional. I.e. immediate returning false instead of setting available to false?

The problem with that change is that nextStreamAvailableCheckTime field is not updated, which can lead to repeated calls to hasMessagePending blocking for 1 millisecond.

davecramer · 2022-10-11T07:14:08Z

Have you tested your solution? Can you provide a PR with the change so we can review it ? Dave Cramer

…

On Tue, 31 May 2022 at 19:05, Fyodor Kupolov ***@***.***> wrote: @hyunkshinft <https://github.com/hyunkshinft> @davecramer <https://github.com/davecramer> I wonder if the change in https://github.com/pgjdbc/pgjdbc/pull/1681/files#diff-1c9bbe3aa0c1f2d0d7a6ecef10d981c2be064a526e8f91e2b47b04823fd50dedR142 is intentional. I.e. immediate returning false instead of setting available to false? The problem with that change is that nextStreamAvailableCheckTime field is not updated, which can lead to repeated calls to hasMessagePending blocking for 1 millisecond. — Reply to this email directly, view it on GitHub <#1592 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADDH5QP4KRF6ILHWW6FMG3VM2LLVANCNFSM4JFIXQ7Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

ejboy · 2022-10-13T16:04:21Z

Thank you for replying.

Have you tested your solution? Can you provide a PR with the change so we
can review it ?

Yes, we've run into this issue due to some odd usage of the driver on our side, where it manifested in essentially 1 millisecond being wasted and that was unexpected for us. The fix has been in place and was tested for a couple of months.

I will prepare a PR soon. (May take week+, cause I need to run it with legal etc)

davecramer added a commit to davecramer/pgjdbc that referenced this issue Oct 29, 2019

fix: Fixes issue pgjdbc#1592 where one thread is reading the copy and…

8ac14c5

… another thread closes the connection

davecramer self-assigned this Nov 28, 2019

davecramer added the mystery label Nov 28, 2019

davecramer added a commit that referenced this issue Jan 30, 2020

fix: Fixes issue #1592 where one thread is reading the copy and anoth…

1191076

…er thread closes the connection (#1594) * fix: Fixes issue #1592 where one thread is reading the copy and another thread closes the connection

42.2.8: Infinite loops with message "Ignoring request to process copy results, already processing" #1592

42.2.8: Infinite loops with message "Ignoring request to process copy results, already processing" #1592

Comments

hyunkshinft commented Oct 25, 2019 • edited Loading

davecramer commented Oct 29, 2019

hyunkshinft commented Oct 29, 2019

hyunkshinft commented Oct 30, 2019

davecramer commented Oct 30, 2019

hyunkshinft commented Oct 30, 2019

davecramer commented Oct 30, 2019

hyunkshinft commented Oct 30, 2019

hyunkshinft commented Oct 30, 2019 • edited Loading

davecramer commented Nov 1, 2019

hyunkshinft commented Nov 1, 2019

hyunkshinft commented Nov 1, 2019 • edited Loading

davecramer commented Nov 1, 2019

hyunkshinft commented Nov 1, 2019

davecramer commented Nov 4, 2019

hyunkshinft commented Nov 8, 2019

davecramer commented Nov 8, 2019

hyunkshinft commented Nov 8, 2019 • edited Loading

hyunkshinft commented Nov 8, 2019

davecramer commented Nov 8, 2019

hyunkshinft commented Nov 8, 2019

hyunkshinft commented Nov 8, 2019

davecramer commented Nov 8, 2019 via email

davecramer commented Nov 8, 2019

hyunkshinft commented Nov 8, 2019 via email • edited Loading

davecramer commented Nov 9, 2019

hyunkshinft commented Nov 9, 2019

davecramer commented Nov 9, 2019

hyunkshinft commented Nov 9, 2019

davecramer commented Nov 9, 2019

hyunkshinft commented Dec 24, 2019 • edited Loading

davecramer commented Dec 26, 2019

hyunkshinft commented Dec 27, 2019

davecramer commented Jan 3, 2020

davecramer commented Jan 3, 2020

davecramer commented Jan 3, 2020

davecramer commented Jan 6, 2020

hyunkshinft commented Jan 10, 2020 • edited Loading

davecramer commented Jan 10, 2020

hyunkshinft commented Jan 13, 2020

davecramer commented Jan 14, 2020 via email

hyunkshinft commented Jan 14, 2020

davecramer commented Jan 25, 2020

hyunkshinft commented Jan 27, 2020

hyunkshinft commented Jan 27, 2020

ejboy commented May 31, 2022

davecramer commented Oct 11, 2022 via email

ejboy commented Oct 13, 2022

hyunkshinft commented Oct 25, 2019 •

edited

Loading

hyunkshinft commented Oct 30, 2019 •

edited

Loading

hyunkshinft commented Nov 1, 2019 •

edited

Loading

hyunkshinft commented Nov 8, 2019 •

edited

Loading

hyunkshinft commented Nov 8, 2019 via email •

edited

Loading

hyunkshinft commented Dec 24, 2019 •

edited

Loading

hyunkshinft commented Jan 10, 2020 •

edited

Loading