HelloWorld compilation times and memory usage increased in 21.2.0-dev #3435

zakkak · 2021-05-27T07:52:27Z

Describe the issue
The last 6 days Mandrel's nightly CI consistently times out in a couple of Quarkus integration test groups.
The first failing run was with 3ec3ea3: https://github.com/graalvm/mandrel/actions/runs/862486805
The last passing run was with 29d856e: https://github.com/graalvm/mandrel/actions/runs/858957124

From some testing it looks like compilation times have increased in 21.1 (compared to 21.0 and 20.3) and now they increased further in 21.2-dev (~13% compared to 21.1 and ~20% compared to 21.0 and 20.3).

Steps to reproduce the issue

For my testing I compile HelloWorld 10 times using GraalVM CE 20.3.2, 21.0.0, 21.1.0, and 21.2.0-dev (built from e279970) and generate violin plots using the [total] time reported by native-image (y-axis is ms).

The raw numbers are:

20.3.2	21.0.0	21.1.0	`e279970`
24839.03	25866.91	26802.98	32330.7
28060.58	28380.79	30629.49	33403.92
28537.7	28232.6	30947.35	34979.75
28586.7	29017.46	30818.94	35582.22
28465.51	28638.91	30669.68	34243.66
28339.83	28685.08	31468.97	34934.68
28503.86	28740.76	30951.99	36682.98
28355.35	28672.92	31117.14	35309.23
28388.52	28446.92	30908.2	34395.56
28388.05	32607.71	32124.05	34705.61

FWIW the script to generate the plot is:

#!/usr/bin/env python3

import re
import sys
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

sns.set(style="whitegrid")

data = pd.read_csv('results.csv', sep=',')

sns.violinplot(data=data, inner="quartile", bw=.2)
plt.show()

Describe GraalVM and your environment:

GraalVM version e279970
JDK major version: 11
OS: Fedora 34
Architecture: AMD64

More details
These last 6 days I have also noticed some out of memory errors (due to me not restricting the max heap size in the workflow), which made me think that memory usage has also increased, which seems true but only for 21.2.0-dev. The numbers used in the violin plots for heap size are the [total] heap usage reported at the end of native-image.

Raw data:

20.3.2	21.0.0	21.1.0	`e279970`
2.34	2.35	2.36	3.29
2.34	2.35	2.36	3.3
2.34	2.35	2.35	3.3
2.34	2.32	2.36	3.29
2.34	2.34	2.36	3.3
2.34	2.35	2.36	3.3
2.35	2.35	3.27	3.29
2.34	2.35	2.36	3.29
2.34	2.34	2.36	3.3
2.34	2.35	2.36	3.3

Edit: I originally pasted the commit hashes in reverse order. We noticed the failure while testing 3ec3ea3 and the last time we checked and didn't fail was 29d856e.

The text was updated successfully, but these errors were encountered:

zakkak · 2021-05-27T08:10:18Z

Note: after restricting the max heap size I get (in the GH workflow):

Exception in thread "native-image pid watcher" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.base/java.util.Arrays.copyOf(Arrays.java:3745)
	at java.base/java.nio.file.Files.read(Files.java:3175)
	at java.base/java.nio.file.Files.readAllBytes(Files.java:3213)
	at com.oracle.svm.hosted.NativeImageGeneratorRunner$1.run(NativeImageGeneratorRunner.java:102)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)
4 fatal errors detected:
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: void com.oracle.svm.core.genscavenge.Timers.logGCLoad(Log, String, String, Timer, Timer)
	at com.oracle.svm.hosted.code.CompileQueue.doInlineTrivial(CompileQueue.java:625)
	at com.oracle.svm.hosted.code.CompileQueue.access$000(CompileQueue.java:175)
	at com.oracle.svm.hosted.code.CompileQueue$TrivialInlineTask.run(CompileQueue.java:303)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at jdk.internal.vm.compiler/org.graalvm.compiler.graph.Edges$EdgesIterator.next(Edges.java:330)
	at jdk.internal.vm.compiler/org.graalvm.compiler.graph.Edges$EdgesIterator.next(Edges.java:265)
	at jdk.internal.vm.compiler/org.graalvm.compiler.graph.Node.replaceAtUsages(Node.java:928)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.inlining.InliningUtil.finishInlining(InliningUtil.java:544)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.inlining.InliningUtil.inline(InliningUtil.java:480)
	at jdk.internal.vm.compiler/org.graalvm.compiler.phases.common.inlining.InliningUtil.inline(InliningUtil.java:348)
	at com.oracle.svm.hosted.code.CompileQueue.tryInlineTrivial(CompileQueue.java:642)
	at com.oracle.svm.hosted.code.CompileQueue.doInlineTrivial(CompileQueue.java:607)
	at com.oracle.svm.hosted.code.CompileQueue.access$000(CompileQueue.java:175)
	at com.oracle.svm.hosted.code.CompileQueue$TrivialInlineTask.run(CompileQueue.java:303)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at com.oracle.graal.pointsto.util.CompletionExecutor$$Lambda$449/0x00000007c494d840.run(Unknown Source)
	... 6 more
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: void com.oracle.svm.core.genscavenge.UnalignedHeapChunk.initialize(UnalignedHeapChunk$UnalignedHeader, UnsignedWord)
	at com.oracle.svm.hosted.code.CompileQueue.doInlineTrivial(CompileQueue.java:625)
	at com.oracle.svm.hosted.code.CompileQueue.access$000(CompileQueue.java:175)
	at com.oracle.svm.hosted.code.CompileQueue$TrivialInlineTask.run(CompileQueue.java:303)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
	at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Fatal error:org.graalvm.compiler.debug.GraalError: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at method: boolean com.oracle.svm.core.genscavenge.UnalignedHeapChunk.walkObjectsInline(UnalignedHeapChunk$UnalignedHeader, ObjectVisitor)
	at com.oracle.svm.hosted.code.CompileQueue.doInlineTrivial(CompileQueue.java:625)
	at com.oracle.svm.hosted.code.CompileQueue.access$000(CompileQueue.java:175)
	at com.oracle.svm.hosted.code.CompileQueue$TrivialInlineTask.run(CompileQueue.java:303)
	at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:173)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
Error: Image build request failed with exit status 1

source: https://github.com/graalvm/mandrel/runs/2681905662?check_suite_focus=true#step:9:316

zakkak · 2021-05-27T13:59:47Z

The heap usage increase has been observed in #3427 as well.

oubidar-Abderrahim · 2021-05-28T11:21:00Z

Tracked internally on GR-31714

thomaswue · 2021-05-28T17:53:08Z

Thanks for sharing! When you say "HelloWorld", do you mean a hello world plain Java application with just a System.out.println or a Quarkus application? Can your infrastructure help with bisecting commits to find the exact spot where it changed? The commit referenced as "first failing" looks very unrelated to native image, so I don't think this one can be the issue.

zakkak · 2021-05-31T11:36:32Z

Thanks for sharing! When you say "HelloWorld", do you mean a hello world plain Java application with just a System.out.println or a Quarkus application?

Ooops, sorry, I should have pasted the source code. I refer to the following plain Java application:

public class HelloWorld {
    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

Can your infrastructure help with bisecting commits to find the exact spot where it changed?

Unfortunately not, we don't have anything in place for achieving this.

The commit referenced as "first failing" looks very unrelated to native image, so I don't think this one can be the issue.

That makes sense, the commit itself is probably not the one at fault. We test nightly so the commit is the one that just happened to be the HEAD of https://github.com/oracle/graal when the workflow started. These commit hashes are just an indication that the change at fault is probably somewhere between 29d856e and 3ec3ea3.

zakkak · 2021-06-04T14:45:45Z

After some manual bisecting between 29d856e and 3ec3ea3 with native-image -J-Xlog:gc HelloWorld, it looks like e92dab0 is the first commit to increase the amount of live objects in the heap.

`e92dab0`

[0.003s][info][gc] Using Parallel
[0.560s][info][gc] GC(0) Pause Young (GCLocker Initiated GC) 133M->10M(981M) 6.543ms
[0.570s][info][gc] GC(1) Pause Young (Metadata GC Threshold) 32M->11M(981M) 4.741ms
[0.596s][info][gc] GC(2) Pause Full (Metadata GC Threshold) 11M->10M(981M) 25.799ms
[0.930s][info][gc] GC(3) Pause Young (Metadata GC Threshold) 106M->21M(981M) 6.713ms
[0.987s][info][gc] GC(4) Pause Full (Metadata GC Threshold) 21M->19M(981M) 57.033ms
[1.416s][info][gc] GC(5) Pause Young (Metadata GC Threshold) 155M->33M(981M) 11.369ms
[1.563s][info][gc] GC(6) Pause Full (Metadata GC Threshold) 33M->31M(981M) 146.741ms
[helloworld:1887676]    classlist:   1,266.65 ms,  0.96 GB
[helloworld:1887676]        (cap):     417.30 ms,  0.96 GB
[helloworld:1887676]        setup:   1,869.69 ms,  0.96 GB
[4.539s][info][gc] GC(7) Pause Young (Allocation Failure) 287M->77M(981M) 27.869ms
[5.753s][info][gc] GC(8) Pause Young (Allocation Failure) 333M->124M(1194M) 59.248ms
[7.887s][info][gc] GC(9) Pause Young (Allocation Failure) 599M->201M(1231M) 95.698ms
[10.229s][info][gc] GC(10) Pause Young (Allocation Failure) 676M->239M(1720M) 84.623ms
[helloworld:1887676]     (clinit):     206.80 ms,  1.68 GB
[helloworld:1887676]   (typeflow):   5,349.99 ms,  1.68 GB
[helloworld:1887676]    (objects):   4,495.02 ms,  1.68 GB
[helloworld:1887676]   (features):     336.39 ms,  1.68 GB
[helloworld:1887676]     analysis:  10,637.72 ms,  1.68 GB
[helloworld:1887676]     universe:   1,074.11 ms,  1.68 GB
[15.892s][info][gc] GC(11) Pause Young (Allocation Failure) 1167M->276M(1742M) 153.362ms
[helloworld:1887676]      (parse):   1,109.08 ms,  1.70 GB
[helloworld:1887676]     (inline):   1,464.19 ms,  1.70 GB
[18.555s][info][gc] GC(12) Pause Young (Allocation Failure) 1204M->403M(2257M) 206.841ms
[23.758s][info][gc] GC(13) Pause Young (Allocation Failure) 1827M->424M(2316M) 157.424ms
[26.724s][info][gc] GC(14) Pause Young (Allocation Failure) 1848M->444M(3156M) 30.051ms
[helloworld:1887676]    (compile):  10,212.36 ms,  3.08 GB
[helloworld:1887676]      compile:  13,404.16 ms,  3.08 GB
[helloworld:1887676]        image:   1,954.98 ms,  3.08 GB
[helloworld:1887676]        write:     197.19 ms,  3.08 GB
# Printing build artifacts to: /home/zakkak/code/tmp/helloworld.build_artifacts.txt
[helloworld:1887676]      [total]:  30,622.37 ms,  3.08 GB

see https://github.com/graalvm/mandrel/actions/runs/906586079 as well

`698d863` (the commit right before `e92dab0`)

[0.002s][info][gc] Using Parallel
[0.548s][info][gc] GC(0) Pause Young (Metadata GC Threshold) 133M->10M(981M) 4.646ms
[0.569s][info][gc] GC(1) Pause Full (Metadata GC Threshold) 10M->10M(981M) 20.775ms
[0.862s][info][gc] GC(2) Pause Young (Metadata GC Threshold) 108M->20M(981M) 6.043ms
[0.917s][info][gc] GC(3) Pause Full (Metadata GC Threshold) 20M->19M(981M) 55.082ms
[1.369s][info][gc] GC(4) Pause Young (Metadata GC Threshold) 149M->33M(981M) 11.615ms
[1.518s][info][gc] GC(5) Pause Full (Metadata GC Threshold) 33M->31M(981M) 148.802ms
[helloworld:1894673]    classlist:   1,252.13 ms,  0.96 GB
[helloworld:1894673]        (cap):     411.04 ms,  0.96 GB
[helloworld:1894673]        setup:   1,988.30 ms,  0.96 GB
[4.728s][info][gc] GC(6) Pause Young (Allocation Failure) 287M->65M(981M) 26.524ms
[6.031s][info][gc] GC(7) Pause Young (Allocation Failure) 321M->85M(981M) 30.111ms
[6.907s][info][gc] GC(8) Pause Young (Allocation Failure) 341M->104M(1231M) 43.677ms
[9.388s][info][gc] GC(9) Pause Young (Allocation Failure) 599M->133M(1248M) 64.585ms
[12.128s][info][gc] GC(10) Pause Young (Allocation Failure) 628M->147M(1757M) 71.155ms
[helloworld:1894673]     (clinit):     190.02 ms,  1.72 GB
[helloworld:1894673]   (typeflow):   5,724.10 ms,  1.72 GB
[helloworld:1894673]    (objects):   5,300.23 ms,  1.72 GB
[helloworld:1894673]   (features):     402.56 ms,  1.72 GB
[helloworld:1894673]     analysis:  11,872.21 ms,  1.72 GB
[helloworld:1894673]     universe:     615.65 ms,  1.72 GB
[17.854s][info][gc] GC(11) Pause Young (Allocation Failure) 1131M->231M(1763M) 130.810ms
[helloworld:1894673]      (parse):   2,162.78 ms,  1.72 GB
[21.222s][info][gc] GC(12) Pause Young (Allocation Failure) 1215M->328M(2348M) 165.646ms
[helloworld:1894673]     (inline):   2,821.91 ms,  2.29 GB
[26.677s][info][gc] GC(13) Pause Young (Allocation Failure) 1859M->344M(2392M) 114.391ms
[29.975s][info][gc] GC(14) Pause Young (Allocation Failure) 1875M->358M(3301M) 51.656ms
[helloworld:1894673]    (compile):  11,839.27 ms,  3.22 GB
[helloworld:1894673]      compile:  17,621.87 ms,  3.22 GB
[helloworld:1894673]        image:   2,241.05 ms,  3.22 GB
[helloworld:1894673]        write:     225.89 ms,  3.22 GB
# Printing build artifacts to: /home/zakkak/code/tmp/helloworld.build_artifacts.txt
[helloworld:1894673]      [total]:  36,080.22 ms,  3.22 GB

see https://github.com/graalvm/mandrel/actions/runs/906585936 as well

Please note however that the committed heap size in 698d863 is still higher than the committed heap size in GraalVM CE 21.1.0 (~3.2 GB vs ~ 2.3 GB), possibly indicating that there was another memory usage increase before this one that slipped our attention.

thomaswue · 2021-06-04T15:07:52Z

Cool, thanks, yes, this commit looks like a possible candidate. It is an architectural improvement (parse bytecodes once instead of twice); but there could be some issues in this initial version. CC @christianwimmer

GraalVM 21.2-dev introduces a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enahancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI.

zakkak · 2021-06-07T09:41:35Z

For anyone interested in a workaround till this gets fixed, you may use -H:-ParseOnce:

-H:±ParseOnce  When true, compiler graphs are parsed only once before static analysis. When false, compiler graphs are parsed for
               static analysis and again for AOT compilation. Default: + (enabled).

GraalVM 21.2-dev introduced a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enahancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI. In GraalVM 21.2 this appears to no longer be an issue.

GraalVM 21.2-dev at some point introduced a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enhancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI. In GraalVM 21.2 this appears to no longer be an issue.

Disable single parsing of compiler graphs till the impact of it on heap usage decreases see oracle/graal#3435 and graalvm/mandrel#304 (comment)

Disable single parsing of compiler graphs till the impact of it on heap usage decreases see oracle/graal#3435 and graalvm/mandrel#304 (comment) quarkusio#19511 didn't take into account runs with constraint memory. Bringing back `-H:-ParseOnce` reduces the heap usage significantly, from 3.9G down to 2.7G in integration-tests/main.

GraalVM 21.2-dev at some point introduced a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enhancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI. In GraalVM 21.2 this appears to no longer be an issue. (cherry picked from commit 054236b)

Disable single parsing of compiler graphs till the impact of it on heap usage decreases see oracle/graal#3435 and graalvm/mandrel#304 (comment) quarkusio#19511 didn't take into account runs with constraint memory. Bringing back `-H:-ParseOnce` reduces the heap usage significantly, from 3.9G down to 2.7G in integration-tests/main. (cherry picked from commit 201b9a6)

GraalVM 21.2-dev at some point introduced a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enhancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI. In GraalVM 21.2 this appears to no longer be an issue. (cherry picked from commit 054236b)

Disable single parsing of compiler graphs till the impact of it on heap usage decreases see oracle/graal#3435 and graalvm/mandrel#304 (comment) quarkusio#19511 didn't take into account runs with constraint memory. Bringing back `-H:-ParseOnce` reduces the heap usage significantly, from 3.9G down to 2.7G in integration-tests/main. (cherry picked from commit 201b9a6)

zakkak · 2022-01-25T10:38:34Z

Closing this as it's now out of date and a better analysis has been performed in graalvm#316. Note:

Depending on the case using ParseOnce results in increasing Peak RSS by up to ~30% while there are cases where Peak RSS is reduced by ~18%, but in most cases we see an increase (as expected) ranging from 1% to 15%.

We no longer need to disable `ParseOnce` as memory usage with it enabled has improved in GraalVM 22.2. See https://docs.google.com/spreadsheets/d/15PJ1Qd7kgneuP61N1T2_AyJ3WBsbXpVHIPKbxgH1qfM/edit#gid=1672873268 for a comparison between using and not using `ParseOnce` with 22.2. Relates to oracle/graal#3435 and graalvm/mandrel#316 Fixes quarkusio#25944

GraalVM 21.2-dev introduces a new enhancement (enabled by default) to avoid parsing compiler graphs twice. This enahancement however results in increased memory usage at the moment (oracle/graal#3435) which then results in Quarkus' Integration Test failures in CI.

We no longer need to disable `ParseOnce` as memory usage with it enabled has improved in GraalVM 22.2. See https://docs.google.com/spreadsheets/d/15PJ1Qd7kgneuP61N1T2_AyJ3WBsbXpVHIPKbxgH1qfM/edit#gid=1672873268 for a comparison between using and not using `ParseOnce` with 22.2. Relates to oracle/graal#3435 and graalvm/mandrel#316 Fixes quarkusio#25944 (cherry picked from commit 03ea11c)

zakkak added bug native-image labels May 27, 2021

oubidar-Abderrahim self-assigned this May 27, 2021

thomaswue assigned christianwimmer May 28, 2021

zakkak mentioned this issue Jun 4, 2021

Disable single parsing of compiler graphs in native-image quarkusio/quarkus#17711

Merged

zakkak mentioned this issue Jun 14, 2021

NPE with new "Inline before static analysis" #3470

Closed

zakkak mentioned this issue Jul 6, 2021

Increased compilation times with GraalVM 21.1.0 quarkusio/quarkus#18360

Closed

This was referenced Aug 19, 2021

native-image memory usage higher in vm-21.2.0 compiled from source compared to the released binary #3697

Closed

Re-enable single parsing of compiler graphs in native-image quarkusio/quarkus#19511

Merged

zakkak added a commit to zakkak/quarkus that referenced this issue Oct 26, 2021

Disable single parsing of compiler graphs

e2c6824

Disable single parsing of compiler graphs till the impact of it on heap usage decreases see oracle/graal#3435 and graalvm/mandrel#304 (comment)

zakkak mentioned this issue Oct 26, 2021

Disable single parsing of compiler graphs by default quarkusio/quarkus#21018

Merged

zakkak closed this as completed Jan 25, 2022

zakkak mentioned this issue Aug 23, 2022

Re-enable ParseOnce starting with GraalVM / Mandrel 22.2 quarkusio/quarkus#27462

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HelloWorld compilation times and memory usage increased in 21.2.0-dev #3435

HelloWorld compilation times and memory usage increased in 21.2.0-dev #3435

zakkak commented May 27, 2021 •

edited

Loading

zakkak commented May 27, 2021 •

edited

Loading

zakkak commented May 27, 2021

oubidar-Abderrahim commented May 28, 2021

thomaswue commented May 28, 2021

zakkak commented May 31, 2021 •

edited

Loading

zakkak commented Jun 4, 2021 •

edited

Loading

thomaswue commented Jun 4, 2021

zakkak commented Jun 7, 2021 •

edited

Loading

zakkak commented Jan 25, 2022

HelloWorld compilation times and memory usage increased in 21.2.0-dev #3435

HelloWorld compilation times and memory usage increased in 21.2.0-dev #3435

Comments

zakkak commented May 27, 2021 • edited Loading

zakkak commented May 27, 2021 • edited Loading

zakkak commented May 27, 2021

oubidar-Abderrahim commented May 28, 2021

thomaswue commented May 28, 2021

zakkak commented May 31, 2021 • edited Loading

zakkak commented Jun 4, 2021 • edited Loading

e92dab0

698d863 (the commit right before e92dab0)

thomaswue commented Jun 4, 2021

zakkak commented Jun 7, 2021 • edited Loading

zakkak commented Jan 25, 2022

zakkak commented May 27, 2021 •

edited

Loading

zakkak commented May 27, 2021 •

edited

Loading

zakkak commented May 31, 2021 •

edited

Loading

zakkak commented Jun 4, 2021 •

edited

Loading

`e92dab0`

`698d863` (the commit right before `e92dab0`)

zakkak commented Jun 7, 2021 •

edited

Loading