Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Artifact removal #668

Merged
merged 92 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from 85 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
381ac90
feat: Artifact store
Natoandro Mar 23, 2024
ba09f0a
remove old artifact service
Natoandro Mar 23, 2024
1d3ea75
remove unnecessary comment
Natoandro Mar 23, 2024
15cae1a
test(sdk/core): Fix tests
Natoandro Mar 24, 2024
4cc1d47
fix tests
Natoandro Mar 26, 2024
be761e5
Merge branch 'main' into upload/s3
Natoandro Mar 26, 2024
60c96b6
Merge branch 'main' into upload/s3
Natoandro Mar 26, 2024
2aee21f
use verdaccio local npm registry
Natoandro Mar 27, 2024
fc8deeb
fix tests
Natoandro Mar 28, 2024
f010c9d
Merge branch 'main' into local-npm-registry
Natoandro Mar 28, 2024
bff7675
fix: clean deno.lock
Natoandro Mar 28, 2024
2bd5836
fix(ci): setup verdaccio for test-full gh workflow
Natoandro Mar 29, 2024
a91c5d2
Merge branch 'main' into local-npm-registry
Natoandro Mar 29, 2024
2bcd01b
fix merge confusions
Natoandro Mar 29, 2024
5ad6a82
Merge branch 'local-npm-registry' into upload/s3
Natoandro Apr 1, 2024
bc339fd
Update artifact spec
Natoandro Apr 2, 2024
ed7cc96
Fix upload
Natoandro Apr 3, 2024
8747c37
Remove console.log in wasmedge.ts
Natoandro Apr 3, 2024
a4b9047
Merge branch 'main' into local-npm-registry
Natoandro Apr 3, 2024
b17df8f
Merge branch 'local-npm-registry' into upload/s3
Natoandro Apr 3, 2024
0d8d46c
Merge branch 'main' into local-npm-registry
Natoandro Apr 3, 2024
0a1f098
test(ci): publish to local npm repository
Natoandro Apr 3, 2024
7413c27
fix typo
Natoandro Apr 3, 2024
ea76957
Update NPM registry URL in tests.yml
Natoandro Apr 3, 2024
5cbc190
fix typo
Natoandro Apr 3, 2024
f1f26ad
fix(ci): npm registry
Natoandro Apr 3, 2024
3c21c2b
Fix cache key in tests.yml workflow
Natoandro Apr 4, 2024
7033202
Merge branch 'main' into local-npm-registry
Natoandro Apr 4, 2024
4fb13fc
Merge branch 'local-npm-registry' into upload/s3
Natoandro Apr 4, 2024
ea9f1dc
feat(gate): add shared artifact store
destifo Apr 5, 2024
17651f0
chore(test): add wasmedge test with sync config(s3)
destifo Apr 5, 2024
accfc63
chore: address comments
destifo Apr 5, 2024
638693e
chore: await store init and set port on test
destifo Apr 7, 2024
2d3ac2c
fix: fix tests
destifo Apr 7, 2024
6d9d918
chore: change expiration queue to redis
destifo Apr 8, 2024
9ffd44f
chore: replace uploadUrls expiration queue with redis expiration set
destifo Apr 8, 2024
b951ab4
fix(tests): destroy s3 during close and close file
destifo Apr 8, 2024
21c8d3a
feat(test): add multiple typegate instances option
destifo Apr 8, 2024
1860d8f
chore: address comments
destifo Apr 9, 2024
c0450ef
Merge branch 'main' of https://github.com/metatypedev/metatype into u…
destifo Apr 9, 2024
1fc1c1f
chore: update snapshots
destifo Apr 9, 2024
ad64586
Merge branch 'main' into local-npm-registry
Natoandro Apr 9, 2024
177b3b4
Merge branch 'local-npm-registry' into upload/s3
Natoandro Apr 9, 2024
77596e6
fix(test): fix tests
destifo Apr 9, 2024
e51099c
Merge branch 'upload/s3' of https://github.com/metatypedev/metatype i…
destifo Apr 9, 2024
cfb3e96
docs: Document the artifact removal processes
Natoandro Apr 9, 2024
43442ae
Merge branch 'upload/s3' into file-removal
Natoandro Apr 9, 2024
7863a72
fix(tests): avoid creating link again during artifact resolution
destifo Apr 9, 2024
f21f6c4
fix(test): redis redundant closing
destifo Apr 10, 2024
5eb8738
Merge branch 'main' into local-npm-registry
Natoandro Apr 10, 2024
1956485
Merge branch 'local-npm-registry' into upload/s3
Natoandro Apr 10, 2024
870f90c
fix resource leak
Natoandro Apr 10, 2024
5302cda
Merge branch 'upload/s3' into file-removal
Natoandro Apr 10, 2024
85958d7
chore(test): add test for multiple typegate instances for wasmedge sy…
destifo Apr 11, 2024
c354e78
refacor and add reference counts
Natoandro Apr 12, 2024
e65cb76
Merge branch 'upload/s3' into file-removal
Natoandro Apr 12, 2024
d1bbf4e
fix pre-commit
Natoandro Apr 12, 2024
e123736
Merge branch 'main' into upload/s3
Natoandro Apr 12, 2024
8df57ed
Merge branch 'upload/s3' into file-removal
Natoandro Apr 12, 2024
855310f
chore: address comments
destifo Apr 12, 2024
2642bf8
fix(SDK): remove dangling import
destifo Apr 12, 2024
b78fddd
fix(tests): remove breaking typgate.terminate() call inside test
destifo Apr 12, 2024
899dc5e
fix
Natoandro Apr 13, 2024
ee14260
Merge branch 'upload/s3' into file-removal
Natoandro Apr 13, 2024
74da8e5
Merge branch 'main' into upload/s3
Natoandro May 2, 2024
2fa279f
fix merge confusions
Natoandro May 2, 2024
b4d4c85
Merge branch 'upload/s3' into file-removal
Natoandro May 2, 2024
56c285b
fix cleanups
Natoandro May 2, 2024
8a766eb
add ghjk test task
Natoandro May 3, 2024
e0d108c
fix merge confusion
Natoandro May 6, 2024
383d946
Merge branch 'main' into upload/s3
Natoandro May 6, 2024
49b3e7a
Merge branch 'upload/s3' into file-removal
Natoandro May 6, 2024
fd7c36d
fix merge confusion
Natoandro May 6, 2024
2c008be
test: test artifacts gc (local)
Natoandro May 7, 2024
7843ea2
Merge branch 'main' into file-removal
Natoandro May 7, 2024
d80df9c
Update docs/artifact-storage.md
Natoandro May 7, 2024
53b472b
Update docs/artifact-storage.md
Natoandro May 7, 2024
1d93ceb
fix tests, remove port and systemTypegraphs test options
Natoandro May 7, 2024
25d68dd
Merge remote-tracking branch 'origin/file-removal' into file-removal
Natoandro May 7, 2024
310a637
update test options
Natoandro May 7, 2024
def6fe0
update docs
Natoandro May 7, 2024
42b4ce8
fix base64 decoding
Natoandro May 7, 2024
e787d09
add more tests for the artifacts
Natoandro May 7, 2024
8d21df4
Update website/docs/concepts/architecture/index.mdx
Natoandro May 7, 2024
d14f53f
Merge remote-tracking branch 'origin/file-removal' into file-removal
Natoandro May 7, 2024
f3d7214
fix deploy_test
Natoandro May 7, 2024
3c4101a
sync test for artifacts
Natoandro May 8, 2024
3fad9ff
remove stale comments
Natoandro May 8, 2024
9422eb6
fix tests
Natoandro May 8, 2024
2d12c82
fix typo, fix undeploy_test
Natoandro May 8, 2024
8c3d99b
optimise getLocalParentDir function
Natoandro May 8, 2024
6c7fd3c
fix wasm tests
Natoandro May 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 196 additions & 0 deletions .ghjk/deno.lock

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions .ghjk/lock.json
Original file line number Diff line number Diff line change
Expand Up @@ -563,6 +563,15 @@
"env": {},
"allowedPortDeps": []
}
},
"test": {
"name": "test",
"dependsOn": [],
"env": {
"installs": [],
"env": {},
"allowedPortDeps": []
}
}
}
}
Expand Down
3 changes: 2 additions & 1 deletion .vscode/extensions.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
"chrischinchilla.vale-vscode",
"hediet.vscode-drawio",
"davidlday.languagetool-linter",
"dtsvet.vscode-wasm"
"dtsvet.vscode-wasm",
"bierner.comment-tagged-templates"
]
}
1 change: 0 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
"website/.docusaurus": true,
"website/node_modules": true
},
"deno.importMap": "typegate/import_map.json",
"deno.enablePaths": [
"typegate",
"dev",
Expand Down
12 changes: 6 additions & 6 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

156 changes: 78 additions & 78 deletions docs/workflows/artifact_upload_protocol.drawio.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
42 changes: 28 additions & 14 deletions ghjk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ const allowedPortDeps = [
defaultInst: thinInstallConfig(fat),
})),
];
export const secureConfig = ghjk.secureConfig({ allowedPortDeps });

const inCi = () => !!Deno.env.get("CI");
const inOci = () => !!Deno.env.get("OCI");
Expand All @@ -63,7 +62,7 @@ ghjk.install(
crateName: "wasm-tools",
version: WASM_TOOLS_VERSION,
locked: true,
}),
})
);

if (!inOci()) {
Expand All @@ -81,7 +80,7 @@ if (!inOci()) {
packageName: "@bytecodealliance/jco",
version: JCO_VERSION,
})[0],
ports.npmi({ packageName: "node-gyp", version: "10.0.1" })[0],
ports.npmi({ packageName: "node-gyp", version: "10.0.1" })[0]
);
}

Expand All @@ -90,7 +89,7 @@ if (Deno.build.os == "linux" && !Deno.env.has("NO_MOLD")) {
ports.mold({
version: MOLD_VERSION,
replaceLd: Deno.env.has("CI") || Deno.env.has("OCI"),
}),
})
);
}

Expand All @@ -100,20 +99,18 @@ if (!Deno.env.has("NO_PYTHON")) {
ports.pipi({
packageName: "poetry",
version: POETRY_VERSION,
})[0],
})[0]
);
if (!inOci()) {
ghjk.install(
ports.pipi({ packageName: "pre-commit" })[0],
);
ghjk.install(ports.pipi({ packageName: "pre-commit" })[0]);
}
}

if (inDev()) {
ghjk.install(
ports.act({}),
ports.cargobi({ crateName: "whiz", locked: true }),
installs.comp_py[0],
installs.comp_py[0]
);
}

Expand All @@ -122,13 +119,12 @@ ghjk.task("clean-deno-lock", {
// jq
],
async fn({ $ }) {
const jqOp1 =
`del(.packages.specifiers["npm:@typegraph/sdk@${METATYPE_VERSION}"])`;
const jqOp1 = `del(.packages.specifiers["npm:@typegraph/sdk@${METATYPE_VERSION}"])`;
const jqOp2 = `del(.packages.npm["@typegraph/sdk@${METATYPE_VERSION}"])`;
const jqOp = `${jqOp1} | ${jqOp2}`;
await Deno.writeTextFile(
"typegate/deno.lock",
await $`jq ${jqOp} typegate/deno.lock`.text(),
await $`jq ${jqOp} typegate/deno.lock`.text()
);
},
});
Expand All @@ -138,8 +134,9 @@ ghjk.task("gen-pyrt-bind", {
allowedPortDeps,
async fn({ $ }) {
await $.removeIfExists("./libs/pyrt_wit_wire/wit_wire");
await $`componentize-py -d ../../wit/wit-wire.wit bindings .`
.cwd("./libs/pyrt_wit_wire");
await $`componentize-py -d ../../wit/wit-wire.wit bindings .`.cwd(
"./libs/pyrt_wit_wire"
);
},
});

Expand Down Expand Up @@ -172,3 +169,20 @@ ghjk.task("dev", {
await $`deno run --allow-all ${script} ${args}`;
},
});

ghjk.task("test", {
async fn({ $, argv }) {
const script = $.path(projectDir).join("dev/test.ts");
await $`deno run --allow-all ${script} ${argv}`;
},
});

export const secureConfig = ghjk.secureConfig({
allowedPortDeps: [
...ghjk.stdDeps(),
...[installs.python_latest, installs.node].map((fat) => ({
manifest: fat.port,
defaultInst: thinInstallConfig(fat),
})),
],
});
Loading
Loading