Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add implementation. Closes #1 #2

Merged
merged 56 commits into from
May 31, 2023
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
7d7f52f
docs
juliangruber May 24, 2023
efbe295
add get job from orchestrator
juliangruber May 24, 2023
9bdc77c
style
juliangruber May 24, 2023
6e91bb4
add fetch car via saturn gateway
juliangruber May 24, 2023
363c56d
add submit success to ingestor
juliangruber May 24, 2023
9069c91
fix submit retrieval
juliangruber May 24, 2023
83db140
log
juliangruber May 24, 2023
327ad17
refactor
juliangruber May 24, 2023
2662dfa
add loop
juliangruber May 24, 2023
911f11d
more stable loop
juliangruber May 24, 2023
d95b315
call `Zinnia.jobCompleted()`
juliangruber May 24, 2023
1c5b98d
collect ttfb
juliangruber May 24, 2023
378d42d
add response status
juliangruber May 24, 2023
fdc6458
abort early if res not ok
juliangruber May 24, 2023
a577f9a
refactor
juliangruber May 24, 2023
0cebf09
add read timeout
juliangruber May 24, 2023
4310754
report error activity
juliangruber May 24, 2023
2d5c611
clean up
juliangruber May 24, 2023
6206c73
refactor
juliangruber May 24, 2023
cf6dfb3
refactor
juliangruber May 24, 2023
cabc58d
start test suite
juliangruber May 24, 2023
02ed3ee
add ci
juliangruber May 24, 2023
79f9935
refactor
juliangruber May 24, 2023
00885d0
test
juliangruber May 29, 2023
71ffb22
add retries to integration test
juliangruber May 29, 2023
08c5568
fix ci
juliangruber May 29, 2023
2a4cd08
add unit test
juliangruber May 29, 2023
6dc0cc9
simplify, add unit test
juliangruber May 29, 2023
826d648
add unit test
juliangruber May 29, 2023
2c41f31
todo
juliangruber May 29, 2023
9cf1e65
todo
juliangruber May 29, 2023
ab157fc
more tests
juliangruber May 29, 2023
da0fd27
add submit more fields
juliangruber May 29, 2023
353c835
fmt
juliangruber May 29, 2023
a140c93
module.js -> main.js
juliangruber May 30, 2023
ee4ba4f
ci: add lint
juliangruber May 30, 2023
800e5b7
docs
juliangruber May 30, 2023
acb147e
add receive headers timeout
juliangruber May 30, 2023
85d2135
add `ActivityState`
juliangruber May 30, 2023
a9427e3
move `test/all.js` to `test.js`
juliangruber May 30, 2023
838e2a2
add missing throw in integration test
juliangruber May 30, 2023
ce7b5c6
refactor using async iterator syntax
juliangruber May 30, 2023
2dc0c03
fix lint
juliangruber May 31, 2023
62f8cce
allow more attempts for integration tests
juliangruber May 31, 2023
633798c
add test order, arguments and length of `fetch` calls
juliangruber May 31, 2023
dfc6707
fix submitRetrieval + test
juliangruber May 31, 2023
27e7832
fix lint
juliangruber May 31, 2023
7e0ca9f
fix tests and condition
juliangruber May 31, 2023
e97820e
update `spark-api` schema
juliangruber May 31, 2023
c157dc0
add check retrieval from api
juliangruber May 31, 2023
0cb2abe
include status code in error message
juliangruber May 31, 2023
5d514b1
`startTimeout` -> `resetTimeout`
juliangruber May 31, 2023
b04e901
make `stats` also accessible if retrieval fails
juliangruber May 31, 2023
89e3d05
use activity class
juliangruber May 31, 2023
33775e5
add assert res.ok
juliangruber May 31, 2023
5167c71
fix lint
juliangruber May 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions lib/spark.js
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,11 @@ export default class Spark {
async fetchCAR (url) {
// TODO: Ensure these fields are persisted by `spark-api`
bajtos marked this conversation as resolved.
Show resolved Hide resolved
const stats = {
start: new Date(),
firstByte: null,
end: null,
startAt: new Date(),
firstByteAt: null,
endAt: null,
byteLength: 0,
status: null
statusCode: null
}
console.log('Fetching CAR...')

Expand All @@ -64,13 +64,13 @@ export default class Spark {
try {
startTimeout()
const res = await this.#fetch(url, { signal })
stats.status = res.status
stats.statusCode = res.status

if (res.ok) {
startTimeout()
for await (const value of res.body) {
if (stats.firstByte === null) {
stats.firstByte = new Date()
if (stats.firstByteAt === null) {
stats.firstByteAt = new Date()
}
stats.byteLength += value.byteLength
startTimeout()
Expand All @@ -80,7 +80,7 @@ export default class Spark {
clearTimeout(timeout)
}

stats.end = new Date()
stats.endAt = new Date()
console.log(stats)
return stats
}
Expand Down Expand Up @@ -125,6 +125,7 @@ export default class Spark {

await this.submitRetrieval(retrieval.id, { success, ...stats })
bajtos marked this conversation as resolved.
Show resolved Hide resolved
Zinnia.jobCompleted()
juliangruber marked this conversation as resolved.
Show resolved Hide resolved
return retrieval.id
}

async run () {
Expand Down
8 changes: 7 additions & 1 deletion test/integration.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,13 @@
import Spark from '../lib/spark.js'
import { test } from 'zinnia:test'
import { assert, assertEquals } from 'zinnia:assert'

test('integration', async () => {
const spark = new Spark()
await spark.nextRetrieval()
const id = await spark.nextRetrieval()
const res = await fetch(`https://spark.fly.dev/retrievals/${id}`)
const retrieval = await res.json()
assert(retrieval.startAt)
juliangruber marked this conversation as resolved.
Show resolved Hide resolved
assert(retrieval.finishedAt)

})