Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(lambda): functionality for updating ID, values, and deleting packages #923

Merged
merged 151 commits into from
Dec 19, 2024
Merged
Changes from 1 commit
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
8a50578
wip
tiffanyvu Nov 25, 2024
cdfdb4e
check sinkmain
tiffanyvu Nov 25, 2024
26fd75b
remove deleted code
tiffanyvu Nov 25, 2024
46e3dc3
rename to handler
tiffanyvu Nov 25, 2024
5ffe3f3
add env config
tiffanyvu Nov 25, 2024
5d7df68
fix parse
tiffanyvu Nov 25, 2024
1183248
fix event body type
tiffanyvu Nov 25, 2024
5db0419
add origin
tiffanyvu Nov 25, 2024
b8814ee
add logs
tiffanyvu Nov 25, 2024
2bf5fb6
log
tiffanyvu Nov 25, 2024
77740a2
fix test and add log
tiffanyvu Nov 26, 2024
4791235
temp bypass record event
tiffanyvu Nov 26, 2024
444faf2
change order of sinkmain if
tiffanyvu Nov 26, 2024
bfc92eb
push into sinkmain
tiffanyvu Nov 26, 2024
b85b6ef
add id into doc
tiffanyvu Nov 26, 2024
0bfcbc4
log in sinkmain
tiffanyvu Nov 26, 2024
89520bf
format deleted package
tiffanyvu Nov 26, 2024
21f1968
add sink change log
tiffanyvu Nov 26, 2024
a981b4a
change order of sinklog changelog
tiffanyvu Nov 26, 2024
839f218
fix schema
tiffanyvu Nov 26, 2024
4f29673
add logs changelog
tiffanyvu Nov 26, 2024
2830a45
try adding delete property
tiffanyvu Nov 26, 2024
5a7a65a
fix property
tiffanyvu Nov 26, 2024
96bf0d8
log query
tiffanyvu Nov 26, 2024
b4ff2cd
change term
tiffanyvu Nov 26, 2024
8b1e39e
letter s
tiffanyvu Nov 26, 2024
db5e947
must not
tiffanyvu Nov 26, 2024
b350788
wip update package fields
tiffanyvu Nov 26, 2024
c4d5e23
add log
tiffanyvu Nov 26, 2024
bfee3a8
omg
tiffanyvu Nov 27, 2024
2c8a6ff
try fix test and changelog update fields
tiffanyvu Nov 27, 2024
f20f10a
transform updated values?
tiffanyvu Nov 27, 2024
2936adb
typo
tiffanyvu Nov 27, 2024
bab9ab9
again
tiffanyvu Nov 27, 2024
07f656f
try
tiffanyvu Nov 27, 2024
a890751
remove origin
tiffanyvu Nov 27, 2024
231bee9
add timestamp?
tiffanyvu Nov 27, 2024
dd76406
fix changelog not showing
tiffanyvu Nov 27, 2024
428f1e0
edit changelog text
tiffanyvu Nov 27, 2024
26a80a9
meg
tiffanyvu Nov 27, 2024
cd8e0d2
changelog admin changes updates, language TBD
tiffanyvu Nov 27, 2024
b4afe70
edit schema parsing for sinkmain
tiffanyvu Nov 27, 2024
333a867
change schema parsing in changelog
tiffanyvu Nov 27, 2024
c877258
rm commented out code
tiffanyvu Nov 28, 2024
54681e2
fix eslint
tiffanyvu Nov 28, 2024
97fe983
clean up
tiffanyvu Dec 2, 2024
0c703fa
remove more logs
tiffanyvu Dec 3, 2024
78cf0d7
readd comment
tiffanyvu Dec 3, 2024
856adb3
cleanup zod schema logic
tiffanyvu Dec 3, 2024
a63755a
change event name
tiffanyvu Dec 3, 2024
4232711
test adminchange for delete
tiffanyvu Dec 3, 2024
6469a36
attachments is undefined for soft deleted packages, add optional
tiffanyvu Dec 3, 2024
5d3183f
Merge branch 'main' into lambda-update-package
tiffanyvu Dec 4, 2024
56a0ab0
wip
tiffanyvu Dec 5, 2024
634e51c
revert safeparse
tiffanyvu Dec 9, 2024
6aa7b0a
remove empty test
tiffanyvu Dec 9, 2024
c45832d
set default attachments value
tiffanyvu Dec 9, 2024
a511c72
move default attachments default value
tiffanyvu Dec 9, 2024
d2de296
delete test, will be handled by sinkmain tests
tiffanyvu Dec 9, 2024
95f8d5b
edit todo msg
tiffanyvu Dec 9, 2024
58e7321
rm unused code
tiffanyvu Dec 9, 2024
601189e
Merge branch 'main' into lambda-update-package
tiffanyvu Dec 9, 2024
d30bba8
Merge branch 'main' into lambda-update-id
tiffanyvu Dec 9, 2024
0d31401
remove origin property
tiffanyvu Dec 9, 2024
abc9719
Merge branch 'main' into lambda-update-id
tiffanyvu Dec 10, 2024
60c31d4
Merge branch 'main' into lambda-update-package
tiffanyvu Dec 10, 2024
a8a6dbc
test
tiffanyvu Dec 10, 2024
e786b18
check for existing package id
tiffanyvu Dec 10, 2024
35b52ca
add update id schema
tiffanyvu Dec 10, 2024
66f6cd8
add admin change type
tiffanyvu Dec 10, 2024
e979456
typo
tiffanyvu Dec 10, 2024
d0c16ed
again
tiffanyvu Dec 10, 2024
8442250
add log
tiffanyvu Dec 10, 2024
f43b796
edit log
tiffanyvu Dec 10, 2024
b7a04cb
edit changelog
tiffanyvu Dec 10, 2024
b2f7ec0
exclude _index
tiffanyvu Dec 10, 2024
e6a38df
fix destructuring
tiffanyvu Dec 10, 2024
684c055
source?
tiffanyvu Dec 10, 2024
81dafd2
test log
tiffanyvu Dec 11, 2024
01d0e5c
hide changelog
tiffanyvu Dec 11, 2024
37e424b
ew test
tiffanyvu Dec 11, 2024
46cb421
add changemade and types to Document type
tiffanyvu Dec 11, 2024
38633c1
add zod to event body
tiffanyvu Dec 11, 2024
266f791
update response status codes
tiffanyvu Dec 11, 2024
f807d7e
Merge branch 'main' into lambda-update-package
tiffanyvu Dec 11, 2024
a59e302
Merge branch 'main' into lambda-update-id
tiffanyvu Dec 11, 2024
ffaf219
Merge branch 'lambda-update-package' into lambda-update-package-funct…
tiffanyvu Dec 11, 2024
c3fb5f5
merge and rm console log
tiffanyvu Dec 11, 2024
b391d25
rm comment
tiffanyvu Dec 11, 2024
493e31e
wip id validation
tiffanyvu Dec 11, 2024
b742679
Merge branch 'main' into update-package-func
asharonbaltazar Dec 12, 2024
5565b2a
test logs
tiffanyvu Dec 12, 2024
aacc361
Merge branch 'update-package-func' of https://github.com/Enterprise-C…
tiffanyvu Dec 12, 2024
3646259
again
tiffanyvu Dec 12, 2024
978ca6d
add logs
tiffanyvu Dec 12, 2024
556aac6
logs
tiffanyvu Dec 12, 2024
73c359c
fix
tiffanyvu Dec 12, 2024
3b6686b
Merge branch 'main' into update-package-func
asharonbaltazar Dec 12, 2024
d45535b
test format
tiffanyvu Dec 12, 2024
f0da974
log
tiffanyvu Dec 12, 2024
902588d
test moving responses into funcs
tiffanyvu Dec 12, 2024
4b7dc81
test
tiffanyvu Dec 12, 2024
0d42864
fix punctuation
tiffanyvu Dec 13, 2024
af196d6
wip address comments
tiffanyvu Dec 13, 2024
fa838cd
rm old code
tiffanyvu Dec 13, 2024
aad435d
add comment
tiffanyvu Dec 13, 2024
d1febcc
Merge branch 'main' into update-package-func
tiffanyvu Dec 13, 2024
ecceed5
rm eslint ignore
tiffanyvu Dec 13, 2024
7d003b3
move topicname logic
tiffanyvu Dec 13, 2024
2f68a24
test package event type in changelog
tiffanyvu Dec 13, 2024
15f00c0
log
tiffanyvu Dec 13, 2024
654aa38
logs
tiffanyvu Dec 13, 2024
64177f3
fix
tiffanyvu Dec 13, 2024
3479e33
move event
tiffanyvu Dec 13, 2024
f3df9b1
move logic
tiffanyvu Dec 13, 2024
78a4d61
logs
tiffanyvu Dec 13, 2024
d86b23f
typo
tiffanyvu Dec 13, 2024
dbcd80f
what
tiffanyvu Dec 13, 2024
b172846
log
tiffanyvu Dec 13, 2024
24e86fd
add mako origin to get to transforms
tiffanyvu Dec 13, 2024
5d968b4
umm touching opensearch
tiffanyvu Dec 13, 2024
a81684e
typo
tiffanyvu Dec 14, 2024
f1bdb10
logs
tiffanyvu Dec 16, 2024
f9e5858
revert
tiffanyvu Dec 16, 2024
7c897d1
try
tiffanyvu Dec 16, 2024
be77182
revert
tiffanyvu Dec 16, 2024
482422c
revert changelog
tiffanyvu Dec 16, 2024
2671c39
hi
tiffanyvu Dec 16, 2024
6aed3b0
logs
tiffanyvu Dec 16, 2024
176e985
add old id to changelog
tiffanyvu Dec 16, 2024
1d25971
offset stuff
tiffanyvu Dec 16, 2024
140485b
logs
tiffanyvu Dec 16, 2024
7b967ed
logs
tiffanyvu Dec 16, 2024
a1e0ea4
duplicate offset
tiffanyvu Dec 16, 2024
31aafc0
fix
tiffanyvu Dec 17, 2024
0a5494b
log hits
tiffanyvu Dec 17, 2024
85217cf
logs
tiffanyvu Dec 17, 2024
9879466
test
tiffanyvu Dec 17, 2024
0a83a4b
fix
tiffanyvu Dec 17, 2024
090e570
fix
tiffanyvu Dec 17, 2024
257ed25
idToBeUpdated
tiffanyvu Dec 17, 2024
5df41a9
add onemac
tiffanyvu Dec 17, 2024
d06cf1d
fix
tiffanyvu Dec 17, 2024
6e1e5e2
clean
tiffanyvu Dec 17, 2024
c0c2906
move around
tiffanyvu Dec 17, 2024
a3575f1
clean
tiffanyvu Dec 17, 2024
622aad9
remove casting
tiffanyvu Dec 18, 2024
2d33626
casting throws type error
tiffanyvu Dec 18, 2024
099d878
sinkmain conflicts
tiffanyvu Dec 18, 2024
b9bd169
fix import and add test
tiffanyvu Dec 18, 2024
ce4ac62
Merge branch 'main' into update-package-func
tiffanyvu Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
change schema parsing in changelog
tiffanyvu committed Nov 27, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 333a867ce6d2b4ebd614e6db717bd33d3faf2daf
98 changes: 72 additions & 26 deletions lib/lambda/sinkChangelog.ts
Original file line number Diff line number Diff line change
@@ -66,6 +66,10 @@ const processAndIndex = async ({
for (const kafkaRecord of kafkaRecords) {
console.log(JSON.stringify(kafkaRecord, null, 2));
const { value, offset } = kafkaRecord;

// enforce type?
let transformedData;

try {
// If a legacy tombstone, continue
if (!value) {
@@ -74,46 +78,88 @@ const processAndIndex = async ({

// Parse the kafka record's value
const record = JSON.parse(decodeBase64WithUtf8(value));
// enforce type?
let transformedData;

if (record.isAdminChange) {
if (record.adminChangeType === "delete") {
const deletedPackageSchema = z.object({
const adminChangeSchemas = {
delete: z
.object({
id: z.string(),
deleted: z.boolean(),
});

transformedData = deletedPackageSchema.transform((schema) => ({
})
.transform((schema) => ({
...schema,
event: "soft-delete",
packageId: schema.id,
id: `${schema.id}-${offset}`,
timestamp: Date.now(),
}));

const result = transformedData.safeParse(record);

if (result.success) {
docs.push(result.data);
} else {
console.log("Skipping package with invalid format", result.error.message);
}
}

if (record.adminChangeType === "update-values") {
transformedData = {
...record,
})),
"update-values": z
.object({
id: z.string(),
})
.transform((schema) => ({
...schema,
event: "update-values",
packageId: record.id,
id: `${record.id}-${offset}`,
packageId: schema.id,
id: `${schema.id}-${offset}`,
timestamp: Date.now(),
};
})),
};

if (record.isAdminChange && record.adminChangeType in adminChangeSchemas) {
const schema =
adminChangeSchemas[record.adminChangeType as keyof typeof adminChangeSchemas];

docs.push(transformedData);
const result = schema.safeParse(record);

if (result.success) {
docs.push(result.data); // Use transformed data
} else {
console.log(
`Skipping package with invalid format for type "${record.adminChangeType}"`,
result.error.message,
);
}
} else if (record.isAdminChange) {
console.log(`Unknown adminChangeType: ${record.adminChangeType}`);
}

// if (record.isAdminChange) {
// if (record.adminChangeType === "delete") {
// const deletedPackageSchema = z.object({
// id: z.string(),
// deleted: z.boolean(),
// });

// transformedData = deletedPackageSchema.transform((schema) => ({
// ...schema,
// event: "soft-delete",
// packageId: schema.id,
// id: `${schema.id}-${offset}`,
// timestamp: Date.now(),
// }));

// const result = transformedData.safeParse(record);

// if (result.success) {
// docs.push(result.data);
// } else {
// console.log("Skipping package with invalid format", result.error.message);
// }
// }

// if (record.adminChangeType === "update-values") {
// transformedData = {
// ...record,
// event: "update-values",
// packageId: record.id,
// id: `${record.id}-${offset}`,
// timestamp: Date.now(),
// };

// docs.push(transformedData);
// }
// }

// If we're not a mako event, continue
// TODO: handle legacy. for now, just continue
if (!record.event || record?.origin !== "mako") {
Loading