Code

One time batch landings#37

Code

Merged

rivernews merged 56 commits intodevrivernews/media-literacy:devfrom one-time-batch-landingsrivernews/media-literacy:one-time-batch-landingsCopy head branch name to clipboardOct 2, 2022

+1,965-643

Commits

Commits on Aug 21, 2022

• temp store all

  rivernews

  committed
  e5b9214

  Copy full SHA for e5b9214

• remove go_poc

  rivernews

  committed
  5333981

  Copy full SHA for 5333981

Commits on Sep 14, 2022

• upgrade so project runs on M1

  rivernews

  committed
  89c666c

  Copy full SHA for 89c666c

Commits on Sep 15, 2022

• Try S3 notification

  rivernews

  committed
  5c60040

  Copy full SHA for 5c60040

• Fix prefix to include newssite alias

  rivernews

  committed
  d5d8186

  Copy full SHA for d5d8186

Commits on Sep 16, 2022

• Fix aws lambda PathError issue

  rivernews

  committed
  fa266bf

  Copy full SHA for fa266bf

Commits on Sep 17, 2022

• Merge pull request #27 from rivernews/temp--s3-notif

  Show description for 24a13ad
  rivernews

  authored
  24a13ad

  Copy full SHA for 24a13ad

• Save to metadata.json complete

  rivernews

  committed
  0f7ab88

  Copy full SHA for 0f7ab88

• add untitled stories in metadata.json

  rivernews

  committed
  b6851c5

  Copy full SHA for b6851c5

• rename stories function to landing_metadata

  rivernews

  committed
  6c6e250

  Copy full SHA for 6c6e250

• rename batch stories fetch tf to metadata

  rivernews

  committed
  512f729

  Copy full SHA for 512f729

• Improved metadata access s3 event

  rivernews

  committed
  9eca283

  Copy full SHA for 9eca283

• Metadata.json trigger computing env

  rivernews

  committed
  a62c33d

  Copy full SHA for a62c33d

• read parse metadata.json

  rivernews

  committed
  bda216e

  Copy full SHA for bda216e

• fetch a story POC

  Show description for e29fdc8
  rivernews

  committed
  e29fdc8

  Copy full SHA for e29fdc8

• Sfn map parallism POC

  Show description for d2b025e
  rivernews

  committed
  d2b025e

  Copy full SHA for d2b025e

Commits on Sep 18, 2022

• randomize requests

  rivernews

  committed
  ba830cd

  Copy full SHA for ba830cd

• Merge pull request #29 from rivernews/fetch-stories

  Show description for 9ad201f
  rivernews

  authored
  9ad201f

  Copy full SHA for 9ad201f

Commits on Sep 20, 2022

• Refactor to allow individual tf modules

  Show description for 98b3bce
  rivernews

  committed
  98b3bce

  Copy full SHA for 98b3bce

• Merge pull request #30 from rivernews/individual-modules

  Show description for 8e8ef7f
  rivernews

  authored
  8e8ef7f

  Copy full SHA for 8e8ef7f

Commits on Sep 21, 2022

• scaffold table

  rivernews

  committed
  578b06f

  Copy full SHA for 578b06f

Commits on Sep 22, 2022

• draft table design

  rivernews

  committed
  58af865

  Copy full SHA for 58af865

Commits on Sep 25, 2022

• create table

  rivernews

  committed
  a0c103a

  Copy full SHA for a0c103a

• Merge pull request #31 from rivernews/provision-table

  Show description for 42d75ea
  rivernews

  authored
  42d75ea

  Copy full SHA for 42d75ea

• Merge pull request #32 from rivernews/table_for_metadata

  Show description for 7a2e9b5
  rivernews

  authored
  7a2e9b5

  Copy full SHA for 7a2e9b5

• Draining mechanism draft - identify all TODOs

  Show description for eb61eef
  rivernews

  committed
  eb61eef

  Copy full SHA for eb61eef

• Draft for put landing page; identified TODOs

  Show description for ec37c70
  rivernews

  committed
  ec37c70

  Copy full SHA for ec37c70

• Complete tf surgery; Identify all TODOs in golang

  Show description for d816fe6
  rivernews

  committed
  d816fe6

  Copy full SHA for d816fe6

Commits on Sep 26, 2022

• fix compile error; progress in metadata cronjob add query

  rivernews

  committed
  2c8f841

  Copy full SHA for 2c8f841

Commits on Sep 29, 2022

• Ready to test

  rivernews

  committed
  6ad69b6

  Copy full SHA for 6ad69b6

• Fix db field first char not lowercase

  Show description for c24ba2d
  rivernews

  committed
  c24ba2d

  Copy full SHA for c24ba2d

• Fix permission of db index, S3 pull

  Show description for 233ce60
  rivernews

  committed
  233ce60

  Copy full SHA for 233ce60

• All tests complete

  Show description for d2f752c
  rivernews

  committed
  d2f752c

  Copy full SHA for d2f752c

Commits on Sep 30, 2022

• Move landing PutItem out to s3 trigger lambda; ready for S3 batch move

  rivernews

  committed
  8d72d88

  Copy full SHA for 8d72d88

• create reusable lambda module; optimize package size

  Show description for 3977565
  rivernews

  committed
  3977565

  Copy full SHA for 3977565

• Fix golang build path

  rivernews

  committed
  4b75811

  Copy full SHA for 4b75811

• Refactor to use our custom lambda module

  rivernews

  committed
  68b4963

  Copy full SHA for 68b4963

• add landing s3 trigger

  rivernews

  committed
  2643218

  Copy full SHA for 2643218

• rm golang module stories that are renamed

  rivernews

  committed
  17e652a

  Copy full SHA for 17e652a

• Fix env var

  rivernews

  committed
  5df3517

  Copy full SHA for 5df3517

• Fix permission for PutItem move from landing to s3 trigger

  rivernews

  committed
  1b79829

  Copy full SHA for 1b79829

• Fix metadata s3 trigger not fired

  rivernews

  committed
  31abe5a

  Copy full SHA for 31abe5a

• Fix s3 trigger not working - S3 notification can only have one resource

  rivernews

  committed
  0917331

  Copy full SHA for 0917331

• Make it easier to test

  rivernews

  committed
  0258564

  Copy full SHA for 0258564

• prod grade setting enabled

  rivernews

  committed
  f19edcb

  Copy full SHA for f19edcb

• In Sfn pin lambda version, so rolling deploy works better for lambda

  rivernews

  committed
  ef44117

  Copy full SHA for ef44117

• Display sfn map result / target stories count info in finalizer

  rivernews

  committed
  b5d71d3

  Copy full SHA for b5d71d3

• stop landing s3 trigger from sending slack logs

  Show description for 9e5bbd6
  rivernews

  committed
  9e5bbd6

  Copy full SHA for 9e5bbd6

• Let Sfn pin lambda version

  Show description for 862ee64
  rivernews

  committed
  862ee64

  Copy full SHA for 862ee64

Commits on Oct 1, 2022

• improve log for metadata trigger

  rivernews

  committed
  a8b7027

  Copy full SHA for a8b7027

• improve cronjob log

  rivernews

  committed
  6d4a032

  Copy full SHA for 6d4a032

• log cronjob event for better understanding of how it get triggered

  rivernews

  committed
  f999c13

  Copy full SHA for f999c13

• Disable cronjob to better debug

  Show description for 7ccc4d4
  rivernews

  committed
  7ccc4d4

  Copy full SHA for 7ccc4d4

• workaround to scale up our Sfn pipeline

  Show description for 4bd649f
  rivernews

  committed
  4bd649f

  Copy full SHA for 4bd649f

• improve log for landing S3 trigger

  rivernews

  committed
  f475441

  Copy full SHA for f475441

• re-enable prod config plus cronjob

  rivernews

  committed
  4df9c3c

  Copy full SHA for 4df9c3c