Skip to content

Latest commit

 

History

History
330 lines (255 loc) · 10.3 KB

load-aws-cloudformation-queue.md

File metadata and controls

330 lines (255 loc) · 10.3 KB

How to load AWS Cloudformation queue

Overview

Instructions for deploying AWS Cloudformation ECS POC simple and loading the SQS queue from a local workstation.

Contents

  1. Launch Cloudformation template
  2. Load SQS
    1. Load SQS from Linux or macOS
    2. Load SQS from Windows
  3. Manual Cloudformation template Launch

Launch Cloudformation template

  1. Launch Cloudformation template.

    1. Visit AWS Cloudformation with Senzing template
      1. In lower-right, click on "Next" button.
      2. If access error, see Manual Cloudformation template Launch to load Cloudformation template. Then proceed to next step.
    2. In Specify stack details
      1. In Stack name
        1. Stack name: senzing-poc
      2. In Parameters
        1. In Acknowledge insecure system
          1. Read, understand, and agree to the nature of the security in the deployment.
        2. In Senzing installation
          1. Accept the End User Licence Agreement
      3. In Options
        1. For "Optional: Would you like to have sample data imported?", select "No".
      4. Other parameters are optional.
      5. In lower-right, click "Next" button.
    3. In Configure stack options
      1. In lower-right, click "Next" button.
    4. In Review senzing-poc
      1. Near the bottom, in Capabilities
        1. Check ":ballot_box_with_check: I acknowledge that AWS CloudFormation might create IAM resources."
      2. In lower-right, click "Create stack" button.
    5. Visit AWS CloudFormation console.
      1. Make sure correct AWS region is selected.
    6. Wait until "senzing-poc" status is CREATE_COMPLETE.
      1. Senzing formation takes about 15 minutes to fully deploy.
      2. May have to hit the refresh button a few times to get updated information.

  2. Add data sources to Senzing configuration.

    1. In the Cloudformation "Outputs" tab, launch the UrlXterm URL.

    2. ✏️ In the Xterm, run G2ConfigTool.py to add data sources. Example:

      $ G2ConfigTool.py

(g2cfg) addDataSource MY_DATA_SOURCE
(g2cfg) addEntityType MY_ENTITY_TYPE
(g2cfg) save

  3. The next steps are done from your local workstation. Depending on your workstation operating system, visit:

    1. Load SQS from Docker
    2. Load SQS from Linux or macOS
    3. Load SQS from Windows

Load SQS

Load SQS from Docker

  1. Get latest docker image. Example:

    docker pull senzing/stream-producer:latest

  2. ✏️ Set AWS variables. Example:

    export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
export AWS_SECRET_ACCESS_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
export AWS_DEFAULT_REGION=us-xxxx-1

  3. ✏️ Identify file to be loaded into Senzing. Example:

    export SENZING_INPUT_FILE=/path/to/my/example.json

  4. ✏️ Set default DATA_SOURCE and ENTITY_TYPE values. Example:

    export SENZING_DEFAULT_DATA_SOURCE=MY_DATA_SOURCE
export SENZING_DEFAULT_ENTITY_TYPE=MY_ENTITY_TYPE

  5. ✏️ Set URL of SQS queue Example:

    export SENZING_SQS_QUEUE_URL="https://sqs.us-xxxx-1.amazonaws.com/000000000000/senzing-poc-sqs-queue"

  6. ✏️ Set stream-producer.py subcommand. Example:

    export SENZING_SUBCOMMAND=json-to-sqs

  7. Run docker container. Example:

    docker run \
  --env AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  --env AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION} \
  --env AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  --env SENZING_DEFAULT_DATA_SOURCE=${SENZING_DEFAULT_DATA_SOURCE} \
  --env SENZING_DEFAULT_ENTITY_TYPE=${SENZING_DEFAULT_ENTITY_TYPE} \
  --env SENZING_INPUT_URL="/data/${SENZING_INPUT_FILE##*/}" \
  --env SENZING_SQS_QUEUE_URL=${SENZING_SQS_QUEUE_URL} \
  --env SENZING_SUBCOMMAND=${SENZING_SUBCOMMAND} \
  --interactive \
  --rm \
  --tty \
  --volume ${SENZING_INPUT_FILE%\/*}:/data \
  senzing/stream-producer

Load SQS from Linux or macOS

  1. Install stream-producer.

    1. Install prerequisites. Example:

      pip3 install -r https://raw.githubusercontent.com/Senzing/stream-producer/main/requirements.txt

    2. ✏️ Specify location for stream-producer.py. Example:

      export SENZING_DOWNLOAD_FILE=/tmp/stream-producer.py

    3. Download stream-producer.py. Example:

      curl -X GET \
  --output ${SENZING_DOWNLOAD_FILE} \
  https://raw.githubusercontent.com/Senzing/stream-producer/main/stream-producer.py

  2. Load SQS queue using stream-producer.

    1. ✏️ Set AWS variables: Example:

      export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
export AWS_SECRET_ACCESS_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
export AWS_DEFAULT_REGION=us-xxxx-1

    2. ✏️ Identify file to be loaded into Senzing. Example:

      export SENZING_INPUT_URL=file:///path/to/my/example.json

    3. ✏️ Set default DATA_SOURCE and ENTITY_TYPE values. Example:

      export SENZING_DEFAULT_DATA_SOURCE=MY_DATA_SOURCE
export SENZING_DEFAULT_ENTITY_TYPE=MY_ENTITY_TYPE

    4. ✏️ Set URL of SQS queue Example:

      export SENZING_SQS_QUEUE_URL="https://sqs.us-xxxx-1.amazonaws.com/000000000000/senzing-poc-sqs-queue"

    5. ✏️ Set subcommand for stream-producer.py Example:

      export SENZING_SUBCOMMAND="json-to-sqs"

    6. Read file and send to SQS queue. Example:

      python /tmp/stream-producer.py ${SENZING_SUBCOMMAND} \
    --default-data-source ${SENZING_DEFAULT_DATA_SOURCE} \
    --default-entity-type ${SENZING_DEFAULT_ENTITY_TYPE} \
    --input-url ${SENZING_INPUT_URL} \
    --sqs-queue-url ${SENZING_SQS_QUEUE_URL}

Load SQS from Windows

  1. Install stream-producer.

    1. Install prerequisites. Example:

      pip install -r https://raw.githubusercontent.com/Senzing/stream-producer/main/requirements.txt

    2. ✏️ Specify location for stream-producer.py. Example:

      SET SENZING_DOWNLOAD_FILE=C:\tmp\stream-producer.py

    3. Download stream-producer.py. Example:

      curl -X GET ^
  --output %SENZING_DOWNLOAD_FILE% ^
  https://raw.githubusercontent.com/Senzing/stream-producer/main/stream-producer.py

  2. Load SQS queue using stream-producer.

    1. ✏️ Set AWS variables: Example:

      SET AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
SET AWS_SECRET_ACCESS_KEY=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
SET AWS_DEFAULT_REGION=us-xxxx-1

    2. ✏️ Identify file to be loaded into Senzing. Example:

      SET SENZING_INPUT_URL=C:\tmp\input-file.csv

    3. ✏️ Set default DATA_SOURCE and ENTITY_TYPE values. Example:

      SET SENZING_DEFAULT_DATA_SOURCE=MY_DATA_SOURCE
SET SENZING_DEFAULT_ENTITY_TYPE=MY_ENTITY_TYPE

    4. ✏️ Set URL of SQS queue Example:

      SET SENZING_SQS_QUEUE_URL="https://sqs.us-xxxx-1.amazonaws.com/000000000000/senzing-poc-sqs-queue"

    5. ✏️ Set subcommand for stream-producer.py Example:

      SET SENZING_SUBCOMMAND="json-to-sqs"

    6. Read file and send to SQS queue. Example:

      python C:\tmp\stream-producer.py %SENZING_SUBCOMMAND% ^
    --default-data-source %SENZING_DEFAULT_DATA_SOURCE% ^
    --default-entity-type %SENZING_DEFAULT_ENTITY_TYPE% ^
    --input-url %SENZING_INPUT_URL% ^
    --sqs-queue-url %SENZING_SQS_QUEUE_URL%

Manual Cloudformation template Launch

Windows - Manual Cloudformation template launch

  1. ✏️ Identify the cloudformation.yaml to be downloaded. Example:

    SET SENZING_CLOUDFORMATION_URL=https://raw.githubusercontent.com/Senzing/aws-cloudformation-ecs-poc-simple/main/cloudformation.yaml

  2. ✏️ Identify where cloudformation.yaml file will be downloaded. Example:

    SET SENZING_DOWNLOAD_FILE=C:\tmp\cloudformation.yaml

  3. Download cloudformation.yaml. Example:

    curl -X GET --output %SENZING_DOWNLOAD_FILE% %SENZING_CLOUDFORMATION_URL%

  4. Visit AWS Cloudformation management console

  5. Click "Create stack" drop-down > "With new resources (standard)".

  6. In "Create stack":

    1. Click "Upload a template file" button.
    2. Click "Choose file" button and locate the file specified by SENZING_DOWNLOAD_FILE.
    3. Click "Next" button.

  7. Return to instructions for Specify stack details.