Skip to content

Latest commit

 

History

History
293 lines (211 loc) · 10.7 KB

README.md

File metadata and controls

293 lines (211 loc) · 10.7 KB

docker-compose-kafka-postgresql

Synopsis

Using docker-compose, bring up a Senzing stack using Kafka and a PostgreSQL database.

Overview

This repository illustrates a reference implementation of Senzing using Kafka as the queue and PostgreSQL as the underlying database.

The instructions show how to set up a system that:

  1. Reads JSON lines from a file on the internet and sends each JSON line to a message queue via the Senzing stream-producer.
    1. In this implementation, the queue is Kafka.
  2. Reads messages from the queue and inserts into Senzing via the Senzing stream-loader.
    1. In this implementation, Senzing keeps its data in a PostgreSQL database.
  3. Reads information from Senzing via Senzing API Server server.
  4. Views resolved entities in a web app.

The following diagram shows the relationship of the Docker containers in this Docker composition. Arrows represent data flow.

Image of architecture

Contents

  1. Prerequisites
  2. Demonstrate
  3. Advanced
  4. Related artifacts
  5. References

Preamble

At Senzing, we strive to create GitHub documentation in a "don't make me think" style. For the most part, instructions are copy and paste. Icons are used to signify additional actions by the user. If the instructions are not clear, please let us know by opening a new Documentation issue describing where we can improve. Now on with the show...

Expectations

  • Space: This repository and demonstration require 7 GB free disk space.
  • Time: Budget 2 hours to get the demonstration up-and-running, depending on CPU and network speeds.
  • Background knowledge: This repository assumes a working knowledge of:

Prerequisites

  1. docker - Minimum version: 20.10.16
  2. docker-compose - Minimum version: 1.29.0

Demonstrate

  1. ✏️ Specify a new directory to hold demonstration artifacts on the local host. Example:

    export SENZING_DEMO_DIR=~/my-senzing
    1. ⚠️ macOS - File sharing must be enabled for SENZING_DEMO_DIR.
    2. ⚠️ Windows - File sharing must be enabled for SENZING_DEMO_DIR.

  2. Set environment variables. Example:

    export PGADMIN_DIR=${SENZING_DEMO_DIR}/pgadmin
export POSTGRES_DIR=${SENZING_DEMO_DIR}/postgres
export SENZING_GID=$(id -g)
export SENZING_UID=$(id -u)
export SENZING_VAR_DIR=${SENZING_DEMO_DIR}/var

  3. Create directories. Example:

    mkdir -p ${PGADMIN_DIR} ${POSTGRES_DIR} ${SENZING_VAR_DIR}
chmod -R 777 ${SENZING_DEMO_DIR}

  4. Get versions of Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-versions-stable.sh \
    https://raw.githubusercontent.com/Senzing/knowledge-base/main/lists/docker-versions-stable.sh
source ${SENZING_DEMO_DIR}/docker-versions-stable.sh

  5. Download docker-compose.yaml and Docker images. Example:

    curl -X GET \
    --output ${SENZING_DEMO_DIR}/docker-compose.yaml \
    "https://raw.githubusercontent.com/senzing-garage/docker-compose-demo/main/resources/postgresql/docker-compose-kafka-postgresql.yaml"
cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose pull

  6. Bring up Senzing docker-compose stack. Example:

    cd ${SENZING_DEMO_DIR}
sudo --preserve-env docker-compose up

  7. Allow time for the components to be downloaded, start, and initialize.

    1. There will be errors in some Docker logs as they wait for dependent services to become available. docker-compose isn't the best at orchestrating Docker container dependencies.

View formation

Once the docker-compose formation is running, different aspects of the formation can be viewed.

Username and password for the following sites were either passed in as environment variables or are the default values seen in docker-compose-kafka-postgresql.yaml.

View Docker containers

  1. A good tool to monitor individual Docker logs is Portainer. When running, Portainer is viewable at localhost:9170.

View Senzing Entity Search WebApp

  1. Senzing Entity Search WebApp is viewable at localhost:8251.
  2. See additional tips for working with Senzing Entity Search WebApp.

View X-Term

The web-based Senzing X-term can be used to run Senzing command-line programs.

  1. Senzing X-term is viewable at localhost:8254.
  2. See additional tips for working with Senzing X-Term.

Use SSH

Instructions to use the senzing/sshd container are viewable in the senzing/docker-sshd repository

View Jupyter notebooks

  1. Change file permissions on database files. Example:

    sudo chmod 777 -R ${POSTGRES_DIR}

  2. Jupyter Notebooks are viewable at localhost:9178.

  3. See additional tips for working with Jupyter Notebooks.

View Senzing API Server

View results from Senzing REST API server. The server supports the Senzing REST API.

  1. The OpenApi Editor with Servers value of http://localhost:8250 can be used to try the Senzing REST API.
  2. Example Senzing REST API request: localhost:8250/heartbeat
  3. See additional tips for working with Senzing API server.

View Kafka

  1. Kafdrop is viewable at localhost:9179.
  2. See additional tips for working with Kafka.

View PostgreSQL

  1. PostgreSQL is viewable at localhost:9171.
    1. Database defaults: username: postgres password: postgres
  2. See additional tips for working with PostgreSQL.

Cleanup

When the docker-compose formation is no longer needed, it can be brought down and directories can be deleted.

  1. Bring down Docker formation. Example:

    cd ${SENZING_DEMO_DIR}
sudo docker-compose down

  2. Remove directories from host system. Example:

    rm -rf ${SENZING_DEMO_DIR:-/tmp/nowhere/for/safety}

Advanced

The following topics discuss variations to the basic docker-compose demonstration.

  1. SSH port
  2. Set sshd password

Docker images

This docker formation brings up the following docker containers:

  1. bitnami/kafka
  2. bitnami/postgres
  3. bitnami/zookeeper
  4. busybox
  5. dpage/pgadmin4
  6. obsidiandynamics/kafdrop
  7. senzing/console
  8. senzing/entity-web-search-app
  9. senzing/jupyter
  10. senzing/redoer
  11. senzing/senzing-poc-server
  12. senzing/senzing-tools
  13. senzing/sshd
  14. senzing/stream-loader
  15. senzing/stream-producer
  16. senzing/xterm

Configuration

Configuration values specified by environment variable or command line parameter.

Related artifacts

  1. DockerHub

References