Skip to content

Commit

Permalink
Merge branch 'raft-tdp-main' into adp/usermgmt-updates
Browse files Browse the repository at this point in the history
  • Loading branch information
andrew-jameson authored Jul 29, 2021
2 parents 0b2bc0c + eb28b36 commit 34e77c7
Show file tree
Hide file tree
Showing 30 changed files with 658 additions and 633 deletions.
80 changes: 62 additions & 18 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,33 @@ commands:
name: Ensure sudo is installed, otherwise install it.
command: ./scripts/sudo-check.sh

upload-codecov:
description: Uploads testing code coverage results to Codecov
parameters:
component:
description: The component of the application being tested, either backend or frontend.
type: enum
enum: ["backend", "frontend"]
coverage-report:
description: The path to the coverage report being uploaded.
type: string
steps:
- run:
name: Ensure Codecov uploader is installed, otherwise install it.
command: ./scripts/codecov-check.sh
- run:
name: Determine Codecov metric flag
command: |
if [ "$CIRCLE_BRANCH" == "main" ] ; then
CURRENT_FLAG=main-<<parameters.component>>
else
CURRENT_FLAG=dev-<<parameters.component>>
fi
echo "export CURRENT_FLAG=$CURRENT_FLAG" >> $BASH_ENV
- run:
name: Upload code coverage report if target branch
command: codecov -t "$CODECOV_TOKEN" -f <<parameters.coverage-report>> -F "$CURRENT_FLAG"

###
# Deployment commands
#
Expand Down Expand Up @@ -287,20 +314,32 @@ jobs:
command: |
cd tdrs-backend;
docker-compose run --rm web bash -c "./wait_for_services.sh && pytest --cov-report=xml"
- run:
name: "Upload Backend Coverage Report to CodeCov If On Target Branch"
command: |
if [ "$CIRCLE_BRANCH" == "main" ] ; then
cd tdrs-backend; bash <(curl -s https://codecov.io/bash) -f ./coverage.xml -F main-backend
elif [ "$CIRCLE_BRANCH" == "raft-tdp-main" ] ; then
cd tdrs-backend; bash <(curl -s https://codecov.io/bash) -f ./coverage.xml -F dev-backend
fi
- upload-codecov:
component: backend
coverage-report: ./tdrs-backend/coverage.xml
- run:
name: "Test: Execute Python OWASP ZAP Vulnerability Scan"
command: "cd tdrs-backend; ./zap-scanner.sh circle"
command: "./scripts/zap-scanner.sh backend circle"
- store_artifacts:
path: tdrs-backend/reports/owasp_report.html

prod-owasp-scan:
executor: machine-executor
working_directory: ~/tdp-apps
steps:
- checkout
- docker-compose-check
- run:
name: "Test: Execute Frontend OWASP ZAP Vulnerability Scan"
command: "./scripts/zap-scanner.sh frontend nightly"
- store_artifacts:
path: tdrs-frontend/reports/owasp_report.html

- run:
name: "Test: Execute Backend OWASP ZAP Vulnerability Scan"
command: "./scripts/zap-scanner.sh backend nightly"
- store_artifacts:
path: tdrs-backend/reports/owasp_report.html
test-frontend:
executor: machine-executor
working_directory: ~/tdp-apps
Expand All @@ -323,14 +362,9 @@ jobs:
- run:
name: "Run Jest Unit Tests"
command: "cd tdrs-frontend; yarn test:ci"
- run:
name: "Upload Frontend Coverage Report to CodeCov If Target Branch"
command: |
if [ "$CIRCLE_BRANCH" == "main" ] ; then
cd tdrs-frontend; bash <(curl -s https://codecov.io/bash) -f ./coverage/lcov.info -F main-frontend
elif [ "$CIRCLE_BRANCH" == "raft-tdp-main" ] ; then
cd tdrs-frontend; bash <(curl -s https://codecov.io/bash) -f ./coverage/lcov.info -F dev-frontend
fi
- upload-codecov:
component: frontend
coverage-report: ./tdrs-frontend/coverage/lcof.info
- run:
name: "Install lib bindings for Cypress"
command: |
Expand All @@ -341,7 +375,7 @@ jobs:
command: "cd tdrs-frontend; yarn cy:run:ci"
- run:
name: "Test: Execute Frontend OWASP ZAP Vulnerability Scan"
command: "cd tdrs-frontend; ./zap-scanner.sh circle"
command: "./scripts/zap-scanner.sh frontend circle"
- store_artifacts:
path: tdrs-frontend/reports/owasp_report.html
- store_artifacts:
Expand Down Expand Up @@ -435,6 +469,16 @@ jobs:
backend-appname: tdp-backend-sandbox
frontend-appname: tdp-frontend-sandbox
workflows:
nightly:
jobs:
- prod-owasp-scan
triggers:
- schedule:
cron: "0 0 * * *"
filters:
branches:
only:
- raft-tdp-main
build-and-test:
jobs:
- test-backend
Expand Down
10 changes: 5 additions & 5 deletions docs/Technical-Documentation/TDP-environments-README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,12 +55,12 @@ Like Staging, there is only one Production deployment. Note that developers do *

- `clamav-rest` - Virus scanner REST service used to scan file uploads.

**Cloud.gov AWS RDS `(tanf-production)`**
- `tdp-db-production` - Stores application-level models (e.g. Users, Reports).
**Cloud.gov AWS RDS `(tanf-prod)`**
- `tdp-db-prod` - Stores application-level models (e.g. Users, Reports).

**Cloud.gov AWS S3 `(tanf-production)`**
- `tdp-staticfiles-production` - Stores static HTML/CSS for Django Admin.
- `tdp-datafiles-production` - Stores the files uploaded by STTs.
**Cloud.gov AWS S3 `(tanf-prod)`**
- `tdp-staticfiles-prod` - Stores static HTML/CSS for Django Admin.
- `tdp-datafiles-prod` - Stores the files uploaded by STTs.
- `tdp-tf-states` - Stores the Terraform state files used to create and re-recreate services infrastructure.

## External Dependencies
Expand Down
52 changes: 26 additions & 26 deletions docs/Technical-Documentation/Zap-Scan-HTML-Report.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
# OWASP Zap Scan HTML Report

We have added a feature to the CI/CD which produces a report in HTML for the OWASP Zap scan.
This will make it much easier to review the results of these scans. This document shows how
to access the report.

### Accessing the HTML Report

1. When a pull request is issued, the CI/CD checks automatically kick off. Go to the bottom of the

Conversation screen to see the checks running.

![image](images/ci-cd-checks.png)

2. To the right of either `ci/circleci: test-backend` or `ci/circleci: test-backend` click the `Details`
link to view the running process at CircleCI

![image](images/circleci.png)

3. Click the `Artifacts` tab at the top of the screen once the process has completed.

![image](images/artifacts.png)

4. Click the `owasp_report.html` link to view the report.

![image](images/report.png)
# OWASP Zap Scan HTML Report

We have added a feature to the CI/CD which produces a report in HTML for the OWASP Zap scan.
This will make it much easier to review the results of these scans. This document shows how
to access the report.

### Accessing the HTML Report

1. When a pull request is issued, the CI/CD checks automatically kick off. Go to the bottom of the

Conversation screen to see the checks running.

![image](images/ci-cd-checks.png)

2. To the right of either `ci/circleci: test-backend` or `ci/circleci: test-backend` click the `Details`
link to view the running process at CircleCI

![image](images/circleci.png)

3. Click the `Artifacts` tab at the top of the screen once the process has completed.

![image](images/artifacts.png)

4. Click the `owasp_report.html` link to view the report.

![image](images/report.png)
28 changes: 28 additions & 0 deletions scripts/codecov-check.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -e
if command -v codecov > /dev/null 2>&1; then
echo The command codecov is available
else
echo The command codecov is not available, installing...
set -x

echo Importing Codecov PGP public keys...
curl https://keybase.io/codecovsecurity/pgp_keys.asc | gpg --import

echo Downloading codecov uploader...
curl -Os https://uploader.codecov.io/latest/linux/codecov

echo Downloading SHA signatures...
curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM
curl -Os https://uploader.codecov.io/latest/linux/codecov.SHA256SUM.sig

echo Verifying package integrity...
sha256sum -c codecov.SHA256SUM
gpg --verify codecov.SHA256SUM.sig codecov.SHA256SUM

echo Validation successful, completing installation...
chmod +x codecov
rm codecov.SHA256SUM
rm codecov.SHA256SUM.sig
sudo mv codecov /usr/bin/
fi
2 changes: 1 addition & 1 deletion scripts/deploy-backend.sh
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ bind_backend_to_services() {
cf bind-service "$CGHOSTNAME_BACKEND" "tdp-datafiles-${env}"
cf bind-service "$CGHOSTNAME_BACKEND" "tdp-db-${env}"

bash ./scripts/set-backend-env-vars.sh "$CGHOSTNAME_BACKEND"
bash ./scripts/set-backend-env-vars.sh "$CGHOSTNAME_BACKEND" "$CF_SPACE"

cf restage "$CGHOSTNAME_BACKEND"
}
Expand Down
20 changes: 11 additions & 9 deletions scripts/set-backend-env-vars.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env bash

CGAPPNAME_BACKEND=$1
CG_SPACE=$2

# Determine the appropriate BASE_URL for the deployed instance based on the
# provided Cloud.gov App Name
Expand All @@ -19,6 +20,16 @@ else
FRONTEND_BASE_URL="$DEFAULT_FRONTEND_ROUTE"
fi

# Dynamically set DJANGO_CONFIGURATION based on Cloud.gov Space
DJANGO_SETTINGS_MODULE="tdpservice.settings.cloudgov"
if [ "$CG_SPACE" = "tanf-prod" ]; then
DJANGO_CONFIGURATION="Production"
elif [ "$CG_SPACE" = "tanf-staging" ]; then
DJANGO_CONFIGURATION="Staging"
else
DJANGO_CONFIGURATION="Development"
fi

echo "Setting environment variables for $CGAPPNAME_BACKEND"

cf set-env "$CGAPPNAME_BACKEND" ACR_VALUES "$ACR_VALUES"
Expand All @@ -34,15 +45,6 @@ cf set-env "$CGAPPNAME_BACKEND" OIDC_OP_LOGOUT_ENDPOINT "$OIDC_OP_LOGOUT_ENDPOIN
cf set-env "$CGAPPNAME_BACKEND" OIDC_OP_TOKEN_ENDPOINT "$OIDC_OP_TOKEN_ENDPOINT"
cf set-env "$CGAPPNAME_BACKEND" OIDC_RP_CLIENT_ID "$OIDC_RP_CLIENT_ID"
cf set-env "$CGAPPNAME_BACKEND" PRIVATE_KEY "$PRIVATE_KEY"
cf set-env "$CGAPPNAME_BACKEND" AWS_ACCESS_KEY "$AWS_ACCESS_KEY"
cf set-env "$CGAPPNAME_BACKEND" AWS_SECRET_ACCESS_KEY "$AWS_SECRET_ACCESS_KEY"
cf set-env "$CGAPPNAME_BACKEND" AWS_BUCKET "$AWS_BUCKET"
cf set-env "$CGAPPNAME_BACKEND" AWS_REGION_NAME "$AWS_REGION_NAME"
cf set-env "$CGAPPNAME_BACKEND" AWS_ACCESS_KEY_ID "$AWS_ACCESS_KEY_ID"
cf set-env "$CGAPPNAME_BACKEND" BUCKET_NAME "$BUCKET_NAME"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_AWS_ACCESS_KEY_ID "$DJANGO_AWS_ACCESS_KEY_ID"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_AWS_SECRET_ACCESS_KEY "$DJANGO_AWS_SECRET_ACCESS_KEY"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_AWS_STORAGE_BUCKET_NAME "$DJANGO_AWS_STORAGE_BUCKET_NAME"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_SU_NAME "$DJANGO_SU_NAME"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_CONFIGURATION "$DJANGO_CONFIGURATION"
cf set-env "$CGAPPNAME_BACKEND" DJANGO_SETTINGS_MODULE "$DJANGO_SETTINGS_MODULE"
96 changes: 96 additions & 0 deletions scripts/zap-scanner.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
#!/bin/sh


TARGET=$1
ENVIRONMENT=$2

TARGET_DIR="$(pwd)/tdrs-$TARGET"
REPORT_NAME=owasp_report.html
REPORTS_DIR="$TARGET_DIR/reports"

if [ "$ENVIRONMENT" = "nightly" ]; then
APP_URL="https://tdp-$TARGET-staging.app.cloud.gov/"
CONFIG_FILE="zap.conf"
elif [ "$ENVIRONMENT" = "circle" ]; then
CONFIG_FILE="zap.conf"
if [ "$TARGET" = "frontend" ]; then
APP_URL="http://tdp-frontend/"
elif [ "$TARGET" = "backend" ]; then
APP_URL="http://web:8080/"
else
echo "Invalid target $TARGET"
exit 1
fi
elif [ "$ENVIRONMENT" = "local" ]; then

# docker-compose down
# docker-compose up -d --build

if [ "$TARGET" = "frontend" ]; then
APP_URL="http://tdp-frontend/"
elif [ "$TARGET" = "backend" ]; then
APP_URL="http://web/"
else
echo "Invalid target $TARGET"
exit 1
fi
else
echo "Invalid environment $ENVIRONMENT"
exit 1

fi


# do an OWASP ZAP scan
export ZAP_CONFIG=" \
-config globalexcludeurl.url_list.url\(0\).regex='.*/robots\.txt.*' \
-config globalexcludeurl.url_list.url\(0\).description='Exclude robots.txt' \
-config globalexcludeurl.url_list.url\(0\).enabled=true \
-config spider.postform=true"

echo "================== OWASP ZAP tests =================="
cd $TARGET_DIR

if [ "$TARGET" = "frontend" ]; then
docker-compose down
docker-compose up -d --build
fi

# Ensure the reports directory can be written to
chmod 777 $(pwd)/reports

if [ -z ${CONFIG_FILE+x} ]; then
echo "No config file"
docker-compose run zaproxy zap-full-scan.py \
-t $APP_URL \
-m 5 \
-z "${ZAP_CONFIG}" \
-r "$REPORT_NAME" | tee /dev/tty | grep -q "FAIL-NEW: 0"
else
echo "Config file $ENVIRONMENT"
docker-compose run zaproxy zap-full-scan.py \
-t $APP_URL \
-m 5 \
-z "${ZAP_CONFIG}" \
-c "$CONFIG_FILE" \
-r "$REPORT_NAME" | tee /dev/tty | grep -q "FAIL-NEW: 0"
fi


# The `grep -q` piped to the end of the previous command will return a
# 0 exit code if the term is found and 1 otherwise.
ZAPEXIT=$?

if [ "$TARGET" = "frontend" ]; then
docker-compose down --remove-orphan
fi

EXIT=0

if [ "$ZAPEXIT" = 1 ] ; then
echo "OWASP ZAP scan failed"
EXIT=1
fi

exit $EXIT
cd ..
8 changes: 3 additions & 5 deletions tdrs-backend/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ services:
environment:
- SERVICES=s3
- DATA_DIR=/tmp/localstack/data
- AWS_BUCKET
- AWS_REGION_NAME
- AWS_BUCKET=tdp-datafiles-localstack
- AWS_REGION_NAME=us-gov-west-1
ports:
- "4566:4566"
volumes:
Expand All @@ -50,14 +50,12 @@ services:
- DB_HOST=postgres
- DJANGO_SETTINGS_MODULE=tdpservice.settings.local
- DJANGO_CONFIGURATION=Local
- USE_LOCALSTACK=1
- AV_SCAN_URL=http://clamav-rest:9000/scan
- LOCALSTACK_HOST=localstack
- USE_LOCALSTACK
- JWT_KEY
- JWT_CERT_TEST
- DJANGO_SU_NAME
- AWS_BUCKET
- AWS_REGION_NAME
volumes:
- .:/tdpapp
image: tdp
Expand Down
Loading

0 comments on commit 34e77c7

Please sign in to comment.