[Issue 1928] analytics local setup (#3108)

## Summary
Fixes
#{[1928]([1928](#1928))}

### Time to review: __10 mins__

## Changes proposed
Updated documentation to run locally

## Context for reviewers


## Additional information

analytics/Makefile

@@ -6,7 +6,7 @@ ORG ?= HHS
 REPO ?= simpler-grants-gov
 SPRINT_PROJECT ?= 13
 ROADMAP_PROJECT ?= 12
-OUTPUT_DIR ?= data
+OUTPUT_DIR ?= $(shell mkdir -p data && echo data)
 CONFIG_DIR ?= config
 PROJECT_CONFIG_FILE ?= $(CONFIG_DIR)/github-projects.json
 ISSUE_FILE ?= $(OUTPUT_DIR)/delivery-data.json

analytics/src/analytics/integrations/slack.py

@@ -3,9 +3,8 @@
 
 import functools
 import logging
-from collections.abc import Callable
 from dataclasses import dataclass
-from typing import Any
+from typing import Any, Callable
 
 from slack_sdk import WebClient
 from slack_sdk.errors import SlackApiError

analytics/src/analytics/logs/decodelog.py

@@ -11,7 +11,7 @@
 
 import datetime
 import json
-from collections.abc import Mapping
+from typing import Mapping  # noqa:  UP035
 
 RED = "\033[31m"
 GREEN = "\033[32m"

analytics/src/analytics/logs/ecs_background_task.py

@@ -5,9 +5,8 @@
 import os
 import time
 import uuid
-from collections.abc import Callable, Generator
 from functools import wraps
-from typing import ParamSpec, TypeVar
+from typing import Callable, Generator, ParamSpec, TypeVar  # noqa:  UP035
 
 import requests
 

analytics/src/analytics/logs/formatters.py

@@ -10,11 +10,10 @@
 
 import json
 import logging
-from collections.abc import Callable
 from datetime import date, datetime, timezone
 from decimal import Decimal
 from enum import Enum
-from typing import Any, TypeVar
+from typing import Any, Callable, TypeVar
 from uuid import UUID
 
 from analytics.logs import decodelog

documentation/analytics/development.md

@@ -47,21 +47,24 @@ After choosing your approach, following the corresponding setup instructions:
 **Steps**
 
 1. Run `make build`
-2. Set a GitHub Token in your terminal, via `export GH_TOKEN=...`. Acquiring the token is a multi-step process:
-  - Go to https://github.com/settings/tokens
-  - Generate a new token (classic)
-  - Give it the following scopes:
-    - repo
-    - read:org
-    - admin:public_key
-    - project
-  - Add `export GH_TOKEN=...` to your `zshrc` or similar
-3. Set the slackbot token and the channel ID for Slack after following the instructions in [configuring secrets](#configuring-secrets). **Note:** replace the `...` with the value of these secrets:
+2. Acquire a GitHub Token using one of the methods below
+  - Via AWS (Project Team)
+    - Retrieve GH_TOKEN from [AWS](https://us-east-1.console.aws.amazon.com/systems-manager/parameters/%252Fanalytics%252Fgithub-token/description?region=us-east-1&tab=Table#list_parameter_filters=Name:Contains:analytics%2Fgithub-token)
+  - Create your own in GitHub (Open Source)
+    - Go to https://github.com/settings/tokens
+    - Generate a new token (classic)
+    - Give it the following scopes:
+      - repo
+      - read:org
+      - admin:public_key
+      - project
+3. Add `export GH_TOKEN=...` to your `zshrc` or similar
+4. Set the slackbot token and the channel ID for Slack after following the instructions in [configuring secrets](#configuring-secrets). **Note:** replace the `...` with the value of these secrets:
    ```
    export ANALYTICS_SLACK_BOT_TOKEN=...
    export ANALYTICS_REPORTING_CHANNEL_ID=...
    ```
-4. Run `make test-audit` to confirm the application is running correctly.
+5. Run `make test-audit` to confirm the application is running correctly.
 
 #### Running natively
 
@@ -75,21 +78,24 @@ After choosing your approach, following the corresponding setup instructions:
 **Steps**
 
 1. Set up the project: `make setup` -- This will install the required packages and prompt you to authenticate with GitHub
-2. Set a GitHub Token in your terminal, via `export GH_TOKEN=...`. Acquiring the token is a multi-step process:
-  - Go to https://github.com/settings/tokens
-  - Generate a new token (classic)
-  - Give it the following scopes:
-    - repo
-    - read:org
-    - admin:public_key
-    - project
-  - Add `export GH_TOKEN=...` to your `zshrc` or similar
-3. Set the slackbot token and the channel ID for Slack after following the instructions in [configuring secrets](#configuring-secrets). **Note:** replace the `...` with the value of these secrets:
+2. Acquire a GitHub Token using one of the methods below
+  - Via AWS (Project Team)
+    - Retrieve GH_TOKEN from [AWS](https://us-east-1.console.aws.amazon.com/systems-manager/parameters/%252Fanalytics%252Fgithub-token/description?region=us-east-1&tab=Table#list_parameter_filters=Name:Contains:analytics%2Fgithub-token)
+  - Create your own in GitHub (Open Source)
+    - Go to https://github.com/settings/tokens
+    - Generate a new token (classic)
+    - Give it the following scopes:
+      - repo
+      - read:org
+      - admin:public_key
+      - project
+3. Add `export GH_TOKEN=...` to your `zshrc` or similar
+4. Set the slackbot token and the channel ID for Slack after following the instructions in [configuring secrets](#configuring-secrets). **Note:** replace the `...` with the value of these secrets:
    ```
    export ANALYTICS_SLACK_BOT_TOKEN=...
    export ANALYTICS_REPORTING_CHANNEL_ID=...
    ```
-4. Run `make test-audit` to confirm the application is running correctly.
+5. Run `make test-audit` to confirm the application is running correctly.
 
 ### Configuring secrets
 
@@ -104,10 +110,9 @@ If you need to be added to the slack workspace or to the list of collaborators f
 
 #### Finding reporting channel ID
 
-1. Go to the `#z_bot-sprint-reporting` channel in the Simpler.Grants.gov slack workspace.
+1. In the Simpler.Grants.gov Slack workspace navigate to the `#z_bot-sprint-reporting` channel. NB: Use`#z_bot-analytics-ci-test` channel for testing.
 2. Click on the name of the channel in the top left part of the screen.
-3. Scroll down to the bottom of the resulting dialog box until you see where it says `Channel ID`.
-4. Copy and paste that ID into your `.secrets.toml` file under the `reporting_channel_id` variable.
+3. Scroll down to the bottom of the resulting dialog box until you see where it says `Channel ID` and copy.
 
 <img alt="Screenshot of dialog box with channel ID" src="../../analytics/static/screenshot-channel-id.png" height=500>
 
@@ -116,7 +121,7 @@ If you need to be added to the slack workspace or to the list of collaborators f
 1. Go to [the dashboard](https://api.slack.com/apps) that displays the slack apps for which you have collaborator access
 2. Click on `Sprint Reporting Bot` to go to the settings for our analytics slackbot
 3. From the side menu, select `OAuth & Permissions` and scroll down to the "OAuth tokens for your workspace" section
-4. Copy the "Bot user OAuth token" which should start with `xoxb` and paste it into your `.secrets.toml` file under the `slack_bot_token` variable.
+4. Copy the "Bot user OAuth token" which should start with `xoxb`
 
 <img alt="Screenshot of slack app settings page with bot user OAuth token" src="../../analytics/static/screenshot-slackbot-token.png" width=750>
 

documentation/analytics/usage.md

@@ -21,7 +21,7 @@ If you want to have more control over how the reports are run, you can also run
 
 ### Export data and run reports
 
-If want to run reports with the most recent data from GitHub, the easiest way to do it is with the `make sprint-reports-with-latest-data`.
+If want to run reports with the most recent data from GitHub, the easiest way to do it is to run `make sprint-reports-with-latest-data`.
 
 That should result in something like the following being logged to the command line:
 
@@ -76,7 +76,7 @@ After following the installation steps above, you can use the following commands
 
 You can use the following command to import data to the analytics database.
 
-```bash
+```bash 
 poetry run analytics export gh_delivery_data \
 	--config-file config/github-projects.json \
 	--output-file  data/delivery-data.json \
@@ -86,7 +86,7 @@ poetry run analytics export gh_delivery_data \
 or the simpler
 
 ```bash
-make gh-delivery-data-export
+make gh-data-export
 ```
 
 A couple of notes about this command:
@@ -126,12 +126,12 @@ When finished, run `docker-compose down` to stop and remove all containers, netw
 
 #### Viewing the data in Metabase
 
-[Metabase](https://www.metabase.com/) is a buisness intelligence that lets you query, organize and view data with a friendly UX. In order to view data in Metabase, start the containers and then navigate to http://localhost:3100/.
+[Metabase](https://www.metabase.com/) is a business intelligence that lets you query, organize and view data with a friendly UX. In order to view data in Metabase, start the containers via `make build`,  and then navigate to http://localhost:3100/.
 
-The first time you access Metabase you will be guided through a small setup process to sync the database to the platform.
+The first time you access Metabase you will be guided through a small setup process to sync the database to the platform. See local.env to set database properties.
 
 ![Screenshot of the landing page in Metabase](../../analytics/static/screenshot-metabase-page.png)
-
+![Screenshot of the analytics db configuration](../../analytics/static/screenshot-metabase-db-config.png)
 ![Screenshot of a row of test data](../../analytics/static/screenshot-metabase-row-data.png)
 
 ### Calculating metrics
@@ -230,10 +230,10 @@ Development is underway on new as-is/as-was reporting capabilities, the foundati
 
 Initialize the ETL DB:
 ```bash
-poetry run analytics etl initialize_database
+poetry run analytics etl db_migrate
 ```
 
 Transform and load a json file into the ETL DB:
 ```bash
-poetry run analytics etl transform_and_load --deliverable-file ./data/test-etl-01.json --effective-date 2024-10-28
+poetry run analytics etl transform_and_load --issue-file ./data/test-etl-01.json --effective-date 2024-10-28 
 ```