Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 0.1 merge #9

Merged
merged 63 commits into from
Jul 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
3f0b804
Add gitignore
Bear-Witness-98 Jul 16, 2024
1b3d8c3
Add packaging files
Bear-Witness-98 Jul 16, 2024
0e731ff
Add formatting to python script files
Bear-Witness-98 Jul 16, 2024
1e56b65
Fixes to make notebook work
Bear-Witness-98 Jul 16, 2024
89fadcc
Merge pull request #1 from Bear-Witness-98/feature/repo-setup
Bear-Witness-98 Jul 16, 2024
09fde77
Notebook review, challange updated.
Bear-Witness-98 Jul 16, 2024
9a5d872
Merge pull request #2 from Bear-Witness-98/feature/notebook-revision
Bear-Witness-98 Jul 16, 2024
80719b8
Fix data loading directory for test
Bear-Witness-98 Jul 16, 2024
06b9145
update gitignore
Bear-Witness-98 Jul 16, 2024
0285c28
fix prediction test
Bear-Witness-98 Jul 16, 2024
7540ec3
Add comment to challange.md
Bear-Witness-98 Jul 16, 2024
3e2ab2b
Passed models tests
Bear-Witness-98 Jul 16, 2024
eae1e08
Add poetry lock file
Bear-Witness-98 Jul 17, 2024
bbefabf
Merge pull request #3 from Bear-Witness-98/feature/model-addition
Bear-Witness-98 Jul 17, 2024
e06d3dd
Fix model bugs
Bear-Witness-98 Jul 17, 2024
8e5831e
Additional model improvements
Bear-Witness-98 Jul 17, 2024
f74012a
Remove ipdb from model training.
Bear-Witness-98 Jul 17, 2024
1030988
Merge pull request #4 from Bear-Witness-98/bugfix/model-implementation
Bear-Witness-98 Jul 17, 2024
0eb6226
Update fastapi version
Bear-Witness-98 Jul 17, 2024
d04e150
Add api
Bear-Witness-98 Jul 17, 2024
c5fcede
Update gitignore
Bear-Witness-98 Jul 17, 2024
6ac63e3
Merge pull request #5 from Bear-Witness-98/feature/api
Bear-Witness-98 Jul 17, 2024
e4536d9
Removed unused imports
Bear-Witness-98 Jul 17, 2024
2bb3174
Update model storing directory
Bear-Witness-98 Jul 17, 2024
a90e15b
Chande model loading
Bear-Witness-98 Jul 17, 2024
19bc0a4
Change directory for storing the model
Bear-Witness-98 Jul 17, 2024
d5735db
Merge pull request #6 from Bear-Witness-98/bugfix/api-model-train
Bear-Witness-98 Jul 17, 2024
b4d1ed5
Update poetry with appropriate versions
Bear-Witness-98 Jul 17, 2024
ebaaace
Add dockerfile
Bear-Witness-98 Jul 17, 2024
c83e428
Update dockerfile command
Bear-Witness-98 Jul 17, 2024
ab3fe51
Add deploy script, and docker fix
Bear-Witness-98 Jul 17, 2024
1ceb3c6
update library version for stress test
Bear-Witness-98 Jul 17, 2024
6c922fa
update stress tests url
Bear-Witness-98 Jul 17, 2024
6200cb6
Update doc
Bear-Witness-98 Jul 17, 2024
3952da4
Add copy of model to dockerfile
Bear-Witness-98 Jul 17, 2024
adb8029
Update dockerfile and poetry lock
Bear-Witness-98 Jul 17, 2024
54ef2fc
Add change to dockerfile
Bear-Witness-98 Jul 17, 2024
db63318
Add training to deploy script
Bear-Witness-98 Jul 17, 2024
1816d7c
Merge pull request #7 from Bear-Witness-98/feature/deploy-api
Bear-Witness-98 Jul 17, 2024
c27b1fe
Moved ci/cd files to appropriate directory.
Bear-Witness-98 Jul 19, 2024
0afcff8
First ci update
Bear-Witness-98 Jul 19, 2024
699de0a
Add paths to testing
Bear-Witness-98 Jul 19, 2024
debb054
Add .github folder to trigger
Bear-Witness-98 Jul 19, 2024
c7ca0ff
add '' to paths
Bear-Witness-98 Jul 19, 2024
f181f04
Add commands to ci
Bear-Witness-98 Jul 19, 2024
7d9ea2a
Fix syntax error
Bear-Witness-98 Jul 19, 2024
e1522e6
Add module files to needed paths
Bear-Witness-98 Jul 19, 2024
de5f770
setup python version
Bear-Witness-98 Jul 19, 2024
8d3e9d0
Check directory
Bear-Witness-98 Jul 19, 2024
5ac5991
Checkout code
Bear-Witness-98 Jul 19, 2024
ce920a4
Fix installation
Bear-Witness-98 Jul 19, 2024
652eddb
add virtualenv folder to executing machine
Bear-Witness-98 Jul 19, 2024
b35f987
Add test dependencies to poetry
Bear-Witness-98 Jul 19, 2024
ee5c8fc
Modified dependencies installation
Bear-Witness-98 Jul 19, 2024
a33bdea
Add model retrival
Bear-Witness-98 Jul 19, 2024
cac268e
Add api test
Bear-Witness-98 Jul 19, 2024
a892d27
rename job
Bear-Witness-98 Jul 19, 2024
72b59d9
Add cd workflow
Bear-Witness-98 Jul 19, 2024
4328d1d
Fix syntax error
Bear-Witness-98 Jul 19, 2024
3ce6e1b
Check secret upload
Bear-Witness-98 Jul 19, 2024
c4574f2
Add gcp authentication
Bear-Witness-98 Jul 19, 2024
79df92c
Add github actions workflows to PRs on main.
Bear-Witness-98 Jul 19, 2024
068627d
Merge pull request #8 from Bear-Witness-98/feature/ci-cd
Bear-Witness-98 Jul 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[flake8]
# select all
select = A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z
max-line-length = 88
inline-quotes = double
exclude = .venv, .git, .eggs, __pycache__, build, dist, __init__.py, .ipynb, .ipynb_checkpoints
ignore =
# disable, whitespace before ':'
E203,
# disable "too many leading '#' for block comment"
E266,
# disable "line break before binary operator" which clashes with black
W503,
# disable "invalid escape sequence"
W605,
48 changes: 48 additions & 0 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: 'Continuous Delivery'

on:
pull_request:
branches:
- develop
- main
paths:
- 'challenge/**'
- 'scripts/**'
- '.github/**'
- 'pyproject.toml'
- 'poetry.lock'

jobs:
run_testing:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Google Cloud SDK
uses: google-github-actions/auth@v1
with:
credentials_json: ${{ secrets.GCP_CREDENTIAL }}
- name: Check directory
run: |
ls
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10.8'
- name: Python version
run: |
python --version
- name: Install poetry and virtualenv
run: |
pip install poetry
- name: Install dependencies
run: |
poetry config virtualenvs.create false
poetry lock --no-update
poetry install
- name: Push to prod
run: |
bash scripts/deploy.sh
- name: Run stress test
run: |
make stress-test
46 changes: 46 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: 'Continuous Integration'

on:
pull_request:
branches:
- develop
- main
paths:
- 'challenge/**'
- 'scripts/**'
- '.github/**'
- 'pyproject.toml'
- 'poetry.lock'

jobs:
run_testing:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check directory
run: |
ls
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.10.8'
- name: Python version
run: |
python --version
- name: Install poetry and virtualenv
run: |
pip install poetry
- name: Install dependencies
run: |
poetry config virtualenvs.create false
poetry lock --no-update
poetry install
- name: Get model
run:
python challenge/model.py
- name: Run model test
run: |
make model-test
- name: Run api test
run: |
make api-test
165 changes: 165 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,165 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
.pybuilder/
target/

# Jupyter Notebook
.ipynb_checkpoints

# IPython
profile_default/
ipython_config.py

# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version

# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock

# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock

# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/

# Celery stuff
celerybeat-schedule
celerybeat.pid

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/
.dmypy.json
dmypy.json

# Pyre type checker
.pyre/

# pytype static type analyzer
.pytype/

# Cython debug symbols
cython_debug/

# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/

reports/*
*.pkl
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.10.8
40 changes: 40 additions & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
// Set correct python path to venv's one
"python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
// Hide .venv from explorer and searchbar
"files.watcherExclude": {
"**/.venv/**": true
},
"files.exclude": {
"**/.venv/": true
},
"search.exclude": {
"**/.venv/": true
},
// Linting and formatting
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": "explicit"
},
// auto formatter
"black-formatter.importStrategy": "fromEnvironment",
// import formatter
"isort.importStrategy": "fromEnvironment",
"isort.args": [
"--settings-path",
"${workspaceFolder}/pyproject.toml"
],
// linting
"flake8.importStrategy": "fromEnvironment",
"flake8.args": [
"--config=${workspaceFolder}/.flake8"
],
"editor.wordWrapColumn": 100,
"files.trimFinalNewlines": true,
"files.trimTrailingWhitespace": true,
//
// Jupyter
//
"jupyter.notebookFileRoot": "${workspaceFolder}",
"jupyter.interactiveWindow.textEditor.executeSelection": true,
}
23 changes: 21 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
# syntax=docker/dockerfile:1.2
FROM python:latest
# put you docker configuration here
FROM python:3.10.11
# put you docker configuration here

WORKDIR /module
RUN apt update && apt install -y build-essential
RUN pip install poetry

# copy all stuff into container
COPY challenge/ ./challenge
COPY pyproject.toml/ .
COPY poetry.lock .
COPY models/ ./models

# add empty README and install
RUN touch README.md
RUN poetry config virtualenvs.create false
RUN poetry lock --no-update
RUN poetry install --only main


CMD ["python", "-m", "uvicorn", "challenge.api:app", "--host", "0.0.0.0", "--port", "8080"]
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,10 @@ install: ## Install dependencies
pip install -r requirements-test.txt
pip install -r requirements.txt

STRESS_URL = http://127.0.0.1:8000
STRESS_URL = https://delay-model-api-qzo5moezqa-uw.a.run.app
.PHONY: stress-test
stress-test:
# change stress url to your deployed app
# change stress url to your deployed app
mkdir reports || true
locust -f tests/stress/api_stress.py --print-stats --html reports/stress-test.html --run-time 60s --headless --users 100 --spawn-rate 1 -H $(STRESS_URL)

Expand Down
Loading
Loading