Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Requirements to pipfile #1497

Merged
merged 58 commits into from
Aug 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
e2f17ff
added pip install to tox-pipenv
tomneeman151293 Jul 21, 2021
83cd744
Merge branch 'master' into Requirements-To-Pipfiles
tomneeman151293 Jul 25, 2021
cdb94c5
added pip install of pip requirements
tomneeman151293 Jul 25, 2021
bd16f9d
added pipfile, pipfile lock, removed requirement files
tomneeman151293 Jul 25, 2021
f70c24a
added command to create requirements file from pipfile lock
tomneeman151293 Jul 25, 2021
37f74f0
changed order of creating requirements file
tomneeman151293 Jul 25, 2021
0f0cde2
commented cache saving
tomneeman151293 Jul 25, 2021
bfba94f
wrapped setup.py with main func
tomneeman151293 Jul 25, 2021
6e4c011
checks
tomneeman151293 Jul 25, 2021
1dc253b
checks
tomneeman151293 Jul 25, 2021
e6b5503
checks
tomneeman151293 Jul 25, 2021
4d5d941
checks
tomneeman151293 Jul 25, 2021
804fa26
added pytest to dependecies
tomneeman151293 Jul 25, 2021
2e7301c
show me what you got
Jul 25, 2021
19fc2ce
Merge branch 'master' into Requirements-To-Pipfiles
tomneeman151293 Aug 2, 2021
94e8109
changes
tomneeman151293 Aug 2, 2021
dd5dbfa
regenerated pip file
tomneeman151293 Aug 2, 2021
f10a5af
tried to add commands to tox ini
tomneeman151293 Aug 2, 2021
27c6e00
checks for hard coded dev req in build config
tomneeman151293 Aug 2, 2021
dff9a8e
checks for hard coded dev req in build config
tomneeman151293 Aug 2, 2021
9c289d4
checks for hard coded dev req in build config
tomneeman151293 Aug 2, 2021
9ab6a0f
fixed yml structure
tomneeman151293 Aug 2, 2021
91792ad
fixed yml structure 2
tomneeman151293 Aug 2, 2021
332a68a
yml checks
tomneeman151293 Aug 2, 2021
b5abbe9
yml checks
tomneeman151293 Aug 2, 2021
df1e9a1
yml checks
tomneeman151293 Aug 2, 2021
53030bb
yml checks
tomneeman151293 Aug 2, 2021
ea35226
yml checks
tomneeman151293 Aug 2, 2021
ffbc47d
regenerated pip file
tomneeman151293 Aug 2, 2021
59ed913
regenerated pip file
tomneeman151293 Aug 2, 2021
23d4e57
fixed piplock
tomneeman151293 Aug 2, 2021
6d40af0
deleted tox pipenv installations
tomneeman151293 Aug 2, 2021
0bcdaf2
added copy of requirements to artifacts
tomneeman151293 Aug 2, 2021
4aa85eb
wip
tomneeman151293 Aug 2, 2021
710b61c
reverted requirements addition
tomneeman151293 Aug 2, 2021
68a5d55
small check
tomneeman151293 Aug 3, 2021
26d2b4d
reverted setup.py check
tomneeman151293 Aug 3, 2021
2bd9df1
cr fixes
tomneeman151293 Aug 3, 2021
b98c5bd
yml structure fix
tomneeman151293 Aug 3, 2021
3c076f7
yml structure fix
tomneeman151293 Aug 3, 2021
3b81eeb
fix build check
tomneeman151293 Aug 3, 2021
ef8da71
fix build check
tomneeman151293 Aug 3, 2021
f99148e
fix build check
tomneeman151293 Aug 3, 2021
307c674
precommit-checks
tomneeman151293 Aug 3, 2021
e4530da
re-added install_requires
tomneeman151293 Aug 3, 2021
4b44324
docs changes
tomneeman151293 Aug 3, 2021
dad79c4
Merge branch 'master' into Requirements-To-Pipfiles
tomneeman151293 Aug 8, 2021
3000151
updated urllib3 version
tomneeman151293 Aug 8, 2021
ddc24e5
changed >= to *
tomneeman151293 Aug 8, 2021
faee29a
updated packages
tomneeman151293 Aug 8, 2021
05fd423
check
tomneeman151293 Aug 8, 2021
d28a229
check
tomneeman151293 Aug 8, 2021
4cb0854
check
tomneeman151293 Aug 8, 2021
f1cfa93
check
tomneeman151293 Aug 8, 2021
ee1448d
check
tomneeman151293 Aug 8, 2021
d075825
fixed problem using pipfile2req in setup.py
tomneeman151293 Aug 8, 2021
ba7d169
Merge branch 'master' into Requirements-To-Pipfiles
tomneeman151293 Aug 8, 2021
2189811
precommit fixes
tomneeman151293 Aug 8, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 26 additions & 13 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,15 +18,33 @@ references:
- checkout-content
- prepare-tox3-7

create_requirement_files: &create_requirement_files
run:
name: Create requirement files
command: |
pip install pipfile-requirements
pipfile2req > requirements.txt
pipfile2req -d > requirements-dev.txt

restore_tox_37_cache: &restore_tox_37_cache
restore_cache:
key: tox-37-cache-{{ checksum "Pipfile" }}-{{ checksum "Pipfile.lock" }}

save_tox_37_cache: &save_tox_37_cache
save_cache:
key: tox-37-cache-{{ checksum "Pipfile" }}-{{ checksum "Pipfile.lock" }}
paths:
- .tox/py37

jobs:
tox3-7-unit-tests:
# docker image circleci/python:3.8.3-buster-node also contains python 3.7
docker:
- image: circleci/python:3.8.3-buster-node
steps:
- checkout
- restore_cache:
key: tox-37-cache-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}
- *create_requirement_files
- *restore_tox_37_cache
- run:
# install npm modules so readme tests run with mdx verification
name: npm install
Expand All @@ -41,18 +59,16 @@ jobs:
no_output_timeout: 15m
command: |
tox -e py37 -v
- save_cache:
key: tox-37-cache-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}
paths:
- .tox/py37
- *save_tox_37_cache

tox3-8-unit-tests:
docker:
- image: circleci/python:3.8.3-buster-node
steps:
- checkout
- *create_requirement_files
- restore_cache:
key: tox-38-cache-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}
key: tox-38-cache-{{ checksum "Pipfile" }}-{{ checksum "Pipfile.lock" }}
- run:
# install npm modules so readme tests run with mdx verification
name: npm install
Expand Down Expand Up @@ -84,8 +100,8 @@ jobs:
- image: circleci/python:3.8.3-buster-node
steps:
- checkout
- restore_cache:
key: tox-37-cache-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}
- *create_requirement_files
- *restore_tox_37_cache
- run:
# install npm modules so readme tests run with mdx verification
name: npm install
Expand All @@ -99,10 +115,7 @@ jobs:
name: Tox build
command: |
tox -e py37 --notest
- save_cache:
key: tox-37-cache-{{ checksum "requirements.txt" }}-{{ checksum "requirements-dev.txt" }}
paths:
- .tox/py37
- *save_tox_37_cache
- persist_to_workspace:
root: ~/project
paths:
Expand Down
32 changes: 16 additions & 16 deletions CONTRIBUTION.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ We build `demisto-sdk` to support python 3.7 and 3.8.

## Getting started

1. [Clone demisto-sdk repository](#1-Clone-demisto-sdk-repository)
2. [Install demisto sdk dev environment](#2-Install-demisto-sdk-dev-environment)
3. [Pre-commit hooks setup](#3-Pre-commit-hooks-setup)
4. [DemistoContentPython Libary](#4-DemistoContentPython-Libary)
5. [Develop new command](#5-Develop-new-command)
1. [Clone demisto-sdk repository](#1-clone-demisto-sdk-repository)
2. [Install demisto sdk dev environment](#2-install-demisto-sdk-dev-environment)
3. [Pre-commit hooks setup](#3-pre-commit-hooks-setup)
4. [DemistoContentPython Library](#4-demistocontentpython-library)
5. [Develop new command](#5-develop-new-command)
6. [Running unit-tests using tox](#6-running-unit-tests-using-tox)
7. [Push changes to GitHub (External PRs)](#7-push-changes-to-github-relevant-only-for-external-prs)
8. [Review Process](#8-review-process)
Expand All @@ -33,33 +33,33 @@ This will be used as your testing environment, you do not need to update it agai

1. Make sure you have python3 installed.

2. Add executable permissions to our setup script by running `chmod a+x ~/dev/demisto/demisto-sdk/demisto-sdk-development.sh`.
2. Make sure you have pipenv installed.

3. Run the script: `~/dev/demisto/demisto-sdk/demisto-sdk-development.sh`.
* You might need to setup your SHELL env variable by running `export SHELL=/bin/zsh` or `export SHELL=/bin/bash`.
3. run `pipenv install --dev`

4. Restart your terminal.
4. To install local version of demisto-sdk, run `pipenv install --keep-outdated -e <path to your demisto-sdk repo>`

You have now setup the your `demisto-sdk-dev` env!
5. For further reading about pipenv, you can refer to [pipenv documentation](https://pipenv.pypa.io/en/latest/).

To activate it simply run: `workon demisto-sdk-dev`
You have now setup the your `demisto-sdk` dev environment!

To activate it simply run: `pipenv shell`.
* Check that the demisto-sdk installed is your local version by running `demisto-sdk -v` - you will should see something similar to `demisto-sdk 1.X.X.dev`.
If not, while your current working directory is the `demisto-sdk` root repo by runnning `cd ~/dev/demisto/demisto-sdk` and then run `pip3 install -e .`.

To deactivate the virtual environment and return simply run: `deactivate`.
To deactivate the virtual environment and return simply run: `exit`.
* Note that your local `demisto-sdk` version should remain unchanged.

---

### 3. Pre-commit hooks setup
We use are using [pre-commit](https://pre-commit.com/) to run hooks on our build. To use it run:
1. Install hook to be performed as a hook before commiting changes - `pre-commit install`
1. Install hook to be performed as a hook before commit changes - `pre-commit install`
2. Enable auto update of pre-commit hooks - `pre-commit autoupdate`
3. In order to run pre-commit without commiting - `pre-commit run -a` (on all files), `pre-commit run` (on staged files)
3. In order to run pre-commit without commit- `pre-commit run -a` (on all files), `pre-commit run` (on staged files)

---

### 4. DemistoContentPython Libary
### 4. DemistoContentPython Library
ContentPython is a python library used to interact with Demisto Content repository, high-level abstraction.
For more information read the following [guide](demisto_sdk/commands/common/content/README.md).

Expand Down
69 changes: 69 additions & 0 deletions Pipfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
astroid = ">=2.4.2,<2.5.4"
autopep8 = "*"
bandit = "*"
bs4 = "*"
chardet = "==3.0.2"
click = "==7.1.2"
colorama = "*"
coloredlogs = "*"
configparser = "*"
coverage = "*"
dateparser = "*"
decorator = "*"
demisto-py = "*"
dictor = "*"
docker = "==4.3.0"
flake8 = "*"
flatten-dict = "*"
gitdb = "*"
google-cloud-storage = "==1.29.0"
inflection = "==0.5.1"
isort = "*"
jsonschema = "*"
mergedeep = "*"
mypy = "==0.782"
networkx = "*"
nltk = "*"
pandas = "*"
pipenv = "*"
prettytable = "==0.7.2"
pykwalify = "*"
pylint = "==2.6.0"
pyspellchecker = "*"
requests = "*"
"ruamel.yaml" = "<0.17"
slackclient = "==2.9.3"
tabulate = "*"
ujson = "*"
urllib3 = "*"
vulture = "*"
wcmatch = "*"
yamlordereddictloader = "*"
GitPython = "*"
Jinja2 = "<3.0,>=2.11.1"
Pebble = "*"
PyPDF2 = "*"
PyYAML = "*"

[dev-packages]
mitmproxy = "==5.1.0"
mock = "*"
pre-commit = "*"
pytest = "*"
pytest-cov = "*"
pytest-datadir-ng = "*"
pytest-mock = "*"
pytest-subprocess = "*"
requests-mock = "*"
tox-pyenv = "*"
Deprecated = "*"
pipfile-requirements = "*"

[requires]
python_version = "3.9"
Loading