Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performances improvement about parallel running of linters #1789

Merged
merged 10 commits into from
Aug 24, 2022

Conversation

nvuillam
Copy link
Member

Fixes #1785

  • Performances improvements: when running linters in parallel, run in the same process only the linters from same descriptor and that can update the same sources (to avoid concurrency). Other linters can be run independently.

@nvuillam nvuillam force-pushed the features/improve-perfs branch from f16cce2 to 57fbf25 Compare August 23, 2022 12:50
@nvuillam
Copy link
Member Author

nvuillam commented Aug 23, 2022

MegaLinter status: ⚠️ WARNING

Descriptor Linter Files Fixed Errors Elapsed time
✅ BASH bash-exec 7 0 0.02s
✅ BASH shellcheck 7 0 0.25s
✅ BASH shfmt 7 0 0 0.28s
✅ COPYPASTE jscpd yes no 1.88s
✅ DOCKERFILE hadolint 104 0 16.67s
✅ JSON eslint-plugin-jsonc 20 0 0 2.03s
✅ JSON jsonlint 18 0 0.2s
✅ JSON v8r 20 0 11.61s
⚠️ MARKDOWN markdownlint 281 0 5 4.78s
✅ MARKDOWN markdown-link-check 281 0 4.81s
✅ MARKDOWN markdown-table-formatter 281 0 0 6.64s
✅ OPENAPI spectral 1 0 0.78s
⚠️ PYTHON bandit 166 51 1.99s
✅ PYTHON black 166 0 0 3.69s
✅ PYTHON flake8 166 0 1.75s
✅ PYTHON isort 166 0 0 0.56s
✅ PYTHON mypy 166 0 93.22s
✅ PYTHON pylint 166 0 16.5s
✅ REPOSITORY checkov yes no 13.42s
✅ REPOSITORY git_diff yes no 0.27s
✅ REPOSITORY secretlint yes no 14.42s
✅ REPOSITORY trivy yes no 21.69s
✅ SPELL cspell 686 0 20.0s
✅ SPELL misspell 516 0 0 0.78s
✅ XML xmllint 3 0 0.01s
✅ YAML prettier 79 0 0 3.17s
✅ YAML v8r 22 0 27.5s
✅ YAML yamllint 80 0 1.12s

See errors details in artifact MegaLinter reports on CI Job page

  - Define `linter_speed` of linter descriptors (default 3). Can be from 1 (super slow) to 5 (super fast). This is used to optimize the processing order of linters.
@nvuillam nvuillam force-pushed the features/improve-perfs branch from d1edadb to 3b2c141 Compare August 23, 2022 14:35
nvuillam added a commit that referenced this pull request Aug 23, 2022
nvuillam added a commit that referenced this pull request Aug 23, 2022
…documentations (#1793)

* Do not allow FILE_EXTENSIONS and FILE_NAMES_REGEX for lint_all_files linters

Related to #1789

* Build doc

* changelog

* [MegaLinter] Apply linters fixes

Co-authored-by: nvuillam <[email protected]>
@nvuillam
Copy link
Member Author

MegaLinter status: ⚠️ WARNING

Descriptor Linter Files Fixed Errors Elapsed time
✅ BASH bash-exec 7 0 0.01s
✅ BASH shellcheck 7 0 0.29s
✅ BASH shfmt 7 0 0 0.05s
✅ COPYPASTE jscpd yes no 2.51s
✅ DOCKERFILE hadolint 104 0 16.77s
✅ JSON eslint-plugin-jsonc 20 0 0 2.45s
✅ JSON jsonlint 18 0 0.26s
✅ JSON v8r 20 0 10.64s
⚠️ MARKDOWN markdownlint 281 2 5 5.43s
✅ MARKDOWN markdown-link-check 281 0 5.21s
✅ MARKDOWN markdown-table-formatter 281 2 0 7.07s
✅ OPENAPI spectral 1 0 0.98s
⚠️ PYTHON bandit 166 51 2.49s
✅ PYTHON black 166 0 0 3.83s
✅ PYTHON flake8 166 0 2.14s
✅ PYTHON isort 166 0 0 0.42s
✅ PYTHON mypy 166 0 122.59s
✅ PYTHON pylint 166 0 19.14s
✅ REPOSITORY checkov yes no 15.36s
⚠️ REPOSITORY devskim yes 59 1.24s
✅ REPOSITORY dustilock yes no 1.93s
✅ REPOSITORY git_diff yes no 0.03s
✅ REPOSITORY secretlint yes no 4.51s
✅ REPOSITORY syft yes no 2.18s
✅ REPOSITORY trivy yes no 16.33s
✅ SPELL cspell 686 0 24.55s
✅ SPELL misspell 516 2 0 0.49s
✅ XML xmllint 3 0 0.0s
✅ YAML prettier 79 0 0 3.0s
✅ YAML v8r 22 0 29.05s
✅ YAML yamllint 80 0 1.39s

See errors details in artifact MegaLinter reports on CI Job page

You could have the same capabilities but better runtime performances if you use a MegaLinter flavor:

@nvuillam nvuillam merged commit 17db737 into main Aug 24, 2022
@nvuillam nvuillam deleted the features/improve-perfs branch August 24, 2022 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enhance performances by grouping linters only when they can fix sources
1 participant