Retrieve licenses for documentation + fix config file + env issue (#1168

)

* [automation] Auto-update linters version, help and documentation

* Disable Github Status reporter

* Breaking change: GITHUB_STATUS_REPORTER disabled by default

* Breaking change: set PRINT_ALL_FILES default to false

* Deploy V6 image (#1035)

* QuickFix CI job

* New flavor Security (#1036)

* Java PMD v0

* Java Pmd v0.01

* Working Java + pmd :)

* changelog + doc

* Genedate doc + manage offline mode

* changelog

* Bring back referring linters

* Fix build

* Automate update of changelog when new release (v0)

* Do not create commit & tag with npm version

* Move markdown formatting

* .md

* PMD install commands

* More pmd descriptor infos + build

* Fix megalinter errors

* Fix PMD install commands + build

* [MegaLinter] Apply linters fixes

* Hardcode some licenses

* [MegaLinter] Apply linters fixes

* Add gitleaks

* Build with gitleaks

* Gitleaks fixes

* changelog

* New descriptor: repository

TODO: deprecate git and credentials

* New descriptor "repository"

* Add Goodcheck


Goodcheck updates

* Fix PMD

* Fix PMD

* [MegaLinter] Apply linters fixes

* Fix pmd installation

* Typo

* wget --quiet

* Add trivy

* Build v6-alpha flavors

* Remove git & credentials descriptors + upgrade python base image

* Upgrade python

* New flavor Security v0

* Fix ansible-lint install

* Deploy v6-alpha documentation

* Fix test class

quick build

* Disable java pmd

* Downgrade python version

* SARIF v0

* Update descriptor JSON schema + retries for sgerrand url

* Add SARIF output for checkov

* Add SARIF output for eslint

* [automation] Auto-update linters version, help and documentation (#1038)

* Fix gitleaks

* Build

* Do not skip build if failed

* Do not test SARIF when SARIF is not active on a linter

* Update ansible install

* Fix goodcheck tests

* Fix markdown-table-formatter

* Fix trivy command

* Fix checkov test case + manage default sarif output file

* use MegaLinter beta for now

* Update CI for new default report folder

+ Manage {{REPORT_FOLDER}} in CLI variables

* Exit code 1 for trivy

* Fix replace_vars definition

* Do not run test mega-linter-runner when docker build failed

* Fix replace_vars

* Downgrade v8r version

* Fix arm test case

* build

* Fix again arm test case

* Fix makedirs for report folder

* Rename default report folder into megalinter-reports

* Fix gitleaks

* Try again gitleaks

* Fix last errors

* Fix internal lint errors

Co-authored-by: nvuillam <[email protected]>

* v6-alpha for github action

* Adapt CI for v6-alpha. (#1062)

* Fix own MegaLinter errors (#1063)

* Fix own MegaLinter errors

* Do not use secretlint on report folder

* Fix gitleaks own config

* Use regexes in gitleaks toml config file

* grmblmblm

* try with paths

* Regex to gather gitleaks errors

* Make gitleaks non blocking

* Update gitleaks descriptor

* Debug gitleaks (#1068)

* Debug gitleaks

* Remove LinkedIn ID gitleaks check

* Ignore .gitleaks.toml files

* Update security flavor + toml gitleaks update to use regexes

* Try again...

* Fix gitleaks config (use regexes in paths)

* Update default .gitleaks.toml files

* Clean dev variables

* Build

* Retrofit latest v5 updates into v6 (#1070)

* [automation] Auto-update linters version, help and documentation (#1038)

* Implement new linter PHPLint (#1037)

* add new phplint linter (see #1031)

* use composer global install for overture/phplint

* update changelog

* fix feature #1043 (#1044)

* Add @babel/eslint-parser as dependency for eslint (#1045)

* [automation] Auto-update linters version, help and documentation (#1046)

* [automation] Auto-update linters version, help and documentation + @babel-core

* Add @babel-core dependency

* Use node.js LTS and not nodejs-current

* Downgrade v8r version

* Fix arm test case

* Fix again arm test case

* [automation] Auto-update linters version, help and documentation

* Build

* [automation] Auto-update linters version, help and documentation (#1050)

* [automation] Auto-update linters version, help and documentation (#1052)

* [automation] Auto-update linters version, help and documentation

* Update doc for gitlab MegaLinter config

* Bandit default config file (#1051)

* remove default parameters

* add default bandit configfile built with bandit-config-generator command

* update changelog

* update bandit doc page

Co-authored-by: Nicolas Vuillamy <[email protected]>

* Try again ktlint auto-upgrade (#976)

* [automation] Auto-update linters version, help and documentation (#1055)

* [automation] Auto-update linters version, help and documentation (#1056)

* Docs & links (#1058)

* Update stale config

* Fix getting linter version of npm plugin. (#804)

* [automation] Auto-update linters version, help and documentation (#1059)

* Fix doc

* [automation] Auto-update linters version, help and documentation (#1065)

* [automation] Auto-update linters version, help and documentation (#1067)

* build

Co-authored-by: Laurent Laville <[email protected]>

* SARIF for gitleaks (#1069)

* SARIF for gitleaks

* Sarif python test class v0

* Lint fix + changelog

* Generate security flavor

* SARIF support for trivy (#1071)

* [automation] Auto-update linters version, help and documentation (#1038)

* Implement new linter PHPLint (#1037)

* add new phplint linter (see #1031)

* use composer global install for overture/phplint

* update changelog

* fix feature #1043 (#1044)

* Add @babel/eslint-parser as dependency for eslint (#1045)

* [automation] Auto-update linters version, help and documentation (#1046)

* [automation] Auto-update linters version, help and documentation + @babel-core

* Add @babel-core dependency

* Use node.js LTS and not nodejs-current

* Downgrade v8r version

* Fix arm test case

* Fix again arm test case

* [automation] Auto-update linters version, help and documentation

* Build

* [automation] Auto-update linters version, help and documentation (#1050)

* [automation] Auto-update linters version, help and documentation (#1052)

* [automation] Auto-update linters version, help and documentation

* Update doc for gitlab MegaLinter config

* Bandit default config file (#1051)

* remove default parameters

* add default bandit configfile built with bandit-config-generator command

* update changelog

* update bandit doc page

Co-authored-by: Nicolas Vuillamy <[email protected]>

* Try again ktlint auto-upgrade (#976)

* [automation] Auto-update linters version, help and documentation (#1055)

* [automation] Auto-update linters version, help and documentation (#1056)

* Docs & links (#1058)

* Update stale config

* Fix getting linter version of npm plugin. (#804)

* [automation] Auto-update linters version, help and documentation (#1059)

* Fix doc

* [automation] Auto-update linters version, help and documentation (#1065)

* [automation] Auto-update linters version, help and documentation (#1067)

* SARIF for gitleaks

* Sarif python test class v0

* Lint fix + changelog

* SARIF support for trivy

* Accelerate and fix internal tests

* Fix SARIF files

* Fix trivy sarif args

* [automation] Auto-update linters version, help and documentation (#1072)

* Fix SarifReporter

* Better handling of SARIF error management

* Fix SARIF default file name use

quick build

* Fix sarif reporter exception management

* Variabilize default release

* Improve runtime perfs when FLAVOR_SUGGESTIONS: false (#1073)

* Improve runtime perfs when FLAVOR_SUGGESTIONS: false

* [MegaLinter] Apply linters fixes

* Fix how to get MEGALINTER_FLAVOR

* Fix ignore flavor suggestion test class

Co-authored-by: nvuillam <[email protected]>

* Build

Co-authored-by: Laurent Laville <[email protected]>
Co-authored-by: nvuillam <[email protected]>

* V6/popularity - Display github stars in all linters documentation (#1075)

* Add popularity

* Update descriptors to add linter_repo when missing

* FLAVOR_SUGGESTIONS:false to improve perfs

* Build

* Enhance documentation (#1078)

* Popularity column in all tables

* More badges in linters list doc

* cspell

* Fix sarif test case (#1079)

* Fix sarif test case

* Fix hardcoded reporter folder name in test case

* Remove dockerfilelint (#1080)

* Remove dockerfilelint

* changelog

* Upgrade to AWS cfn-lint (#1085)

* Upgrade to AWS cfn-lint

* Build

* Add SARIF management for cfn-lint

* Add cfn-lint example SARIF output file

* SARIF management for Hadolint (#1089)

* SARIF management for Hadolint

* Update .gitleaks.toml to ignore .mypy_cache folder

* Catch SARIF error

* Upgrade hadolint to 2.8.0-alpine

* SARIF for checkstyle (#1093)

* SARIF for ktlint (#1095)

* SARIF for all eslint uses (#1094)

* SARIF management for go revive (#1092)

* SARIF management for go revive

* fix descriptor

quick build

* SARIF Management for PHP PSALM (#1096)

* Build doc

* More SARIF examples

* V6/docker standalone linters (#1099)

* Build unique linters dockerfiles

* Dockerfiles for unique linters + Improve perfs

* CI to build standalone images

* Fix CI

* Matrix jobs: do not fail other jobs if one fails

* Manage docker build --squash

* Add test cases to "only" docker images

* No codecov call when not running all tests

* Create output sarif folder when SARIF is from stdout

* Remove useless python packages for ML runtime

* Remove pytest-cov pytest-timeout from runtime image

* Optimize Until dotenv linter

* gcc in all dockerfiles + go for actionlint

* Add libffi-dev in all Docker images

* Add make

* add bash

* add musl-dev

* Fix linters install

* gnugpg for php

* Automate node & ruby-dev installation if packages are used in install

* Fix dotnet install

* More preset variables on single-linter docker images

* typo

* typo2

* Add ruby-bundler for gem packages

* Remove PHP_BUILTIN as it has been replaced by PHP_PHPLINT

* Fix descriptors and test class

* Improve CI perfs for only-linters build

* More fixes

* Fix kotlin install

* Fix raku install

* Fix R

* Fix scala

* Fix TSX

* Fix Vb .NET

* Build

* Linters page

* rollback raku

* Fix terraform_fmt & rollback raku

* Try fix rakudo

* Fix duplicate ARG and FROM in dockerfile

* Fix python lint err

* Disable lint_all_other_linters_files=true if in SINGLE_LINTER mode

* lint fix

* Reduce docker layers

* Reduce again the list of docker build steps

* Do not keep text logs if not wanted + linter delete SARIF files

* Fix CI

* Fix SARIF test case

* Disable deprecated test case

* Terrascan SARIF management (#1103)

* Terrascan SARIF management

* tflint SARIF management

* Fix terrascan args

* cspell

* MegaLinter server (#1106)

* PHP version switch (#1083)

* fixe issue #1060

* fix hadolint error DL3059

* [automation] Auto-update linters version, help and documentation (#1088)

* [automation] Auto-update linters version, help and documentation

* Increase trivy timeout

* [automation] Auto-update linters version, help and documentation (#1091)

* Release MegaLinter v5.3.0

* [automation] Auto-update linters version, help and documentation (#1097)

* Fix CHANGELOG

* [automation] Auto-update linters version, help and documentation (#1098)

* [automation] Auto-update linters version, help and documentation (#1100)

* [automation] Auto-update linters version, help and documentation (#1101)

* [automation] Auto-update linters version, help and documentation (#1104)

* KEEP_ALIVE_MEGALINTER v0

* openssh

* Build standalone linters only for linters managing SARIF

* Entrypoint

* Entrypoint: exit when test cases called

* Add openrc

* Try another way

https://github.com/danielguerra69/alpine-sshd

* Startup lines at the end

* Test simple case with password

* rhaaaa

* Remove ssh stuff as we loose env variables

* Server v0

* Server v0

* Add workspace argument

* Expose on port 80

* Manage running processes

* try debug flask

* debug mode for server

* v0 ?

* Fix server for report file name

* SARIF in http response

* [automation] Auto-update linters version, help and documentation (#1107)

* [automation] Auto-update linters version, help and documentation (#1109)

* [automation] Auto-update linters version, help and documentation (#1113)

* Bring back ssh

* Set env variables for all users

* [automation] Auto-update linters version, help and documentation (#1114)

* [automation] Auto-update linters version, help and documentation (#1116)

* [automation] Auto-update linters version, help and documentation (#1117)

* new SSH way with tmux

* Build

* Allow PWD auth for testing

* [automation] Auto-update linters version, help and documentation (#1118)

* [automation] Auto-update linters version, help and documentation (#1121)

* Release MegaLinter v5.4.0

* [automation] Auto-update linters version, help and documentation (#1122)

* [automation] Auto-update linters version, help and documentation (#1123)

* [automation] Auto-update linters version, help and documentation (#1124)

* [automation] Auto-update linters version, help and documentation (#1129)

* [clj-kondo] update docs and version to 2021.12.19 (#1126)

Co-authored-by: nvuillam <[email protected]>

* fix "MegaLinter vs Super-Linter" link in README (#1130)

Co-authored-by: Nicolas Vuillamy <[email protected]>

* Move ssh management to entrypoint

* QuickFix

* try again

* Stop docker container if entrypoint fails

* Sh in container

* alias for megalinger when running in server mode (#1133)

makes running megalinter in server mode easier

* aliases

* Build

* tmux_exec

* Build !

* [automation] Auto-update linters version, help and documentation (#1131)

* gha

* motd

* build

* -input and -output v0

* Use argparse

* Use parse_known_args

* config.set

* [automation] Auto-update linters version, help and documentation (#1135)

* DOCKER_DEFAULT_WORKSPACE_DIR & absolue path for --input argument

* introduce shell script to convenience running from ssh (#1137)

* Create megalinter_tmux

* Update build.py

* Build

Co-authored-by: nvuillam <[email protected]>

* Use --input

* Fix --input + CI

* CI for Inbar (create branch named v6_inbar )

* [automation] Auto-update linters version, help and documentation (#1138)

* [automation] Auto-update linters version, help and documentation (#1141)

* [automation] Auto-update linters version, help and documentation (#1142)

* Release MegaLinter v5.5.0

* V6 inbar (#1143)

* capture std is optional

capture std is optional

* improve megalinter_tmux script

add verbosity as an optional flag, improve some messages, add comments, fix script name in examples

* Bash file as executable

* Fix shellcheck issue https://github.com/koalaman/shellcheck/wiki/SC2124

* Python & Dockerfiles lint errors

* cspell

* Build

* Ignore tmux_exec for bash controlling

* pylint err

* cspell

Co-authored-by: Laurent Laville <[email protected]>
Co-authored-by: John Practicalli <[email protected]>
Co-authored-by: Adam Ralph <[email protected]>
Co-authored-by: Inbar <[email protected]>

* V6 semgrep (#1147)

* Add semgrep

* Manage cli_config_default_value

* Manage case when SARIF output is only the last sdout line

* Add python test files for semgrep

* semgrep --error

* use regex to extract json

* Use find / rfind instead of regex

* fix

* fix crash

* reverse does not have a return value

* Fix sarif_default_output_file process

* Check SARIF consistency in test classes

* Fix position to extract JSON from stdout

* position again

* Remove SARIF file from previous runs

* Stupid typo + remove aggregate SARIF if pre-existing ^^

* changelog

* Fix lint errors + descriptor json schema

* Build doc

* Automatically select security rulesets for semgrep (#1148)

* Automatically select security rulesets for semgrep

* Test classes for semgrep security rulesets

* Do not overwrite test class if already existing

* semgrep check failure

* Any v6- and v6_ branch generates standalone linters

* lint fixes

* document additional megalinter semgrep config vars

* build

* SARIF enhancements (and trivy failed attempts) (#1150)

* More general info in sarif

* Fix trivy install for SARIF

* Build

* Force sarif template path + upgrade trivy version

* fix wget call

* try chmod on sarif template

* typo

* trivy

* Add devskim (#1152)

* devskim v0

* devskim bad example

* Fix number of errors count

* fix & Build doc

* changelog

* cspell

* Fix CI

* cspell

* Fix trivy ?

* fix trivy ? (2)

* Try fix gitleaks (#1157)

* Try fix gitleaks

* trivy

* Workaround until next version of trivy

* build

* Revert "Try fix gitleaks"

This reverts commit e7c1be6.

* Remove default gitleaks use + --no-git

* --no-git back

* SARIF for npm-groovy-lint (#1158)

* SARIF for npm-groovy-lint

* Fix GroovyLinter custom class

* Factorize

* Manage dynamic --no-git presence for gitleaks (#1162)

* Manage dynamic --no-git presence for gitleaks

* lint fix

* Test upload SARIF to github

* Retrieve licenses

* Fix mixed config file + env variables config

* Link to licenses

* Update links to licenses

* Fixes

* fix lint err

* [MegaLinter] Apply linters fixes

Co-authored-by: nvuillam <[email protected]>
Co-authored-by: Laurent Laville <[email protected]>
Co-authored-by: John Practicalli <[email protected]>
Co-authored-by: Adam Ralph <[email protected]>
Co-authored-by: Inbar <[email protected]>

.automation/build.py

@@ -2004,6 +2004,7 @@ def generate_documentation_all_linters():
             linter_licenses = json.load(json_file)
             license = ""
             md_license = "<!-- -->"
+            linter_license_md_file = None
             # get license from github api
             if (
                 hasattr(linter, "linter_repo")
@@ -2020,19 +2021,48 @@ def generate_documentation_all_linters():
                 session = requests_retry_session()
                 r = session.get(api_github_url, headers=api_github_headers)
                 if r is not None:
+                    # Update license key for licenses file
                     resp = r.json()
-                    if "license" in resp and "spdx_id" in resp["license"]:
-                        license = (
-                            resp["license"]["spdx_id"]
-                            if resp["license"]["spdx_id"] != "NOASSERTION"
-                            else resp["license"]["name"]
-                            if "name" in resp["license"]
-                            else resp["license"]["key"]
-                            if "key" in resp["license"]
-                            else ""
+                    if resp is not None and not isinstance(resp, type(None)):
+                        if "license" in resp and "spdx_id" in resp["license"]:
+                            license = (
+                                resp["license"]["spdx_id"]
+                                if resp["license"]["spdx_id"] != "NOASSERTION"
+                                else resp["license"]["name"]
+                                if "name" in resp["license"]
+                                else resp["license"]["key"]
+                                if "key" in resp["license"]
+                                else ""
+                            )
+                            if license != "":
+                                linter_licenses[linter.linter_name] = license
+                    # Fetch and update license file if not in repo
+                    linter_license_md_file = (
+                        f"{REPO_HOME}/docs/licenses/{linter.linter_name}.md"
+                    )
+                    if not os.path.isfile(linter_license_md_file):
+                        api_github_license_url = api_github_url + "/license"
+                        r_license = session.get(
+                            api_github_license_url, headers=api_github_headers
                         )
-                        if license != "":
-                            linter_licenses[linter.linter_name] = license
+                        if r_license is not None:
+                            resp_license = r_license.json()
+                            if "download_url" in resp_license:
+                                license_downloaded = session.get(
+                                    resp_license["download_url"]
+                                )
+                                with open(
+                                    linter_license_md_file, "w", encoding="utf-8"
+                                ) as license_out_file:
+                                    license_out_file.write(license_downloaded.text)
+                                    logging.info(
+                                        f"Copied license of {linter.linter_name} in {linter_license_md_file}"
+                                    )
+                            else:
+                                logging.warning(
+                                    f"WARNING: No download_url returned in {api_github_license_url}"
+                                )
+
             # get license from descriptor
             if (
                 license == ""
@@ -2045,7 +2075,11 @@ def generate_documentation_all_linters():
                 license = linter_licenses[linter.linter_name]
             # build md_license
             if license != "":
-                md_license = license
+                if linter_license_md_file is not None:
+                    license_doc_url = f"licenses/{linter.linter_name}.md"
+                    md_license = f"[{license}]({license_doc_url})"
+                else:
+                    md_license = license
         # Update licenses file
         with open(LICENSES_FILE, "w", encoding="utf-8") as outfile:
             json.dump(linter_licenses, outfile, indent=4, sort_keys=True)

.cspell.json

@@ -5,6 +5,7 @@
         "**/.automation/generated/**",
         "**/.git/**",
         "**/package-lock.json",
+        "**/licenses/*.md",
         "lib",
         "report",
         ".vscode"

.jscpd.json

@@ -25,6 +25,7 @@
     "**/megalinter/tests/test_megalinter/helpers/utilstest.py",
     "**/*.test.js",
     "**/CHANGELOG.md",
-    "**/mega-linter-runner/README.md"
+    "**/mega-linter-runner/README.md",
+    "**/licenses/*.md"
   ]
 }

CHANGELOG.md

@@ -8,6 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 Note: Can be used with `megalinter/megalinter@beta` in your GitHub Action mega-linter.yml file, or with `megalinter/megalinter:beta` docker image
 
+- Add linters licenses to online documentation
+- Fix issue when config vars are both from ENV and from config file ([#1154](https://github.com/megalinter/megalinter/issues/1154))
+
 - Linter versions upgrades
   - [coffeelint](http://www.coffeelint.org) from 5.2.2 to **5.2.3** on 2022-01-09
   - [phpstan](https://phpstan.org/) from 1.3.0 to **1.3.3** on 2022-01-09

build.sh

@@ -12,7 +12,9 @@ fi
 # Prettify markdown tables
 echo "Formatting markdown tables..."
 # shellcheck disable=SC2086
-MD_FILES=$(find . -type f -name "*.md" -not -path "*/node_modules/*" -not -path "*/.automation/*") && npx markdown-table-formatter $MD_FILES
+MD_FILES=$(find . -type f -name "*.md" -not -path "*/node_modules/*" -not -path "*/.automation/*" -not -path "*/licenses/*") && npx markdown-table-formatter $MD_FILES
+# shellcheck disable=SC2086
+MD_FILES_2=$(find . -type f -name "*.md" -not -path "*/node_modules/*" -not -path "*/.automation/*" -path "*/licenses/*") && npx markdown-table-formatter $MD_FILES_2
 
 # Build online documentation
 if type python3 >/dev/null 2>/dev/null; then