Merge branch 'v4.0/dev' into update-year-readme

.changes-pending.md

@@ -1,31 +0,0 @@
-* chore: parse changelog PR author names from contributors (Max Leske) [#3408]
-* fix: Added missing target name to logdata (932260 PL1, 932240 PL2) (Ervin Hegedus) [#3409]
-* fix: remove overly specific rule with limited benefits and lack of cross-engine compatibility (934131 PL2) (Andrea Menin) [#3378]
-* fix: remove base64 transformation due to limited effectiveness and to align behavior across ModSecurity v2.x and libModSecurity v3.x engines (934130 PL1) (Andrea Menin) [#3378]
-* chore: improve changelog-pr workflow (Max Leske) [#3416]
-* fix: solve false positive by shifting ``Field cannot be empty`` to PL2 (953100 PL1, 953101 PL2) (Esad Cetiner) [#3407]
-* feat: added new webshells and tests (955100 PL1) (Jozef Sudolský) [#3405]
-* fix: multiple fixes when generating changelog PR (Max Leske) [#3418], [#3420], [#3422], [#3424] [#3429]
-* fix: reword comment (900300 config) (Christian Folini) [#3417]
-* feat: improve detection by adding missing javascript `prompt` and `confirm` methods (941390 PL1) (Jitendra Patro) [#3395]
-* fix: handle false positive by fixing whitespace matching after PHP command (933160 PL1) (Max Leske) [#3432]
-* docs: add link to run tests (Ervin Hegedus) [#3438]
-* fix: improve rule by matching non-word-boundary of commands with options (932237 PL3) (Max Leske) [#3425]
-* fix: replace backend docker container for tests to fix JSON Unicode reflection (Max Leske) [#3464]
-* feat: add .vscode to restricted-files.data (930130 PL1) (Frederik Himpe) [#3471]
-* feat: remove redundant t:lowercase for a little performance (922110 PL1) (Jozef Sudolský) [#3469]
-* fix: update comments (922110 PL1, 942440 PL2) (Jozef Sudolský) [#3468]
-* feat: add unix commands pyversions and py3versions (932235 PL1, 932260 PL1, 932236 PL2, 932237 PL3, 932239 PL2) (Jitendra Patro) [#3465]
-* feat: add new test method: check for tags on rules against allowlist (Ervin Hegedus) [#3437]
-* fix: prevent unintended match of character set substrings in multipart/form-data requests (922100 PL1) (Jozef Sudolský) [#3470]
-* fix: handle false positives with word "settings" (932236 PL2, 932237 PL3, 932239 PL2) (Esad Cetiner) [#3394]
-* fix: prevent false positives against brackets in User-Agent header (932131 PL2) (Max Leske) [#3486]
-* fix: prevent FPs against names due to "axel" and "perl" (932235 PL1, 932260 PL1, 932236 PL2, 932239 PL2, 932237 PL3) (@superlgn) [#3492]
-* feat: detect User-Agent of Tsunami Security Scanner (913100 PL1) (@hoexter) [#3480]
-* fix: correct numerical values used for HTML entity evasion detection (941220 PL1) (Jitendra Patro) [#3479]
-* fix: prevent FP on keywords more and time in Unix RCE (932236 PL2) (Franziska Bühler) [#3487]
-* feat: detect 'dialog' tag in XSS no-script payloads (941160 PL1) (Jitendra Patro) [#3473]
-* fix: add urlDecodeUni transformation rules with REQUEST_URI / REQUEST_BASENAME in phase 1 (921240 PL1, 920440 PL1, 920201 PL2, 920202 PL4) (Christian Folini) [#3411]
-* feat: add BlockCypher.log to restricted-files.data (930130 PL1) (Jozef Sudolský) [#3501]
-* fix: avoid FPs in RCE detections against words 'environment' and 'performance' (932230 PL1, 932235 PL1, 932260 PL1, 932236 PL2, 932237 PL3, 932239 PL2) (Esad Cetiner) [#3477]
-* fix: prevent FP on keywords 'more' and 'time' in Unix RCE (932235 PL1) (Franziska Bühler) [#3488]

.github/create-changelog-prs.py

@@ -106,7 +106,7 @@ def generate_content(prs: list, merged_by: str) -> (str, str):
 		pr_number = pr["number"]
 		pr_title = pr["title"]
 		pr_author = get_pr_author_name(pr["author"]["login"])
-		new_line = f"* {pr_title} ({pr_author}) [#{pr_number}]\n"
+		new_line = f" * {pr_title} ({pr_author}) [#{pr_number}]\n"
 		pr_body += new_line
 		pr_links += f"- #{pr_number}\n"
 

.github/workflows/test.yml

@@ -26,7 +26,7 @@ jobs:
       - name: "Install dependencies"
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          GO_FTW_VERSION: '0.6.3'
+          GO_FTW_VERSION: '0.6.4'
         run: |
           gh release download -R coreruleset/go-ftw v${GO_FTW_VERSION} -p "ftw_${GO_FTW_VERSION}_linux_amd64.tar.gz" -O - | tar -xzvf - ftw
 

regex-assembly/932236.ra

@@ -4,18 +4,12 @@
 ##!+ i
 
 ##!> assemble
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix-start-of-string
-  ##!<
-
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix
-  ##!<
+  ##!> include unix-shell-evasion-prefix-start-of-string.ra
 ##!<
 ##!=>
 
 ##! These patterns are approximations of the patterns used by the cmdline
 ##! processor for `@` and `~`.
 ##! These patterns are used across multiple files, change with care.
-##!> include-except unix-shell-upto3 unix-shell-fps-pl2 -- @ [\s<>&|)] ~ \S
-##!> include-except unix-shell-4andup unix-shell-fps-pl2 -- @ [\s<>&|)] ~ \S
+##!> include-except unix-shell-upto3 unix-shell-fps-pl2 unix-shell-fps-pl2-start-of-string -- @ [\s<>&|)] ~ \S
+##!> include-except unix-shell-4andup unix-shell-fps-pl2 unix-shell-fps-pl2-start-of-string -- @ [\s<>&|)] ~ \S

regex-assembly/932238.ra

@@ -4,13 +4,7 @@
 ##!+ i
 
 ##!> assemble
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix-start-of-string
-  ##!<
-
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix
-  ##!<
+  ##!> include unix-shell-evasion-prefix-start-of-string
 ##!<
 ##!=>
 

regex-assembly/932239.ra

@@ -4,18 +4,12 @@
 ##!+ i
 
 ##!> assemble
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix-start-of-string
-  ##!<
-
-  ##!> assemble
-    ##!> include unix-shell-evasion-prefix
-  ##!<
+  ##!> include unix-shell-evasion-prefix-start-of-string
 ##!<
 ##!=>
 
 ##! These patterns are approximations of the patterns used by the cmdline
 ##! processor for `@` and `~`.
 ##! These patterns are used across multiple files, change with care.
-##!> include-except unix-shell-upto3 unix-shell-fps-pl2 unix-shell-fps-useragents -- @ [\s<>&|)] ~ \S
-##!> include-except unix-shell-4andup unix-shell-fps-pl2 unix-shell-fps-useragents -- @ [\s<>&|)] ~ \S
+##!> include-except unix-shell-upto3 unix-shell-fps-pl2 unix-shell-fps-pl2-start-of-string unix-shell-fps-useragents -- @ [\s<>&|)] ~ \S
+##!> include-except unix-shell-4andup unix-shell-fps-pl2 unix-shell-fps-pl2-start-of-string unix-shell-fps-useragents -- @ [\s<>&|)] ~ \S

regex-assembly/932240.ra

@@ -16,10 +16,21 @@
 ##! - globbing pattern expansion: {n$u\c$u,-nlvp,777}
 ##! - globbing: garb=cur[l];$garb+google.com
 
+##! kill '-'9
 ##!> assemble
   [a-z0-9_-]+
   ##!=>
-  [\x5c'\"\[\]]+
+  \s*['\"][^'\"\s]+['\"]
+  ##!=>
+  [a-z0-9_-]+
+##!<
+
+##!> assemble
+  [a-z0-9_-]+
+  ##!=>
+  ##! py""thon
+  ['\"]['\"]+
+  [\x5c\[\]]+
   \$+[\x5ca-z0-9_@?!#{*-]+
   ##! process substitution
   ``

regex-assembly/942500.ra

@@ -0,0 +1,9 @@
+##! Please refer to the documentation at
+##! https://coreruleset.org/docs/development/regex_assembly/.
+
+##!> define comment-contents (?:[\w\s=_\-()]+)
+##!> define c-style-modifiers \s*?[!+]
+
+##!+ i
+
+/\*{{c-style-modifiers}}{{comment-contents}}?\*/

regex-assembly/exclude/unix-shell-fps-pl2-start-of-string.ra

@@ -0,0 +1,19 @@
+##! Please refer to the documentation at
+##! https://coreruleset.org/docs/development/regex_assembly/.
+
+##! This list excludes command words that are prone to cause false
+##! positives from the following include files:
+##! - unix-shell-upto3.ra
+##! - unix-shell-4andup.ra
+
+##! To reduce complexity, this file simply lists all possible
+##! variants of a word, so when a word would be changed from, e.g.,
+##! `awk@` to `awk~`, this list would not have to be updated.
+##! See also unix-shell-fps-pl1.ra.
+
+as
+as@
+as~
+at
+at@
+at~

regex-assembly/include/unix-shell-evasion-prefix-start-of-string.ra

@@ -1,56 +1,6 @@
 ##! Please refer to the documentation at
 ##! https://coreruleset.org/docs/development/regex_assembly/.
 
-##! This assembly constructs the prefix used by 932250 and 932260
-
-##! ifconfig
+##! ifconfig (start of string)
 ^
-##! ={ifconfig}
-=
-##!=>
-
-##! match possible white space between prefix expressions
-\s*
-##!=>
-
-##! commands prefix
-##!> assemble
-  ##! time ifconfig
-  ##!> cmdline unix
-    time
-  ##!<
-  ##! { ifconfig }
-  \{
-  ##! ( ifconfig )
-  \s*\(\s*
-  ##! VARNAME=xyz ifconfig
-  \w+=(?:[^\s]*|\$.*|\$.*|<.*|>.*|\'.*\'|\".*\")\s+
-  ##! ! ifconfig
-  !\s*
-  ##! $ifconfig
-  \$
-##!<
-##!=>
-
-*
-##!=>
-
-##! match possible white space between prefix expressions
-\s*
-##!=>
-
-##! quoting prefix
-##!> assemble
-  ##! 'ifconfig'
-  '
-  ##! "ifconfig"
-  \"
-##!<
-##!=>
-
-*
-##!=>
-
-##! paths prefix (+ evasion prevention suffix [\x5c'\"]*)
-(?:[\?\*\[\]\(\)\-\|+\w'\"\./\x5c]+/)?[\x5c'\"]*
-##!=>
+##!> include unix-shell-evasion-prefix.ra

regex-assembly/include/unix-shell-evasion-prefix.ra

@@ -6,12 +6,20 @@
 ##! Separate rules target commands that do not follow this prefix,
 ##! as the chance of false positives is higher without a prefix match.
 
-##! time ifconfig
+##! <some command> ifconfig
 ##!> cmdline unix
+  busybox
+  command
+  ltrace
+  strace
   time
+  timeout
+  watch
 ##!<
 ##! ;ifconfig
 ;
+##! =ifconfig
+=
 ##! {ifconfig}
 \{
 ##! |ifconfig
@@ -40,6 +48,8 @@
 >\(
 ##! a() ( ifconfig; ); a
 \(\s*\)
+##! `cat<<<ifconfig` or `cat<<< ifconfig`
+<<<
 ##!=>
 
 ##! match possible white space between prefix expressions

regex-assembly/include/windows-commands-prefix.ra

@@ -9,10 +9,6 @@
 ##! Note: the quoting prefixes are part of the command prefixes, except for ^
 ##!       which, for unknown reasons, is not part of the expression
 
-##! time cmd
-##!> cmdline windows
-  time
-##!<
 ##! ;cmd
 ;
 ##! {cmd