Merge pull request #10 from userjack6880/dev

Dev to Main
userjack6880 · May 25, 2023 · 4bb81fb · 4bb81fb
2 parents a4b80cf + fe130fd
commit 4bb81fb
Show file tree

Hide file tree

Showing 11 changed files with 409 additions and 166 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -24,8 +24,8 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Software Version**
+[] Version 0 Alpha 3
 [] Version 0 Alpha 2
-[] Version 0 Alpha 1
 
 **Server (please complete the following information):**
 - Server OS [e.g. CentOS 7.3]

diff --git a/.github/ISSUE_TEMPLATE/critical_report.md b/.github/ISSUE_TEMPLATE/critical_report.md
@@ -24,8 +24,9 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.
 
 **Software Version**
+[] Version 0 Alpha 3
 [] Version 0 Alpha 2
-[] Version 0 Alpha 1
+[] Other
 
 **Server (please complete the following information):**
 - Server OS [e.g. CentOS 7.3]

diff --git a/README.md b/README.md
@@ -8,7 +8,7 @@ A Perl based tool to parse DMARC reports, based on John Levine's [rddmarc](http:
 
 Open Report Parser is a fork of [techsneeze's dmarcts-report-parser](https://github.com/techsneeze/dmarcts-report-parser), and was forked to more closely match the needs of [Open DMARC Analyzer](https://github.com/userjack6880/Open-DMARC-Analyzer).
 
-Open Report Parser Version 0 Alpha 1 (0-α2) is an [Anomaly \<Codebase\>](https://systemanomaly.com/codebase) project by John Bradley ([email protected]).
+Open Report Parser Version 0 Alpha 3 (0-α3) is an [Anomaly \<Codebase\>](https://systemanomaly.com/codebase) project by John Bradley ([email protected]).
 
 # Minimum Requirements
 
@@ -23,7 +23,7 @@ Open Report Parser Version 0 Alpha 1 (0-α2) is an [Anomaly \<Codebase\>](https:
 ```
 apt-get install libfile-mimeinfo-perl libmail-imapclient-perl libmime-tools-perl libxml-simple-perl \
 libio-socket-inet6-perl libio-socket-ip-perl libperlio-gzip-perl \
-libmail-mbox-messageparser-perl unzip
+libmail-mbox-messageparser-perl libwww-perl unzip
 ```
 
 - For MySQL: `libdbd-mysql-perl`
@@ -34,7 +34,7 @@ libmail-mbox-messageparser-perl unzip
 
 ```
 sudo dnf install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \
-perl-Socket6 perl-PerlIO-gzip unzip
+perl-Socket6 perl-PerlIO-gzip perl-libwww-perl unzip
 ```
 
 - For MySQL: `perl-DBD-MySQL`
@@ -45,7 +45,7 @@ perl-Socket6 perl-PerlIO-gzip unzip
 ```
 yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
 yum install perl-File-MimeInfo perl-Mail-IMAPClient perl-MIME-tools perl-XML-Simple perl-DBI \
-perl-Socket6 perl-PerlIO-gzip unzip perl-Mail-Mbox-MessageParser
+perl-Socket6 perl-PerlIO-gzip perl-libwww-perl unzip perl-Mail-Mbox-MessageParser
 ```
 
 - For MySQL: `perl-DBD-MySQL`
@@ -54,7 +54,7 @@ perl-Socket6 perl-PerlIO-gzip unzip perl-Mail-Mbox-MessageParser
 ## on FreeBSD (FreeBSD 11.4)
 
 ```
-sudo pkg install p5-File-MimeInfo p5-Mail-IMAPClient p5-MIME-tools p5-XML-Simple p5-DBI p5-Socket6 p5-PerlIO-gzip p5-Mail-Mbox-MessageParser unzip
+sudo pkg install p5-File-MimeInfo p5-Mail-IMAPClient p5-MIME-tools p5-XML-Simple p5-DBI p5-Socket6 p5-PerlIO-gzip p5-Mail-Mbox-MessageParser p5-libwww unzip
 ```
 
 - For MySQL: `p5-DBD-MySQL`
@@ -68,6 +68,7 @@ update-mime-database /usr/local/share/mime
 perl -MCPAN -e 'install Mail::IMAPClient'
 perl -MCPAN -e 'install Mail::Mbox::MessageParser'
 perl -MCPAN -e 'install File::MimeInfo'
+perl -MCPAN -e 'install LWP::UserAgent
 ```
 
 - For MySQL: `perl -MCPAN -e 'install DBD::mysql'`
@@ -127,6 +128,11 @@ $imapssl          = '0'; # If set to 1, remember to change server port to 993 an
 $imaptls          = '0';
 $tlsverify        = '0';
 $imapignoreerror  = '0'; # recommended if you use MS Exchange 2007, ...
+#$imapauth        = 'simple'; # supported - simple, oauth2 - defaults to simple if unset
+
+# see documentation for detailed setup
+#$oauthclientid   = ''; 
+#$oauthuri        = '';
 
 $imapdmarcfolder  = 'dmarc';
 $imaptlsfolder    = 'tls';
@@ -142,7 +148,9 @@ $imaptlsfolder    = 'tls';
 # $imaptlserr   = 'tls.notProcessed';
 ```
 
-These settings are ignored when using the -m flag.
+These settings are ignored when using the -m flag. When using SSL, TLS needs to be disabled and the port used should be changed to 993. TLS Verify is ignored.
+
+Setting `$imapauth` to 'oauth2' enables OAuth2 authentication, and requires an initial dance to verify the application with your provider. Once this is done, it should be able to renew the token automatically on subsequent runs. Currently, only OAuth2 with M365 has been tested.
 
 **XML Storage Options**
 
@@ -207,16 +215,15 @@ Currently, processing of both DMARC and TLS reports during the same run is only
 
 # Latest Changes
 
-## 0-α2
-- Fixed errors in previous release incorporating postgres support related to table creation.
-- Added MTA-TLS report support.
-- More useful debug output.
-- Code consolodation (eg, subroutine repetative code).
+## 0-α3
+- Postgres fixes (and validation). Fixes Issue #8.
+- Initial Oauth2 Support code (untested).
 
 # Tested System Configurations
-| OS          | Perl      | SQL             |
-| ----------- | --------- | --------------- |
-| Debian 11.6 | Perl 5.32 | MariaDB 10.5.18 |
+| OS          | Perl      | SQL             | Source                          |
+| ----------- | --------- | --------------- | ------------------------------- |
+| Debian 11.6 | Perl 5.32 | MariaDB 10.5.18 | Postfix/Dovecot IMAP Basic Auth |
+| Debian 11.6 | Perl 5.32 | PostgreSQL 13.9 | Postfix/Dovecot IMAP Basic Auth |
 
 # Release Cycle and Versioning
 
@@ -228,8 +235,9 @@ Support will be provided as outlined in the following schedule. For more details
 
 | Version                             | Support Level    | Released         | End of Support   | End of Life      |
 | ----------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
-| Version 1 Alpha 2                   | Full Support     | 26 April 2023    | TBD              | TBD              |
-| Version 1 Alpha 1                   | Critical Support | 19 April 2023    | 26 April 2023    | TBD              |
+| Version 1 Alpha 3                   | Full Support     | 25 May 2023      | TBD              | TBD              |
+| Version 1 Alpha 2                   | Critical Support | 26 April 2023    | 25 May 2023      | TBD              |
+| Version 1 Alpha 1                   | End of Life      | 19 April 2023    | 26 April 2023    | 25 May 2023      |
 
 # Contributing
 

diff --git a/dbx_mysql.pl b/dbx_mysql.pl
@@ -1,12 +1,12 @@
 # -----------------------------------------------------------------------------
 #
-# Open Report Parser - Open Source DMARC report parser
+# Open Report Parser - Open Source report parser
 # Copyright (C) 2023 John Bradley (userjack6880)
 # Copyright (C) 2016 TechSneeze.com
 # Copyright (C) 2012 John Bieling
 #
-# report-parser.pl
-#   main script
+# dbx_mysql.pl
+#   mysql db module
 #
 # Available at: https://github.com/userjack6880/Open-Report-Parser
 #
@@ -27,35 +27,6 @@
 #  this program.  If not, see <https://www.gnu.org/licenses/>.
 #
 # -----------------------------------------------------------------------------
-#
-# The subroutines storeXMLInDatabase() and getXMLFromMessage() are based on
-# John R. Levine's rddmarc (http://www.taugh.com/rddmarc/). The following
-# special conditions apply to those subroutines:
-#
-# Copyright 2012, Taughannock Networks. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# Redistributions of source code must retain the above copyright notice, this
-# list of conditions and the following disclaimer.
-#
-# Redistributions in binary form must reproduce the above copyright notice, this
-# list of conditions and the following disclaimer in the documentation and/or
-# other materials provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-#
-# -----------------------------------------------------------------------------
 
 %dbx = (
   epoch_to_timestamp_fn => 'FROM_UNIXTIME',
@@ -143,7 +114,19 @@
       additional_definitions  => "KEY serial (serial, send_ip), KEY serial6 (serial, send_ip6)",
       table_options           => "",
       indexes                 => [],
-      }
+      },
+    "oauth" => {
+      column_definitions      => [
+        "id"                  , "int"           , "unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY",
+        "access_token"        , "varchar(4096)"  , "",
+        "refresh_token"       , "varchar(1024)"  , "",
+        "expire"              , "timestamp"     , "NOT NULL",
+        "valid"               , "int"           , "unsigned NOT NULL",
+        ],
+      additional_definitions  => "",
+      table_options           => "",
+      indexes                 => [],
+    }
     },
 
   add_column => sub {

diff --git a/dbx_postgres.pl b/dbx_postgres.pl
@@ -5,8 +5,8 @@
 # Copyright (C) 2016 TechSneeze.com
 # Copyright (C) 2012 John Bieling
 #
-# report-parser.pl
-#   main script
+# dbx_postgres.pl
+#   postgres db module
 #
 # Available at: https://github.com/userjack6880/Open-Report-Parser
 #
@@ -27,35 +27,6 @@
 #  this program.  If not, see <https://www.gnu.org/licenses/>.
 #
 # -----------------------------------------------------------------------------
-#
-# The subroutines storeXMLInDatabase() and getXMLFromMessage() are based on
-# John R. Levine's rddmarc (http://www.taugh.com/rddmarc/). The following
-# special conditions apply to those subroutines:
-#
-# Copyright 2012, Taughannock Networks. All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions are met:
-#
-# Redistributions of source code must retain the above copyright notice, this
-# list of conditions and the following disclaimer.
-#
-# Redistributions in binary form must reproduce the above copyright notice, this
-# list of conditions and the following disclaimer in the documentation and/or
-# other materials provided with the distribution.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
-# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
-# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
-# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
-# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
-# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
-# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
-# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-#
-# -----------------------------------------------------------------------------
 
 %dbx = (
   epoch_to_timestamp_fn => 'TO_TIMESTAMP',
@@ -150,8 +121,8 @@
       additional_definitions  => "PRIMARY KEY (id)",
       table_options           => "",
       indexes                 => [
-        "CREATE INDEX tlsrecord_idx_serial ON tlsrecord (serial, ip);",
-        "CREATE INDEX tlsrecord_idx_serial6 ON tlsrecord (serial, ip6);",
+        "CREATE INDEX tlsrecord_idx_serial ON tlsrecord (serial, send_ip);",
+        "CREATE INDEX tlsrecord_idx_serial6 ON tlsrecord (serial, send_ip6);",
         ],
       }
     },

diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
@@ -1,11 +1,15 @@
 # Changelog
 
-## 0-α1
-- Fork renamed
-- Incorporate changes made to original repository after fork ([commit 51ba1de](https://github.com/userjack6880/Open-Report-Parser/commit/51ba1de8521559647ebe4b8a1db291c26b572de4))
+## 0-α3
+- Postgres fixes (and validation). Fixes Issue #8.
+- Initial Oauth2 Support code (untested).
 
 ## 0-α2
 - Fixed errors in previous release incorporating postgres support related to table creation.
 - Added MTA-TLS report support.
 - More useful debug output.
-- Code consolodation (eg, subroutine repetative code).
+- Code consolodation (eg, subroutine repetative code).
+
+## 0-α1
+- Fork renamed
+- Incorporate changes made to original repository after fork ([commit 51ba1de](https://github.com/userjack6880/Open-Report-Parser/commit/51ba1de8521559647ebe4b8a1db291c26b572de4))
diff --git a/docs/SECURITY.md b/docs/SECURITY.md
@@ -20,5 +20,6 @@ Open DMARC Analyzer Version Security Support
 
 | Version                    | Supported | End of Life      |
 | -------------------------- | --------- | ---------------- |
+| Version 0 Alpha 3          | YES       | TBD              |
 | Version 0 Alpha 2          | YES       | TBD              |
-| Version 0 Alpha 1          | YES       | TBD              |
+| Version 0 Alpha 1          | YES       | 25 May 2023      |
diff --git a/docs/SUPPORT.md b/docs/SUPPORT.md
@@ -4,8 +4,9 @@ Due to the limited time available to developrs to maintain support, dates must b
 
 | Version                             | Support Level    | Released         | End of Support   | End of Life      |
 | ----------------------------------- | ---------------- | ---------------- | ---------------- | ---------------- |
-| Version 1 Alpha 2                   | Full Support     | 26 April 2023    | TBD              | TBD              |
-| Version 1 Alpha 1                   | Critical Support | 19 April 2023    | 26 April 2023    | TBD              |
+| Version 1 Alpha 3                   | Full Support     | 25 May 2023      | TBD              | TBD              |
+| Version 1 Alpha 2                   | Critical Support | 26 April 2023    | 25 May 2023      | TBD              |
+| Version 1 Alpha 1                   | End of Life      | 19 April 2023    | 26 April 2023    | 25 May 2023      |
 
 # Support Cycle