Fork test pr 1 (#7)

* Add a markdown file for dashboard descriptions, link new file in README.md (cisagov#138) * Add a markdown file for dashboard descriptions ## 🗣 Description ##  Add dashboard-descriptions.md in /docs/markdown/reference. Add a link to this file within the main README.md's table of contents. ### 💭 Motivation and context     The LME repository does not have a location for dashboard descriptions. ## 🧪 Testing    N/A ## ✅ Pre-approval checklist ## - [x] Changes are limited to a single goal **AND** the title reflects this in a clear human readable format - [x] I have read and agree to LME's [CONTRIBUTING.md](https://github.com/cisagov/LME/CONTRIBUTING.md) document. - [x] The PR adheres to LME's requirements in [RELEASES.md](https://github.com/cisagov/LME/RELEASES.md#steps-to-submit-a-PR) - [x] These code changes follow [cisagov code standards](https://github.com/cisagov/development-guide). - [x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR. ## ✅ Post-merge Checklist - [x] Squash all commits into one PR level commit - [x] Delete the branch to keep down number of branches * Update README.md to include dashboard-descriptions.md * Update wording for computer software overview dashboard * Fix some grammatical changes in dashboard-descriptions.md * Release 1.3.1 merge into main (cisagov#154) * Update retention function to fix retention policy bug (cisagov#143) * Updated troubleshooting guide to account for index management (cisagov#134) * Update upgrading.md to account for 1.3.1 (cisagov#151) * Update upgrading.md * Update upgrading.md --------- Co-authored-by: Andrew Arz <[email protected]> * Fixes dashboard update not importing on fresh install (cisagov#167) (cisagov#169) * Fixes dashboard update not importing on fresh install cisagov#165 * Update upgrading.md to include status on v1.3.2, along with revisions to the document overall * remove step 4 from upgrading.md; add additional instructions for v1.3.2 --------- Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]> * create run_release.yaml github action workflow * create test commits for sample PR * test custom date and body --------- Co-authored-by: Andrew Arz <[email protected]> Co-authored-by: Clint Baxley <[email protected]> Co-authored-by: Clint Baxley <[email protected]>
mitchelbaker-cisa · Feb 2, 2024 · fcf0d15 · fcf0d15
1 parent d6ea1f4
commit fcf0d15
Show file tree

Hide file tree

Showing 10 changed files with 142 additions and 61 deletions.
diff --git a/.github/workflows/run_release.yaml b/.github/workflows/run_release.yaml
@@ -22,12 +22,17 @@ jobs:
       - name: Build
         run: git ls-files | zip LME-${{ inputs.version }}.zip -@
 
+      - name: Get current date 
+        id: date
+        run: echo "{date}={$(date + '%Y-%m-%d')}" >> $GITHUB_STATE
+
       - name: Release 
         uses: softprops/action-gh-release@v1
         with:
           name: ${{ inputs.releaseName }}
           tag_name: v${{ inputs.version }}
-          files: LME-${{ inputs.version }}.zip
+          files: LME-${{ steps.tag.outputs.tag }}.zip
           draft: true
           generate_release_notes: true 
+          body: "## [${{ steps.tag.outputs.tag }}] - Timberrrrr! - ${{ env.date }}"
           fail_on_unmatched_files: true
diff --git a/Chapter 3 Files/dashboard_update.sh b/Chapter 3 Files/dashboard_update.sh
@@ -9,7 +9,7 @@ if [ -r /opt/lme/lme.conf ]; then
   #reference this file as a source
   . /opt/lme/lme.conf
   #check if the version number is equal to the one we want
-  if [ "$version" == "1.3.0" ]; then
+  if [ "$version" == "1.3.0" ] || [ "$FRESH_INSTALL" = "true" ]; then
     echo -e "\e[32m[X]\e[0m Updating from git repo"
     git -C /opt/lme/ pull
     #make sure the hostname variable is present

diff --git a/Chapter 3 Files/deploy.sh b/Chapter 3 Files/deploy.sh
@@ -534,35 +534,39 @@ function pipelineupdate() {
 }
 
 function data_retention() {
-  #show ext4 disk
+  # Show ext4 disk
   DF_OUTPUT="$(df -h -l -t ext4 --output=source,size /var/lib/docker)"
 
-  #pull dev name
+  # Pull device name
   DISK_DEV="$(echo "$DF_OUTPUT" | grep -Po '[0-9]+G')"
 
-  #pull dev size
-  DISK_SIZE_ROUND="${DISK_DEV/G/}"
+  # Pull device size
+  DISK_SIZE="${DISK_DEV/G/}"
 
-  #lets do math to get 75% (%80 is low watermark for ES but as curator uses this we want to delete data *before* the disk gets full)
-  DISK_80=$((DISK_SIZE_ROUND * 80 / 100))
+  # Check if DISK_SIZE is empty or not a number
+  if ! [[ "$DISK_SIZE" =~ ^[0-9]+$ ]]; then
+    echo -e "\e[31m[!]\e[0m DISK_SIZE not an integer or is empty - exiting."
+    exit 1
+  fi
 
-  echo -e "\e[32m[X]\e[0m We think your main disk is $DISK_DEV"
+  echo -e "\e[32m[X]\e[0m We think your main disk is $DISK_DEV and its size is $DISK_SIZE gigabytes"
 
-  if [ "$DISK_80" -lt 30 ]; then
-    echo -e "\e[31m[!]\e[0m LME Requires 128GB of space usable for log retention - exiting"
-    exit 1
-  elif [ "$DISK_80" -ge 90 ] && [ "$DISK_80" -le 179 ]; then
+  if [ "$DISK_SIZE" -lt 128 ]; then
+    echo -e "\e[33m[!]\e[0m Warning: Disk size less than 128GB, recommend a larger disk for production environments. Install continuing..."
+    sleep 3
     RETENTION="30"
-  elif [ "$DISK_80" -ge 180 ] && [ "$DISK_80" -le 359 ]; then
+  elif [ "$DISK_SIZE" -ge 128 ] && [ "$DISK_SIZE" -le 179 ]; then
+    RETENTION="45"
+  elif [ "$DISK_SIZE" -ge 180 ] && [ "$DISK_SIZE" -le 359 ]; then
     RETENTION="90"
-  elif [ "$DISK_80" -ge 360 ] && [ "$DISK_80" -le 539 ]; then
+  elif [ "$DISK_SIZE" -ge 360 ] && [ "$DISK_SIZE" -le 539 ]; then
     RETENTION="180"
-  elif [ "$DISK_80" -ge 540 ] && [ "$DISK_80" -le 719 ]; then
+  elif [ "$DISK_SIZE" -ge 540 ] && [ "$DISK_SIZE" -le 719 ]; then
     RETENTION="270"
-  elif [ "$DISK_80" -ge 720 ]; then
+  elif [ "$DISK_SIZE" -ge 720 ]; then
     RETENTION="365"
   else
-    echo -e "\e[31m[!]\e[0m Unable to determine retention policy - exiting"
+    echo -e "\e[31m[!]\e[0m Unable to determine disk size - exiting."
     exit 1
   fi
 
@@ -736,6 +740,7 @@ function fixreadability() {
 
 
 function install() {
+  export FRESH_INSTALL="true"
   echo -e "Will execute the following intrusive actions:\n\t- apt update & upgrade\n\t- install docker (please uninstall before proceeding, or indicate skipping the install)\n\t- initialize docker swarm (execute \`sudo docker swarm leave --force\`  before proceeding if you are part of a swarm\n\t- automatic os updates via unattened-upgrades\n\t- checkout lme directory to latest version, and throw away local changes)"
 
   prompt "Proceed?"
@@ -748,10 +753,11 @@ function install() {
   fi
 
   echo -e "\e[32m[X]\e[0m Updating OS software"
-  apt update && apt upgrade -y
+  apt-get update
+  DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get  upgrade -yq
 
   echo -e "\e[32m[X]\e[0m Installing prerequisites"
-  apt install ${REQUIRED_PACKS[*]} -y -q
+  DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get install ${REQUIRED_PACKS[*]} -yq
 
   if [ -f /var/run/reboot-required ]; then
     echo -e "\e[31m[!]\e[0m A reboot is required in order to proceed with the install."
@@ -1169,7 +1175,7 @@ then
   ready "Will install the following packages: ${missing_pkgs[*]}. These are required for LME." 
   sudo apt-get update
   #confirm install
-  sudo apt-get --yes install ${missing_pkgs[*]}
+  sudo DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=a apt-get -yq install ${missing_pkgs[*]}
 fi
 
 #Change current working directory so relative filepaths work

diff --git a/README.md b/README.md
@@ -66,7 +66,8 @@ Figure 1: The 3 primary groups of computers in the LME architecture, their descr
 
 ### Reference:
  - [FAQ](/docs/markdown/reference/faq.md)  
- - [Troubleshooting](/docs/markdown/reference/troubleshooting.md)  
+ - [Troubleshooting](/docs/markdown/reference/troubleshooting.md)
+ - [Dashboard Descriptions](/docs/markdown/reference/dashboard-descriptions.md)
  - [Guide to Organizational Units](/docs/markdown/chapter1/guide_to_ous.md)
 
 ### Maintenance:

diff --git a/docs/markdown/maintenance/upgrading.md b/docs/markdown/maintenance/upgrading.md
@@ -6,8 +6,6 @@ Below you can find the upgrade paths that are currently supported and what steps
 
 Applying these changes is automated for any new installations. But, if you have an existing installation, you need to conduct some extra steps. **Before performing any of these steps it is advised to take a backup of the current installation using the method described [here](/docs/markdown/maintenance/backups.md).**
 
-To upgrade to the latest version from Release 1.2.0 to Release 1.3.0 [go here](#6-upgrade-from-120-to-130).
-
 ## 1. Finding your LME version (and the components versions)
 When reporting an issue or suggesting improvements, it is important to include the versions of all the components, where possible. This ensures that the issue has not already been fixed! 
 
@@ -26,10 +24,10 @@ When reporting an issue or suggesting improvements, it is important to include t
 
 
 ## 2. Upgrade from versions prior to v0.5
-LME does not support upgrading directly from versions prior to 0.5 to 1.0. Prior to switching to CISA's repo, first upgrade to the latest version of LME published by the NCSC (v0.5.1). Then follow the instructions above to upgrade to v1.0.
+LME does not support upgrading directly from versions prior to v0.5 to v1.0. Prior to switching to CISA's repo, first upgrade to the latest version of LME published by the NCSC (v0.5.1). Then follow the instructions above to upgrade to v1.0.
 
 
-## 3. Upgrade from v0.5 to 1.0.0
+## 3. Upgrade from v0.5 to v1.0.0
 
 Since LME's transition from the NCSC to CISA, the location of the LME repository has changed from `https://github.com/ukncsc/lme` to `https://github.com/cisagov/lme`. To obtain any further updates to LME on the ELK server, you will need to transition to the new git repository. Because vital configuration files are stored within the same folder as the git repo, it's simpler to copy the old LME folder to a different location, clone the new repo, copy the files and folders unique to your system, and then optionally delete the old folder. You can do this by running the following commands:
 
@@ -111,55 +109,27 @@ LME v1.0 made a minor change to the file structure used in the SYSVOL folder, so
 3. Is the LME folder inside SYSVOL properly structured? Refer to the checklist listed at the end of chapter 2.
 4. Are the events from all clients visible inside elastic? Refer to [4.1.2 Check you are receiving logs](/docs/markdown/chapter4.md#412-check-you-are-receiving-logs).
 
+## 4. Upgrade to v1.3.1 
 
-## 4. Upgrade from 1.0.0 to 1.1.0
-To fetch the latest changes, on the Linux server, run the following commands as root:
-```
-cd /opt/lme
-git pull
-```
+This is a hotfix to the install script and some additional troubleshooting steps added to documentation on space management. Unless you're encountering problems with your current installation, or if your logs are running out of space, there's no need to upgrade to v1.3.1, as it doesn't offer any additional functionality changes.
 
-To manually update the dashboards, see [How to update dashboards](/Chapter%204%20Files/dashboards#how-to-update-dashboards).
+## 5. Upgrade to v1.3.2 
 
-Additionally, to fix a potential file permission issue present in v1.0.0, run the following command on the Linux server:
-```
-sudo chown -R 1000:1000 /opt/lme/backups
-```
-
-See [Directory permission issues](/docs/markdown/reference/troubleshooting.md#directory-permission-issues) for more details.
+This is a hotfix to address dashboards which failed to load on a fresh install of v1.3.1. If you are currently running v1.3.0, you do not need to upgrade at this time.  If you are running versions **before** 1.3.0 or are running v1.3.1, we recommend you upgrade to the latest version.
 
+Please refer to the [Upgrading to latest version](/docs/markdown/maintenance/upgrading.md#upgrading-to-latest-version) to apply the hotfix.
 
-## 5. Upgrade from 1.1.0 to 1.2.0
+## 6. Upgrade to latest version 
 To fetch the latest changes, on the Linux server, run the following commands as root:
 ```
-cd /opt/lme/Chapter\ 3\ Files/
-sudo ./deploy.sh uninstall
-cd /opt/lme
 git pull
-cd Chapter\ 3\ Files/
-sudo ./deploy.sh install
-```
-
-The deploy.sh script should have now created new files on the Linux server at location /opt/lme/files_for_windows.zip . This file needs to be copied across and used on the Windows Event Collector server like it was explained in Chapter 3 sections [3.2.4 & 3.3 ](/docs/markdown/chapter3/chapter3.md#324-download-files-for-windows-event-collector). 
-
-Then reboot your Client computers & Windows Event Collector. On Windows Event Collector open services.msc as an administrator and make sure the winlogbeat service is set to start automatically, and is running.
-
-## 6. Upgrade from 1.2.0 to 1.3.0
-To fetch the latest changes, run the following commands as root on the Linux server:
-```
+git checkout main
 cd /opt/lme/Chapter\ 3\ Files/
 sudo ./deploy.sh uninstall
 cd /opt/lme
-git pull
 cd Chapter\ 3\ Files/
 sudo ./deploy.sh install
 ```
 
 The deploy.sh script should have now created new files on the Linux server at location /opt/lme/files_for_windows.zip . This file needs to be copied across and used on the Windows Event Collector server like it was explained in Chapter 3 sections [3.2.4 & 3.3 ](/docs/markdown/chapter3/chapter3.md#324-download-files-for-windows-event-collector). 
 
-Then reboot your Client computers & Windows Event Collector. On Windows Event Collector open services.msc as an administrator and make sure the winlogbeat service is set to start automatically, and is running.
-
-
-
-
-
diff --git a/docs/markdown/reference/dashboard-descriptions.md b/docs/markdown/reference/dashboard-descriptions.md
@@ -0,0 +1,40 @@
+# Dashboard Descriptions
+
+## Purpose   
+Logging Made Easy (LME) releases new dashboards on GitHub periodically. Here are the dashboard descriptions.
+
+## User Human Resources  
+
+The User Human Resources Dashboard provides a comprehensive overview of network activity and displays domains, users, workstations, activity times and days of the week. It includes details on general logon events, logoff events and distinguishes between in-person and remote logons. Analogous to a security guard monitoring a camera, the dashboard facilitates network monitoring by revealing overall network traffic, user locations, peak hours and the ratio of remote-to-in-person logons. Users can filter and analyze individual or specific computer activity logs. 
+
+## Computer Software Overview
+
+The Computer Software Overview Dashboard displays application usage on host computers, logging events for application failures, hangs and external connection attempts. Monitoring application usage is crucial for assessing network health, as frequent crashes may indicate larger issues, and applications making frequent external requests could signal malicious activity.  
+
+## Security Log
+
+The Security Log Dashboard actively presents forwarded security log events, tallies failed logon attempts, identifies computers with failed logon events, specifies reasons for failed logons and distinguishes types of logons and reports on credential status (clear text or cached). It also discloses whether the event log or Windows Security audit log is cleared, highlights user account changes and notes the assignment of special privileges to a logon session. Users can quickly detect unusual events, prompting further investigation and remediation actions. 
+
+## Process Explorer 
+
+The Process Explorer Dashboard thoroughly monitors networks, tracks processes, users, processes per user, files, filenames in the download directory, Sysmon process creation and registry events. It offers user-friendly filtering for process names and process identifiers or PID’s. The download directory is often targeted for initial malware installations due to lenient write privileges. This dashboard investigates unusual registry changes and closely examine spikes in processes created by specific users, as these could indicate potential malicious activity. 
+
+## Sysmon Summary
+
+The Sysmon Summary Dashboard highlights Sysmon events and features event count, event types, the percentage breakdown by event code and top hosts generating Sysmon data. Vigilance towards any deviations or shifts in activity levels helps administrators to promptly identify both desired and undesired activities. 
+
+## User Security
+
+The User Security Dashboard provides a comprehensive view of network activity and showcases logon attempts, user logon/logoff events, logged-on computers and detailed network connections by country and protocol. Additionally, it highlights critical information such as PowerShell events, references to temporary files and Windows Defender alerts for malware detection and actions taken. The dashboard supports effective monitoring by allowing users to filter events based on users, domains and hosts. Understanding the nature and origin of network connections is vital, and the dashboard facilitates the identification of suspicious activities, enabling operators to target their inquiries for enhanced network health assessment. 
+
+## Alert
+
+The Alert Dashboard enables users to define rules that detect complex conditions within networks/environments. It also uses trigger actions in case of suspicious activities. These alerts contain pre-built rules that detects suspicious activities.  There are options that schedule how these suspicious activities are detected and actions taken when these conditions are detected. 
+
+## Healthcheck 
+
+The HealthCheck Dashboard gives users the ability to view different processes such as unexpected shutdowns, events by each machine, total hosts and total number of logged in admins with data that is based on a selected date range.  Users can verify the health of their system by observing events such as if there are more admin users than expected or if an unexpected shutdown occurs. 
+
+
+
+For more information or to seek additional help, [Click Here](https://github.com/cisagov/LME) 
diff --git a/docs/markdown/reference/troubleshooting.md b/docs/markdown/reference/troubleshooting.md
@@ -327,3 +327,62 @@ sudo curl -X POST "https://127.0.0.1:9200/_security/user/elastic/_password" -H "
 Replace 'currentpassword' with your current password and 'newpassword' with the password you would like to change it to. 
 
 Utilize environment variables in place of currentpassword and newpassword to avoid saving your password to console history. If not we recommend you clear your history after changing the password with ```history -c```
+
+## Index Management
+
+If you are having issues with your hard disk filling up too fast you can use these steps to delete logs earlier than your current settings.
+
+1. **Log in to Elastic**
+   - Access the Elastic platform and log in with your credentials.
+
+2. **Navigate to Management Section**
+   - In the main menu, scroll down to "Management."
+
+3. **Access Stack Management**
+   - Within the Management section, select "Stack Management."
+
+4. **Select Index Lifecycle Policies**
+   - In Stack Management, find and choose "Index Lifecycle Policies."
+
+5. **Choose the Relevant ILM Policy**
+   - From the list, select `lme_ilm_policy` for editing.
+
+6. **Adjust the Hot Phase Settings**
+   - Navigate to the 'Hot Phase' section.
+   - Expand 'Advanced settings'.
+   - Uncheck "Use recommended defaults."
+   - Change the "Maximum age" setting to match your desired delete phase duration.
+
+     > **Note:** Aligning the maximum age in the hot phase with the delete phase ensures consistency in data retention.
+
+7. **Adjust the Delete Phase Settings**
+   - Scroll to the 'Delete Phase' section.
+   - Find and adjust the "Move data into phase when:" setting.
+   - Ensure the delete phase duration matches the maximum age set in the hot phase.
+
+     > **Note:** This setting determines the deletion timing of your logs. Ensure to back up necessary data before changes.
+
+8. **Save Changes**
+   - Save the adjustments you've made.
+
+9. **Verify the Changes**
+   - Review and ensure that the changes are functioning as intended. Indices may not delete immediately - allow time for job to run.
+
+10. **Document the Changes**
+    - Record the modifications for future reference.
+
+You can also manually delete an index from the GUI under Management > Index Managment or by using the following command: 
+
+```
+curl -X DELETE "https://127.0.0.1:9200/your_index_name" -H "Content-Type: application/json" --cacert /opt/lme/Chapter\ 3\ Files/certs/root-ca.crt -u elastic:yourpassword
+```
+> **Note:**    Ensure this is not your current winlogbeat index in use. You should only delete indices that have already rolled over. i.e. if you have index winlogbeat-00001 and winlogbeat-00002 do NOT delete winlogbeat-00002.
+
+If you only have one index you can manually force a rollover with the following command: 
+
+```
+curl -X POST "https://127.0.0.1:9200/winlogbeat-alias/_rollover" -H "Content-Type: application/json" --cacert /opt/lme/Chapter\ 3\ Files/certs/root-ca.crt -u elastic:yourpassword
+```
+
+This will rollover winlogbeat-00001 and create winlogbeat-00002. You can now manually delete 00001. 
+
diff --git a/test/test1.txt b/test/test1.txt
diff --git a/test/test2.txt b/test/test2.txt
diff --git a/test/test3.txt b/test/test3.txt