From 7b13fa7222afb0bf5668e442727982e4a2b35caf Mon Sep 17 00:00:00 2001 From: Seth Grover Date: Thu, 22 Sep 2022 11:59:20 -0600 Subject: [PATCH] documentation links work in progress --- README.md | 115 +++------------------------------------------------ docs/main.md | 107 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 110 deletions(-) create mode 100644 docs/main.md diff --git a/README.md b/README.md index 8db7ab20e..4f88aefea 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Malcolm -![](./images/logo/Malcolm_banner.png) +![](./docs/images/logo/Malcolm_banner.png) [Malcolm](https://github.com/idaholab/Malcolm) is a powerful network traffic analysis tool suite designed with the following goals in mind: @@ -15,105 +15,14 @@ Although all of the open source tools which make up Malcolm are already availabl In short, Malcolm provides an easily deployable network analysis tool suite for full packet capture artifacts (PCAP files) and Zeek logs. While Internet access is required to build it, it is not required at runtime. +## Documentation + +See the [**Malcolm documentation**](docs/main.md). + ## Share your feedback You can help steer Malcolm's development by sharing your ideas and feedback. Please take a few minutes to complete [this survey ↪](https://forms.gle/JYt9QwA5C4SYX8My6) (hosted on Google Forms) so we can understand the members of the Malcolm community and their use cases for this tool. -## Table of Contents - -* [Automated Build Workflows Status](#BuildBadges) -* [Quick start](docs/quickstart.md#QuickStart) - * [Getting Malcolm](docs/quickstart.md#GetMalcolm) - * [User interface](docs/quickstart.md#UserInterfaceURLs) -* [Overview](#Overview) -* [Components](docs/components.md#Components) -* [Supported Protocols](docs/protocols.md#Protocols) -* [Development](docs/development.md#Development) - * [Building from source](docs/development.md#Build) - * [Pre-Packaged installation files](docs/development.md#Packager) -* [Preparing your system](#Preparing) - * [Recommended system requirements](#SystemRequirements) - * [System configuration and tuning](docs/malcolm-config.md#ConfigAndTuning) - * [`docker-compose.yml` parameters](docs/malcolm-config.md#DockerComposeYml) - * [Linux host system configuration](#HostSystemConfigLinux) - * [macOS host system configuration](#HostSystemConfigMac) - * [Windows host system configuration](#HostSystemConfigWindows) -* [Running Malcolm](#Running) - * [OpenSearch instances](docs/opensearch-instances.md#OpenSearchInstance) - * [Authentication and authorization for remote OpenSearch clusters](#OpenSearchAuth) - * [Configure authentication](docs/authsetup.md#AuthSetup) - * [Local account management](docs/authsetup.md#AuthBasicAccountManagement) - * [Lightweight Directory Access Protocol (docs/LDAP) authentication](authsetup.md#AuthLDAP) - - [LDAP connection security](#AuthLDAPSecurity) - * [TLS certificates](docs/authsetup.md#TLSCerts) - * [Starting Malcolm](docs/running.md#Starting) - * [Stopping and restarting Malcolm](docs/running.md#StopAndRestart) - * [Clearing Malcolm's data](docs/running.md#Wipe) - * [Temporary read-only interface](#ReadOnlyUI) -* [Capture file and log archive upload](docs/upload.md#Upload) - - [Tagging](docs/upload.md#Tagging) - - [Processing uploaded PCAPs with Zeek and Suricata](docs/upload.md#UploadPCAPProcessors) -* [Live analysis](#LiveAnalysis) - * [Using a network sensor appliance](docs/live-analysis.md#Hedgehog) - * [Monitoring local network interfaces](#LocalPCAP) - * [Manually forwarding logs from an external source](#ExternalForward) -* [Arkime](#Arkime) - * [Zeek log integration](docs/arkime.md#ArkimeZeek) - - [Correlating Zeek logs and Arkime sessions](docs/arkime.md#ZeekArkimeFlowCorrelation) - * [Help](#ArkimeHelp) - * [Sessions](docs/arkime.md#ArkimeSessions) - * [PCAP Export](#ArkimePCAPExport) - * [SPIView](#ArkimeSPIView) - * [SPIGraph](docs/arkime.md#ArkimeSPIGraph) - * [Connections](#ArkimeConnections) - * [Hunt](docs/arkime.md#ArkimeHunt) - * [Statistics](#ArkimeStats) - * [Settings](#ArkimeSettings) -* [OpenSearch Dashboards](docs/dashboards.md#Dashboards) - * [Discover](#Discover) - - [Screenshots](#DiscoverGallery) - * [Visualizations and dashboards](docs/dashboards.md#DashboardsVisualizations) - - [Prebuilt visualizations and dashboards](#PrebuiltVisualizations) - - [Screenshots](#PrebuiltVisualizationsGallery) - - [Building your own visualizations and dashboards](docs/dashboards.md#BuildDashboard) - + [Screenshots](#NewVisualizationsGallery) -* [Search Queries in Arkime and OpenSearch](docs/queries-cheat-sheet.md#SearchCheatSheet) -* [Other Malcolm features](#MalcolmFeatures) - - [Automatic file extraction and scanning](docs/file-scanning.md#ZeekFileExtraction) - - [Automatic host and subnet name assignment](docs/host-and-subnet-mapping.md#HostAndSubnetNaming) - + [IP/MAC address to hostname mapping via `host-map.txt`](docs/host-and-subnet-mapping.md#HostNaming) - + [CIDR subnet to network segment name mapping via `cidr-map.txt`](docs/host-and-subnet-mapping.md#SegmentNaming) - + [Defining hostname and CIDR subnet names interface](docs/host-and-subnet-mapping.md#NameMapUI) - + [Applying mapping changes](#ApplyMapping) - - [OpenSearch index management](docs/index-management.md#IndexManagement) - - [Event severity scoring](docs/severity.md#Severity) - + [Customizing event severity scoring](#SeverityConfig) - - [Zeek Intelligence Framework](docs/zeek-intel.md#ZeekIntel) - + [STIX™ and TAXII™](docs/zeek-intel.md#ZeekIntelSTIX) - + [MISP](docs/zeek-intel.md#ZeekIntelMISP) - - [Anomaly Detection](#AnomalyDetection) - - [Alerting](docs/alerting.md#Alerting) - + [Email Sender Accounts](#AlertingEmail) - - ["Best Guess" Fingerprinting for ICS Protocols](docs/ics-best-guess.md#ICSBestGuess) - - [Asset Management with NetBox](docs/netbox.md#NetBox) - - [CyberChef](#CyberChef) - - [API](docs/api.md#API) - + [Examples](docs/api-examples.md#APIExamples) -* [Ingesting Third-party Logs](#ThirdPartyLogs) -* [Malcolm installer ISO](docs/malcolm-iso.md#ISO) - * [Installation](docs/malcolm-iso.md#ISOInstallation) - * [Generating the ISO](#ISOBuild) - * [Setup](#ISOSetup) - * [Time synchronization](#ConfigTime) - * [Hardening](docs/hardening.md#Hardening) - * [Compliance Exceptions](#ComplianceExceptions) -* [Installation example using Ubuntu 22.04 LTS](docs/ubuntu-install-example.md#InstallationExample) -* [Upgrading Malcolm](docs/malcolm-upgrade.md#UpgradePlan) -* [Modifying or Contributing to Malcolm](#Contributing) -* [Forks](#Forks) -* [Copyright](#Footer) -* [Contact](#Contact) - ## Automated Builds Status See [**Building from source**](docs/development.md#Build) to read how you can use GitHub [workflow files](./.github/workflows/) to build Malcolm. @@ -138,20 +47,6 @@ See [**Building from source**](docs/development.md#Build) to read how you can us ![malcolm-iso-build-docker-wrap-push-ghcr](https://github.com/mmguero-dev/Malcolm/workflows/malcolm-iso-build-docker-wrap-push-ghcr/badge.svg) ![sensor-iso-build-docker-wrap-push-ghcr](https://github.com/mmguero-dev/Malcolm/workflows/sensor-iso-build-docker-wrap-push-ghcr/badge.svg) -## Overview - -![Malcolm Network Diagram](./images/malcolm_network_diagram.png) - -Malcolm processes network traffic data in the form of packet capture (docs/PCAP) files or Zeek logs. A [sensor](live-analysis.md#Hedgehog) (packet capture appliance) monitors network traffic mirrored to it over a SPAN port on a network switch or router, or using a network TAP device. [Zeek](https://www.zeek.org/index.html) logs and [Arkime](https://molo.ch/) sessions are generated containing important session metadata from the traffic observed, which are then securely forwarded to a Malcolm instance. Full PCAP files are optionally stored locally on the sensor device for examination later. - -Malcolm parses the network session data and enriches it with additional lookups and mappings including GeoIP mapping, hardware manufacturer lookups from [organizationally unique identifiers (docs/OUI)](http://standards-oui.ieee.org/oui/oui.txt) in MAC addresses, assigning names to [network segments](host-and-subnet-mapping.md#SegmentNaming) and [hosts](host-and-subnet-mapping.md#HostNaming) based on user-defined IP address and MAC mappings, performing [TLS fingerprinting](#https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967), and many others. - -The enriched data is stored in an [OpenSearch](https://opensearch.org/) document store in a format suitable for analysis through two intuitive interfaces: OpenSearch Dashboards, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime, a powerful tool for finding and identifying the network sessions comprising suspected security incidents. These tools can be accessed through a web browser from analyst workstations or for display in a security operations center (SOC). Logs can also optionally be forwarded on to another instance of Malcolm. - -![Malcolm Data Pipeline](./images/malcolm_data_pipeline.png) - -For smaller networks, use at home by network security enthusiasts, or in the field for incident response engagements, Malcolm can also easily be deployed locally on an ordinary consumer workstation or laptop. Malcolm can process local artifacts such as locally-generated Zeek logs, locally-captured PCAP files, and PCAP files collected offline without the use of a dedicated sensor appliance. - ## Forks [CISA](https://www.cisa.gov/) maintains the upstream source code repository for Malcolm at [https://github.com/cisagov/Malcolm](https://github.com/cisagov/Malcolm). The [Idaho National Lab](https://inl.gov/)'s fork of Malcolm, which is currently kept up-to-date with CISA's upstream development, can be found at [https://github.com/idaholab/Malcolm](https://github.com/idaholab/Malcolm). diff --git a/docs/main.md b/docs/main.md new file mode 100644 index 000000000..6f4155345 --- /dev/null +++ b/docs/main.md @@ -0,0 +1,107 @@ +![](./images/logo/Malcolm_banner.png) + +## Overview + +![Malcolm Network Diagram](./images/malcolm_network_diagram.png) + +Malcolm processes network traffic data in the form of packet capture (docs/PCAP) files or Zeek logs. A [sensor](live-analysis.md#Hedgehog) (packet capture appliance) monitors network traffic mirrored to it over a SPAN port on a network switch or router, or using a network TAP device. [Zeek](https://www.zeek.org/index.html) logs and [Arkime](https://molo.ch/) sessions are generated containing important session metadata from the traffic observed, which are then securely forwarded to a Malcolm instance. Full PCAP files are optionally stored locally on the sensor device for examination later. + +Malcolm parses the network session data and enriches it with additional lookups and mappings including GeoIP mapping, hardware manufacturer lookups from [organizationally unique identifiers (docs/OUI)](http://standards-oui.ieee.org/oui/oui.txt) in MAC addresses, assigning names to [network segments](host-and-subnet-mapping.md#SegmentNaming) and [hosts](host-and-subnet-mapping.md#HostNaming) based on user-defined IP address and MAC mappings, performing [TLS fingerprinting](#https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967), and many others. + +The enriched data is stored in an [OpenSearch](https://opensearch.org/) document store in a format suitable for analysis through two intuitive interfaces: OpenSearch Dashboards, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime, a powerful tool for finding and identifying the network sessions comprising suspected security incidents. These tools can be accessed through a web browser from analyst workstations or for display in a security operations center (SOC). Logs can also optionally be forwarded on to another instance of Malcolm. + +![Malcolm Data Pipeline](./images/malcolm_data_pipeline.png) + +For smaller networks, use at home by network security enthusiasts, or in the field for incident response engagements, Malcolm can also easily be deployed locally on an ordinary consumer workstation or laptop. Malcolm can process local artifacts such as locally-generated Zeek logs, locally-captured PCAP files, and PCAP files collected offline without the use of a dedicated sensor appliance. + +## Table of Contents + +* [Automated Build Workflows Status](#BuildBadges) +* [Quick start](quickstart.md#QuickStart) + * [Getting Malcolm](quickstart.md#GetMalcolm) + * [User interface](quickstart.md#UserInterfaceURLs) +* [Overview](#Overview) +* [Components](components.md#Components) +* [Supported Protocols](protocols.md#Protocols) +* [Development](development.md#Development) + * [Building from source](development.md#Build) + * [Pre-Packaged installation files](development.md#Packager) +* [Preparing your system](#Preparing) + * [Recommended system requirements](#SystemRequirements) + * [System configuration and tuning](malcolm-config.md#ConfigAndTuning) + * [`docker-compose.yml` parameters](malcolm-config.md#DockerComposeYml) + * [Linux host system configuration](#HostSystemConfigLinux) + * [macOS host system configuration](#HostSystemConfigMac) + * [Windows host system configuration](#HostSystemConfigWindows) +* [Running Malcolm](#Running) + * [OpenSearch instances](opensearch-instances.md#OpenSearchInstance) + * [Authentication and authorization for remote OpenSearch clusters](#OpenSearchAuth) + * [Configure authentication](authsetup.md#AuthSetup) + * [Local account management](authsetup.md#AuthBasicAccountManagement) + * [Lightweight Directory Access Protocol (LDAP) authentication](authsetup.md#AuthLDAP) + - [LDAP connection security](#AuthLDAPSecurity) + * [TLS certificates](authsetup.md#TLSCerts) + * [Starting Malcolm](running.md#Starting) + * [Stopping and restarting Malcolm](running.md#StopAndRestart) + * [Clearing Malcolm's data](running.md#Wipe) + * [Temporary read-only interface](#ReadOnlyUI) +* [Capture file and log archive upload](upload.md#Upload) + - [Tagging](upload.md#Tagging) + - [Processing uploaded PCAPs with Zeek and Suricata](upload.md#UploadPCAPProcessors) +* [Live analysis](#LiveAnalysis) + * [Using a network sensor appliance](live-analysis.md#Hedgehog) + * [Monitoring local network interfaces](#LocalPCAP) + * [Manually forwarding logs from an external source](#ExternalForward) +* [Arkime](#Arkime) + * [Zeek log integration](arkime.md#ArkimeZeek) + - [Correlating Zeek logs and Arkime sessions](arkime.md#ZeekArkimeFlowCorrelation) + * [Help](#ArkimeHelp) + * [Sessions](arkime.md#ArkimeSessions) + * [PCAP Export](#ArkimePCAPExport) + * [SPIView](#ArkimeSPIView) + * [SPIGraph](arkime.md#ArkimeSPIGraph) + * [Connections](#ArkimeConnections) + * [Hunt](arkime.md#ArkimeHunt) + * [Statistics](#ArkimeStats) + * [Settings](#ArkimeSettings) +* [OpenSearch Dashboards](dashboards.md#Dashboards) + * [Discover](#Discover) + - [Screenshots](#DiscoverGallery) + * [Visualizations and dashboards](dashboards.md#DashboardsVisualizations) + - [Prebuilt visualizations and dashboards](#PrebuiltVisualizations) + - [Screenshots](#PrebuiltVisualizationsGallery) + - [Building your own visualizations and dashboards](dashboards.md#BuildDashboard) + + [Screenshots](#NewVisualizationsGallery) +* [Search Queries in Arkime and OpenSearch](queries-cheat-sheet.md#SearchCheatSheet) +* [Other Malcolm features](#MalcolmFeatures) + - [Automatic file extraction and scanning](file-scanning.md#ZeekFileExtraction) + - [Automatic host and subnet name assignment](host-and-subnet-mapping.md#HostAndSubnetNaming) + + [IP/MAC address to hostname mapping via `host-map.txt`](host-and-subnet-mapping.md#HostNaming) + + [CIDR subnet to network segment name mapping via `cidr-map.txt`](host-and-subnet-mapping.md#SegmentNaming) + + [Defining hostname and CIDR subnet names interface](host-and-subnet-mapping.md#NameMapUI) + + [Applying mapping changes](#ApplyMapping) + - [OpenSearch index management](index-management.md#IndexManagement) + - [Event severity scoring](severity.md#Severity) + + [Customizing event severity scoring](#SeverityConfig) + - [Zeek Intelligence Framework](zeek-intel.md#ZeekIntel) + + [STIX™ and TAXII™](zeek-intel.md#ZeekIntelSTIX) + + [MISP](zeek-intel.md#ZeekIntelMISP) + - [Anomaly Detection](#AnomalyDetection) + - [Alerting](alerting.md#Alerting) + + [Email Sender Accounts](#AlertingEmail) + - ["Best Guess" Fingerprinting for ICS Protocols](ics-best-guess.md#ICSBestGuess) + - [Asset Management with NetBox](netbox.md#NetBox) + - [CyberChef](#CyberChef) + - [API](api.md#API) + + [Examples](api-examples.md#APIExamples) +* [Ingesting Third-party Logs](#ThirdPartyLogs) +* [Malcolm installer ISO](malcolm-iso.md#ISO) + * [Installation](malcolm-iso.md#ISOInstallation) + * [Generating the ISO](#ISOBuild) + * [Setup](#ISOSetup) + * [Time synchronization](#ConfigTime) + * [Hardening](hardening.md#Hardening) + * [Compliance Exceptions](#ComplianceExceptions) +* [Installation example using Ubuntu 22.04 LTS](ubuntu-install-example.md#InstallationExample) +* [Upgrading Malcolm](malcolm-upgrade.md#UpgradePlan) +* [Modifying or Contributing to Malcolm](#Contributing) \ No newline at end of file