Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v5.2.3 development #187

Merged
merged 11 commits into from
Jan 31, 2022
Merged

v5.2.3 development #187

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
a981c48
bump version for v5.2.3 development
mmguero Jan 25, 2022
3cdc617
more thorough settings for read-only mode to set indexes to read-only
mmguero Jan 26, 2022
b5b0849
better documentation for read-only mode
mmguero Jan 26, 2022
8018507
bump Zeek to v4.2.0 (https://github.com/zeek/zeek/releases/tag/v4.2.0)
mmguero Jan 26, 2022
1f6c294
for idaholab/Malcolm#73, change the path for tcpdump from /usr/sbin t…
mmguero Jan 26, 2022
e38e14e
bump arkime to 3.3.1
mmguero Jan 26, 2022
a26d85a
handle zeek.files.ftime field added by the spicy/zip analyzer
mmguero Jan 26, 2022
1bcfe65
fix idaholab/Malcolm#72 improving parsing into ECS file.path, file.di…
mmguero Jan 27, 2022
bd71f4d
improved files dashboard
mmguero Jan 27, 2022
92b7bf7
improved files dashboard
mmguero Jan 27, 2022
a660431
point at upstream for 0xl3x1/zeek-EternalSafety now that 0xl3x1/zeek-…
mmguero Jan 31, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dockerfiles/arkime.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ FROM debian:bullseye-slim AS build

ENV DEBIAN_FRONTEND noninteractive

ENV ARKIME_VERSION "3.3.0"
ENV ARKIME_VERSION "3.3.1"
ENV ARKIMEDIR "/opt/arkime"
ENV ARKIME_URL "https://github.com/arkime/arkime.git"
ENV ARKIME_LOCALELASTICSEARCH no
Expand Down
1 change: 1 addition & 0 deletions Dockerfiles/dashboards-helper.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ ADD dashboards/malcolm_template.json /data/malcolm_template.json
ADD shared/bin/docker-uid-gid-setup.sh /usr/local/bin/
ADD shared/bin/opensearch_status.sh /data/
ADD shared/bin/opensearch_index_size_prune.py /data/
ADD shared/bin/opensearch_read_only.py /data/

RUN apk --no-cache add bash python3 py3-pip curl procps psmisc npm shadow jq && \
npm install -g http-server && \
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/zeek.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ ENV PUSER_PRIV_DROP true

# for download and install
ARG ZEEK_LTS=
ARG ZEEK_VERSION=4.1.1-0
ARG ZEEK_VERSION=4.2.0-0
ARG SPICY_VERSION=1.3.0

ENV ZEEK_LTS $ZEEK_LTS
Expand Down
80 changes: 45 additions & 35 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,22 +172,22 @@ You can then observe that the images have been retrieved by running `docker imag
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/api 5.2.2 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.2 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.2 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.2 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.2 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.2 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.2 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.2 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.2 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.2 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.2 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.2 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.2 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.2 xxxxxxxxxxxx 2 days ago 938MB
malcolmnetsec/api 5.2.3 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.3 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.3 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.3 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.3 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.3 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.3 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.3 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.3 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.3 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.3 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.3 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.3 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.3 xxxxxxxxxxxx 2 days ago 938MB
```

#### Import from pre-packaged tarballs
Expand Down Expand Up @@ -856,13 +856,23 @@ Run `./scripts/wipe` to stop the Malcolm instance and wipe its OpenSearch databa

### <a name="ReadOnlyUI"></a>Temporary read-only interface

To temporarily set the Malcolm user interaces into a read-only configuration, run the following command from the Malcolm installation directory:
To temporarily set the Malcolm user interaces into a read-only configuration, run the following commands from the Malcolm installation directory.

First, to configure [Nginx] to disable access to the upload and other interfaces for changing Malcolm settings, and to deny HTTP methods other than `GET` and `POST`:

```
docker-compose exec nginx-proxy bash -c "cp /etc/nginx/nginx_readonly.conf /etc/nginx/nginx.conf && nginx -s reload"
```

This command must be re-run every time you restart Malcolm.
Second, to set the existing OpenSearch data store to read-only:

```
docker-compose exec dashboards-helper /data/opensearch_read_only.py -i _cluster
```

These commands must be re-run every time you restart Malcolm.

Note that after you run these commands you may see an increase of error messages in the Malcolm containers' output as various background processes will fail due to the read-only nature of the indices. Additionally, some features such as Arkime's [Hunt](#ArkimeHunt) and [building your own visualizations and dashboards](#BuildDashboard) in OpenSearch Dashboards will not function correctly in read-only mode.

## <a name="Upload"></a>Capture file and log archive upload

Expand Down Expand Up @@ -3270,7 +3280,7 @@ Building the ISO may take 30 minutes or more depending on your system. As the bu

```
…
Finished, created "/malcolm-build/malcolm-iso/malcolm-5.2.2.iso"
Finished, created "/malcolm-build/malcolm-iso/malcolm-5.2.3.iso"
…
```

Expand Down Expand Up @@ -3657,22 +3667,22 @@ Pulling zeek ... done

user@host:~/Malcolm$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
malcolmnetsec/api 5.2.2 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.2 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.2 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.2 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.2 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.2 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.2 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.2 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.2 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.2 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.2 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.2 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.2 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.2 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.2 xxxxxxxxxxxx 2 days ago 938MB
malcolmnetsec/api 5.2.3 xxxxxxxxxxxx 2 days ago 155MB
malcolmnetsec/arkime 5.2.3 xxxxxxxxxxxx 2 days ago 811MB
malcolmnetsec/dashboards 5.2.3 xxxxxxxxxxxx 2 days ago 970MB
malcolmnetsec/dashboards-helper 5.2.3 xxxxxxxxxxxx 2 days ago 154MB
malcolmnetsec/filebeat-oss 5.2.3 xxxxxxxxxxxx 2 days ago 621MB
malcolmnetsec/file-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 586MB
malcolmnetsec/file-upload 5.2.3 xxxxxxxxxxxx 2 days ago 259MB
malcolmnetsec/freq 5.2.3 xxxxxxxxxxxx 2 days ago 132MB
malcolmnetsec/htadmin 5.2.3 xxxxxxxxxxxx 2 days ago 242MB
malcolmnetsec/logstash-oss 5.2.3 xxxxxxxxxxxx 2 days ago 1.27GB
malcolmnetsec/name-map-ui 5.2.3 xxxxxxxxxxxx 2 days ago 142MB
malcolmnetsec/nginx-proxy 5.2.3 xxxxxxxxxxxx 2 days ago 117MB
malcolmnetsec/opensearch 5.2.3 xxxxxxxxxxxx 2 days ago 1.18GB
malcolmnetsec/pcap-capture 5.2.3 xxxxxxxxxxxx 2 days ago 122MB
malcolmnetsec/pcap-monitor 5.2.3 xxxxxxxxxxxx 2 days ago 214MB
malcolmnetsec/zeek 5.2.3 xxxxxxxxxxxx 2 days ago 938MB
```

Finally, we can start Malcolm. When Malcolm starts it will stream informational and debug messages to the console. If you wish, you can safely close the console or use `Ctrl+C` to stop these messages; Malcolm will continue running in the background.
Expand Down
3 changes: 2 additions & 1 deletion arkime/etc/config.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,7 @@ zeek.files.depth=db:zeek.files.depth;group:zeek_files;kind:integer;friendly:Sour
zeek.files.analyzers=db:zeek.files.analyzers;group:zeek_files;kind:termfield;friendly:Analyzer;help:Analyzer
zeek.files.mime_type=db:zeek.files.mime_type;group:zeek_files;kind:termfield;friendly:File Magic;help:File Magic
zeek.files.filename=db:zeek.files.filename;group:zeek_files;kind:termfield;friendly:Filename;help:Filename
zeek.files.ftime=db:zeek.files.ftime;group:zeek_files;kind:termfield;friendly:File Timestamp;help:File Timestamp
zeek.files.duration=db:zeek.files.duration;group:zeek_files;kind:termfield;friendly:Analysis Duration;help:Analysis Duration
zeek.files.local_orig=db:zeek.files.local_orig;group:zeek_files;kind:termfield;friendly:Local Originator;help:Local Originator
zeek.files.is_orig=db:zeek.files.is_orig;group:zeek_files;kind:termfield;friendly:Originator is Transmitter;help:Originator is Transmitter
Expand Down Expand Up @@ -1377,7 +1378,7 @@ zeek_ecat_foe_info=require:zeek.ecat_foe_info;title:Zeek ecat_foe_info.log;field
zeek_ecat_soe_info=require:zeek.ecat_soe_info;title:Zeek ecat_soe_info.log;fields:zeek.ecat_soe_info.opcode,zeek.ecat_soe_info.incomplete,zeek.ecat_soe_info.error,zeek.ecat_soe_info.drive_num,zeek.ecat_soe_info.element,zeek.ecat_soe_info.index
zeek_ecat_arp_info=require:zeek.ecat_arp_info;title:Zeek ecat_arp_info.log;fields:zeek.ecat_arp_info.arp_type,zeek.ecat_arp_info.orig_proto_addr,zeek.ecat_arp_info.orig_hw_addr,zeek.ecat_arp_info.resp_proto_addr,zeek.ecat_arp_info.resp_hw_addr
zeek_enip=require:zeek.enip;title:Zeek enip.log;fields:zeek.enip.enip_command,zeek.enip.enip_command_code,zeek.enip.length,zeek.enip.session_handle,zeek.enip.enip_status,zeek.enip.sender_context,zeek.enip.options
zeek_files=require:zeek.files;title:Zeek files.log;fields:zeek.files.tx_hosts,zeek.files.rx_hosts,zeek.files.conn_uids,zeek.files.source,zeek.files.depth,zeek.files.analyzers,zeek.files.mime_type,zeek.files.filename,zeek.files.duration,zeek.files.local_orig,zeek.files.is_orig,zeek.files.seen_bytes,zeek.files.total_bytes,zeek.files.missing_bytes,zeek.files.overflow_bytes,zeek.files.timedout,zeek.files.parent_fuid,zeek.files.md5,zeek.files.sha1,zeek.files.sha256,zeek.files.extracted,zeek.files.extracted_cutoff,zeek.files.extracted_size
zeek_files=require:zeek.files;title:Zeek files.log;fields:zeek.files.tx_hosts,zeek.files.rx_hosts,zeek.files.conn_uids,zeek.files.source,zeek.files.depth,zeek.files.analyzers,zeek.files.mime_type,zeek.files.filename,zeek.files.ftime,zeek.files.duration,zeek.files.local_orig,zeek.files.is_orig,zeek.files.seen_bytes,zeek.files.total_bytes,zeek.files.missing_bytes,zeek.files.overflow_bytes,zeek.files.timedout,zeek.files.parent_fuid,zeek.files.md5,zeek.files.sha1,zeek.files.sha256,zeek.files.extracted,zeek.files.extracted_cutoff,zeek.files.extracted_size
zeek_ftp=require:zeek.ftp;title:Zeek ftp.log;fields:zeek.ftp.command,zeek.ftp.arg,zeek.ftp.mime_type,zeek.ftp.file_size,zeek.ftp.reply_code,zeek.ftp.reply_msg,zeek.ftp.data_channel_passive,zeek.ftp.data_channel_orig_h,zeek.ftp.data_channel_resp_h,zeek.ftp.data_channel_resp_p
zeek_gquic=require:zeek.gquic;title:Zeek gquic.log;fields:zeek.gquic.version,zeek.gquic.server_name,zeek.gquic.user_agent,zeek.gquic.tag_count,zeek.gquic.cyu,zeek.gquic.cyutags
zeek_http=require:zeek.http;title:Zeek http.log;fields:zeek.http.trans_depth,zeek.http.method,zeek.http.host,zeek.http.uri,zeek.http.origin,zeek.http.post_password_plain,zeek.http.post_username,zeek.http.referrer,zeek.http.version,zeek.http.user_agent,zeek.http.request_body_len,zeek.http.response_body_len,zeek.http.status_code,zeek.http.status_msg,zeek.http.info_code,zeek.http.info_msg,zeek.http.tags,zeek.http.proxied,zeek.http.orig_fuids,zeek.http.orig_filenames,zeek.http.orig_mime_types,zeek.http.resp_fuids,zeek.http.resp_filenames,zeek.http.resp_mime_types
Expand Down
76 changes: 0 additions & 76 deletions arkime/patch/viewer_330_large_or_xor_packet_fix_f13e9366.patch
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions arkime/wise/source.zeeklogs.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ class MalcolmSource extends WISESource {
"zeek.files.extracted_cutoff",
"zeek.files.extracted_size",
"zeek.files.filename",
"zeek.files.ftime",
"zeek.files.is_orig",
"zeek.files.local_orig",
"zeek.files.md5",
Expand Down
Loading