Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zeek-Logs get reingested after container restart #101

Closed
dapfeffer opened this issue Jun 7, 2022 · 2 comments
Closed

Zeek-Logs get reingested after container restart #101

dapfeffer opened this issue Jun 7, 2022 · 2 comments
Assignees
@mmguero
Labels
bug Something isn't working upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek

Comments

@dapfeffer
Copy link

After installation of a Malcolm instance via ISO-Image I started capturing local traffic. All the traffic is captured, analyzed and ingested into Opensearch as intended. But after rebooting the Malcolm-Stack or the whole machine any captured PCAP-Data gets reingested and Zeek-Logs are doubled in Opensearch. Anytime the container is restarted, the files are ingested again.

I guess whatever component is watching for new PCAPs to analyze should somehow persist already processed files so that even after container-reboot the data won't get ingested again.
@mmguero
Copy link
Collaborator

mmguero commented Jun 7, 2022

Thanks for logging the issue. While there are some docker-compose values to explicitly prevent that from happening (EXTRACTED_FILE_IGNORE_EXISTING and PCAP_PIPELINE_IGNORE_PREEXISTING) which you might be able to use as a stopgap in the meantime while I investigate the issue, it still should not be duplicating your data during processing. I'll look at this.

@mmguero mmguero self-assigned this Jun 7, 2022
@mmguero mmguero added bug Something isn't working upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek labels Jun 7, 2022
@mmguero mmguero added this to Malcolm Jun 7, 2022
@mmguero mmguero moved this to Triage in Malcolm Jun 30, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 5, 2022
@mmguero
Should fix idaholab#101 which was causing PCAPs to be re-ingested upo…
f23e401 
…n restart as the query to the arkime PCAP files index needed to be updated for newer versions of opensearch
@mmguero
Copy link
Collaborator

mmguero commented Jul 5, 2022

Fixed for v6.1.0

@mmguero mmguero closed this as completed Jul 5, 2022
Repository owner moved this from Triage to Done in Malcolm Jul 5, 2022
This was referenced Jul 12, 2022
Merged
Merged
@mmguero mmguero moved this from Done to Released in Malcolm Jul 13, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jul 28, 2022
@mmguero
Should fix idaholab#101 which was causing PCAPs to be re-ingested upo…
5f48756 
…n restart as the query to the arkime PCAP files index needed to be updated for newer versions of opensearch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
bug Something isn't working upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek
Projects
Malcolm
Status: Released
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mmguero @dapfeffer