Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding ScanSave scanner #362

Merged
merged 7 commits into from
Apr 18, 2023
Merged

Adding ScanSave scanner #362

merged 7 commits into from
Apr 18, 2023

Conversation

keiche
Copy link
Contributor

@keiche keiche commented Apr 10, 2023

Describe the change
This change adds ScanSave which exposes the raw binary contents of a file as a compressed and encoded string.

Closes #358

The raw contents of a file may contain a large amount of data which is why this scanner is disabled by default in the backend.yaml configuration. To help reduce the size of the result a compression algorithm can be used on the data (gzip (default), bzip2, or lzma). Also, to ensure character set compatibility with the JSON output, the binary data is encoded (base64 (default) or base85). Additional processing on this data would then be required to save the data to a file. Multiple compression and encoding algorithms are available to allow users options for compatibility, post-processing requirements, and result size.

Example post-processing

$ i=0; for line in $(jq -r '.scan.save.file' strelka-oneshot.log); do echo $line | base64 -d | gunzip > file_${i}; ((i++)); done

Future development could add functionality to Strelka frontend commands to decode, decompress, and save the data to the local file system.

Side note: The pre-commit hook version of isort was updated due to a bug between old versions of isort and the latest poetry version.

Describe testing procedures

  1. The module was tested via pytest using docker-compose -f build/docker-compose.yaml build
      => => # tests/test_scan_save.py ....
    
  2. Pre-commit hooks were used to perform linting and syntax validation
    black....................................................................Passed
    check yaml...............................................................Passed
    fix end of files.........................................................Passed
    trim trailing whitespace.................................................Passed
    check for added large files..............................................Passed
    check for merge conflicts................................................Passed
    mixed line ending........................................................Passed
    don't commit to branch...................................................Passed
    flake8...................................................................Passed
    isort....................................................................Passed
    
  3. Ran the docker-compose containers locally and ran samples through using strelka-oneshot
    1. Validated the output to ensure it matched what was expected
    2. Decoded and decompressed the save.file output to ensure the original file was able to be reconstructed

Sample output

{
  "compression": "gzip",
  "elapsed": 7.7e-05,
  "encoding": "base64",
  "file": "H4sIABv5M2QC/ytJLS7hAgDGNbk7BQAAAA=="
}

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of and tested my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings

@phutelmyer phutelmyer added the enhancement New feature or request label Apr 17, 2023
@phutelmyer
Copy link
Contributor

Everything looks great here @keiche. An interesting idea for a scanner that I never considered.
I'll review and test with some larger files.
Thanks for putting this together.

@phutelmyer
Copy link
Contributor

Works as advertised. Clear output and well documented.
No issues with it, but will definitely keep it turned off by default as provided.
Great idea - thank you for submitting, @keiche!

Approved and Merged.

@phutelmyer phutelmyer merged commit 7a9ed42 into target:master Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[REQUEST] Access raw data of parsed sub-files from client applications
2 participants