Feature/boefjes to oci images #2709

Donnype · 2024-03-22T16:56:23Z

Changes

This adds Docker (OCI) image builder setups for both python-native boefjes and boefjes that were using docker containers already. As an example for this first version, only ~~dns-records and~~ dns-sec were changed.

To get it working for the dnssec image, I had to change the boefje_meta argument to be a dict since else we'd be adding pydantic there and a job_models file. This can be a point of discussion.

I also added a step in the make build that we run during installations. We now also do make -C boefjes images that builds the two images. There the target also provides examples of how to manually run the docker build commands with the right arguments and tags.

~~One issue here is that some alpine images only have python 3.5 available which messes a bit with some of the APIs in the standard lib. Hence the ignoring the dnssec main.py for pre-commit.~~ We'll try to migrate to new python3.11-slim images where possible. In this case, it was possible.

I used boefje.Dockerfile per the Dockerfile naming convention and the fact that potentially normalizer Dockerfiles could perhaps also live in the same directory with the name normalizer.Dockerfile for instance.

Follow up: do we still care about the group and user ids? We do not pass them anyways currently and the main reason for implementing them was specific requirements for the apps themselves, if I recall correctly. No, removed.

Issue link

Closes #2443

Demo & QA notes

This feature should not change the current flow or setup, but please verify that both the dns-records and dns-sec boefje work properly on a clean install (i.e. after make kat or make reset, which should build the images). Perhaps a rough notion of the new performance would also be nice as the risk here is that the new setup will induce significant overhead to smaller boefjes.

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
I have written unit tests for the changes or fixes I made.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
I have made corresponding changes to the documentation, if necessary.
I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Change the signature to a dictionary for containerized boefjes.

…cess triggered through `make kat`

…e-types

boefjes/Makefile

boefjes/boefjes/docker_boefjes_runner.py

boefjes/boefjes/plugins/kat_dnssec/boefje.Dockerfile

boefjes/boefjes/plugins/kat_dns/main.py

boefjes/images/base.Dockerfile

boefjes/images/boefje_entrypoint.sh

boefjes/boefjes/plugins/kat_dnssec/boefje.Dockerfile

Use the `nonroot` user in the boefje images. Explanation for setting the DOCKER_DEFAULT_PLATFORM in the boefje Makefile

Handle the boefje api requests in Python as well, using httpx Use the docker_adapter as entrypoint, making the entrypoint redundant.

Handle the boefje api requests in Python as well, using httpx Use the docker_adapter as entrypoint, making the entrypoint redundant

stephanie0x00 · 2024-03-27T10:52:03Z

Checklist for QA:

I have checked out this branch, and successfully ran a fresh make reset.
I confirmed that there are no unintended functional regressions in this branch:
- I have managed to pass the onboarding flow
- Objects and Findings are created properly
- Tasks are created and completed properly
I confirmed that the PR's advertised feature or hotfix works as intended.

What works:

DNSSec boefje works and appear no error messages in the logs. After Review is done it can be moved to Ready to merge.

I followed the following approach while timing: I let the DNSrecords and DNSzone boefjes run and finish, then enable the dnssec boefje and start the stopwatch. Data is against mispo.es with L1 and timed until the KAT-NO-DNSSEC Finding Type is completed under the Normalizer tab:

Current main it takes the times below to finish. It takes about ~30s for the first dnssec task to show up.
- 1st time:- ~40 seconds
- 2nd time: ~23 seconds
- 3rd time: ~45 seconds
- 4th time: ~47 seconds
This branch to run. It takes about ~40seconds for the first dnssec related task to show up in the list:
- 1st time: 1m15seconds
- 2nd time: ~40 seconds
- 3rd time: ~55 seconds
- 4th time: 1m05s.

What doesn't work:

n/a

Bug or feature?:

n/a

Signed-off-by: Donny Peeters <[email protected]>

underdarknl · 2024-03-29T10:55:58Z

boefjes/boefjes/plugins/kat_dnssec/main.py

    domain = input_["name"]

-    client = docker.from_env()
-
    # check for string pollution in domain. This check will fail if anything other characters than a-zA-Z0-9_.- are
    # present in the hostname
    if not re.search(r"^[\w.]+[\w\-.]+$", domain.lower()):


Have we checked punicode domainname compatibility here?

* main: (51 commits) Fix static files for container images/Debian packages when DEBUG is on (#2742) OOI selection at Aggregate report does not remember changed selection (#2619) fix schema errors on empty / missing schemas (#2744) Updated `phonenumbers` and `django-phonenumber-field` (#2757) Remove octopoes coverage workflow (#2755) Bump actions/configure-pages from 4 to 5 (#2745) Add xtdb-cli tool to Octopoes (#2733) Dont report vulnerabilities without version info of the software for snyk (#2730) Feature/boefjes to oci images (#2709) Query non-reference fields and subclass-specific fields through path queries (#2662) Fix in System Specific (#2732) Plugins overview in appendix not showing any plugins (#2694) Feat stepper design v2 (#2704) Undo project-directory in Rocky (#2734) Remove Docker Compose: "version" (#2718) Upgrade `pre-commit` hooks (#2729) Fix #1739 (#2705) Improve generate report (#2633) Fix critical vulnerability counter (#2712) Fix pdf alignment (#2674) ...

Donnype added 5 commits March 22, 2024 14:52

Create base.Dockerfile can build the Python-only boefjes

1150237

Create a setup for dnssec and remove some debugging tooling.

3a9986d

Change the signature to a dictionary for containerized boefjes.

Add building images from source as a part of the kat installation pro…

2f4026d

…cess triggered through `make kat`

Fix race condition with duplicate raw files using the same set of mim…

fb882d3

…e-types

Style

f43bbcc

Donnype requested a review from a team as a code owner March 22, 2024 16:56

dekkers reviewed Mar 25, 2024

View reviewed changes

Disable the dns-records image for now.

8e543a8

Use the `nonroot` user in the boefje images. Explanation for setting the DOCKER_DEFAULT_PLATFORM in the boefje Makefile

Donnype mentioned this pull request Mar 26, 2024

More refinement in the docker image runners and the error logging to bytes #2728

Open

Donnype added 3 commits March 26, 2024 17:35

Update images to use python3.11-alpine

cec7912

Handle the boefje api requests in Python as well, using httpx Use the docker_adapter as entrypoint, making the entrypoint redundant.

Update images to use python3.11-alpine

6685f7d

Handle the boefje api requests in Python as well, using httpx Use the docker_adapter as entrypoint, making the entrypoint redundant

Style

b6a30df

Donnype requested a review from dekkers March 26, 2024 17:55

Donnype and others added 4 commits March 28, 2024 10:20

Merge branch 'main' into feature/boefjes-to-oci-images

829aa91

Merge branch 'main' into feature/boefjes-to-oci-images

143a6b6

Remove redundant ignore clause for ruff

29d0cea

Signed-off-by: Donny Peeters <[email protected]>

Merge branch 'main' into feature/boefjes-to-oci-images

4bd49e2

underdarknl reviewed Mar 29, 2024

View reviewed changes

underdarknl approved these changes Mar 29, 2024

View reviewed changes

underdarknl merged commit 33ef390 into main Mar 29, 2024
10 checks passed

underdarknl deleted the feature/boefjes-to-oci-images branch March 29, 2024 10:57

Donnype mentioned this pull request Apr 18, 2024

Docs update for containerized boefjes #2756

Closed

stephanie0x00 mentioned this pull request May 15, 2024

Update PR template #2950

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/boefjes to oci images #2709

Feature/boefjes to oci images #2709

Donnype commented Mar 22, 2024 •

edited

Loading

stephanie0x00 commented Mar 27, 2024

underdarknl Mar 29, 2024

Feature/boefjes to oci images #2709

Feature/boefjes to oci images #2709

Conversation

Donnype commented Mar 22, 2024 • edited Loading

Changes

Issue link

Demo & QA notes

Code Checklist

Communication

Checklist for code reviewers:

Checklist for QA:

stephanie0x00 commented Mar 27, 2024

Checklist for QA:

What works:

What doesn't work:

Bug or feature?:

underdarknl Mar 29, 2024

Choose a reason for hiding this comment

Donnype commented Mar 22, 2024 •

edited

Loading