Give me the reports generated between 10-10-2024 and 21-10-2024" implies fetching the history of every report.

If I understand correctly, only if the result changes between the two valid_times which can be queried.

Questions:

• I suppose we can update the input_ooi field when reusing the report and see what were the inputs over time?
• A report recipe does not have an organization code/name/etc. Is it safe to assume that since it can only live in 1 XTDB database that it refers to only 1 organization?
• Are these the fields we should filter on to get the right Report to update? Writing this down I just remembered once drawing the conclusion that the "ReportRecipe -> Report" relation would be one-to-one. Looking at the code I should correct that and say that this only holds for "ReportRecipe -> ParentReport" when we have multiple reports. So the steps to update the reports in place would be:
  1. Determine if there are multiple reports by looking at the report_data
  2. If there are multiple, find the parent report that points to this ReportRecipe, else search for a single report that points to this recipe
  3. Find all subreports of this parent report if we have a parent
  4. Based on the report type and input ooi, try to find the subreports that already exist to update those in-place as well
• Or should we not update subreports in-place? It might be strange that the amount of updates vs. creates happening during reporting can depend on the input oois that result from the input query.

Im looking at this in a different Light I think:

An aggregate report is the result of the underlying reports being combined. We currently have the relation the wrong way round I think..
A recipe holds a list of Input OOI's, or a query (resulting in a live list of input OOI's possibly changing over time).
The reporting job derived from the recipe's schedule on a given moment in time produces a set of Asset-reports (input-ooi * report-type). those reports should get a reference to the recipe that was used to create them, and a reference to the job. (no news there I suppose?).
The asset-reports should get a (I think) deterministic OOI-ID, which encodes the input-ooi, the used report type, and any settings that change the actual data being stored (settings that only apply to the rendered version, but not the json dont need to be part of the hash/ooi-ID. [1]
The aggregate report should now hold the underlying report-ooi's OOI-ID's and their valid-time at the time of combination as its input-OOI list. It could also hold a reference to the recipe or job that triggered the aggregation to run.
If a given OOI is no longer available for the asset-reporting, we could include the most recent report or decide to not include the Asset-report anymore. The underlying report might have been generated yesterday, or might have been manually uploaded by a user, in any case we do or don't include it in the input-ooi's for the given agregate-report.

1: This means, multiple recipe's and reporting jobs might write into the same report-OOI at different intervals. This would mean we are making more 'snapshots' of the data at those points.

• We have here a scenario where we select 2 OOIs and want to make 2 reports.
• We want an Aggregate Report
• So we have in total 4 sub/child reports. We create a Report Recipe with:

Parent report: Aggregate Organisation Report
assets: mispo.es, minvws.nl
subreports: DNS Report, Findings Reports

flowchart TD
    A[Report Recipe]
    B[Scheduler]
    C[Octopoes]
    D[Rocky Worker]
    E[Bytes holds Report Data]
    F(DNS Report for mispo.es)
    G(Findings Report for mispo.es)
    H(DNS Report for minvws.nl)
    I(Findings Report for minvws.nl)
    J[Aggregate Data]
    K[Aggregate Report]
    A -- create recipe OOI --> C
    A -- schedule --> B
    B --> D
    D --> C
    C -- Report OOI --> F
    C -- Report OOI --> G
    C -- Report OOI --> H
    C -- Report OOI --> I
    F --> E
    G --> E
    H --> E
    I --> E
    E --> J
    J --> K

• What I see here is that an Aggregate Report is basically a way of formatting the data, we aggregate the data, make summaries.
• So it make sense that an Aggregate Report holds a list of ReportData or we have to fetch for each report OOI the bytes data

We can maybe use ReportData which is already an OOI. ReportData is the evidence and what we are comparing.

class ReportData(OOI):
    object_type: Literal["ReportData"] = "ReportData"
    organization_code: str
    organization_name: str
    organization_tags: list[str]
    data: dict

    _natural_key_attrs = ["organization_code"]

    @classmethod
    def format_reference_human_readable(cls, reference: Reference) -> str:
        return f"Report data of organization {reference.tokenized.organization_code}"

What really is updating is the data it generates, which means that we can also have an edited Report data. We can compare ReportData's or we can aggregate report datas, or we can show them separately in the case of an Concat report.

report data matches template is what we see in a report.

So what if we keep track of the data and also keep updating the OOIs

I would remove an aggregate OOI so parent reports does not really exists, it is a way of displaying the underlying reports.

I would remove an aggregate OOI so parent reports does not really exists, it is a way of displaying the underlying reports.

@Rieven If we do this, we still need a field in the ReportRecipe to know what kind of report should be created.

I would remove an aggregate OOI so parent reports does not really exists, it is a way of displaying the underlying reports.

@Rieven If we do this, we still need a field in the ReportRecipe to know what kind of report should be created.

We already have those fields, we have the report types and the input oois or live set.

We already have those fields, we have the report types and the input oois or live set.

@Rieven
Currently we use the field parent_report_type in the ReportRecipe to tell if it's an aggregate report or not. The fields report_types and input_recipe (with input_oois/query) just tells if it's a single report or a concatinated report. So what I wanted to say is that even if we remove the parent Report OOI, we should make sure not to remove the parent_report_type in the ReportRecipe. Otherwise we would never know if it's an aggregate report or not.

Besides that, I think the problem with removing the parent Report OOI entirely would be opening the aggregate report (on the history page and scheduled reports page). Where would you link to when clicking on the generated report?

Possible future improvement. Store the various report-export-formats in bytes at runtime of the job instead of storing only the json. This would avoid concatting / producing these rich output formats at runtime once the user clicks on the html/pdf links.
Eg:

class BaseReport(OOI):
    data_raw_id: str
    data_raw_html_id: str | None = None
    data_raw_pdf_id: str | None = None
    data_raw_html_concat_id: str | None = None
    data_raw_html_concat_zip_id: str | None = None
    data_raw_pdf_concat_id: str | None = None
    data_raw_pdf_concat_zip: str | None = None