Search for an Catena-X asset takes longer after searching non-existing assets multiple times.

[krenzalb]

Hello,
We have noticed in our dpp-backend (we do not use the frontend) that the search for a Catena-X ID (consisting of manufacturer part ID and part instance ID) takes longer after a few failed attempts (e.g. because the asset does not exist).

Our search runs in the following steps.

• /api/contract/create with the manufacturer part ID of the asset
• /api/contract/search with the process ID and the part instance ID of the asset
• /api/contract/agree to give consent to the contract
• /api/contract/status/<process-id> wait until data transfer is ready
• /api/data retrieve the data of the asset.

Normally the whole process takes about 30 seconds. However, if we search several times for an ID that does not exist, the overall process will eventually take more then 60 seconds (causing a Gateway timeout error).
What we see is that the dpp-backend does not clean up the old process ID folders in the tmp/process directory. It also creates multiple subfolder in the data/process folder. When we delete the content of these folders the issue is resolved. Searching for an asset now takes only 30 seconds.

The question now is: What could be the reason for this increase in search time? Is it intentional that data in these directories is not deleted?

[saudkhan116]

Hi @krenzalb,

As far as I know, a temporary storage is implemented to persist the previously requested data in processes, which is valid for 12 hours (default) here.

@matbmoser, could you please explain a reason of increasing search time?

Thanks!

[matbmoser]

We have a timeout for waiting for the response of the digital twin registry.

Its an asnyc request:

sequenceDiagram
    participant DPPF as Digital Product Pass Frontend
    participant DPP as Digital Product Pass Backend
    participant EDCC as EDC Controlplane
    participant EDCD as EDC Dataplane
    participant DTR as Digital Twin Registry

    DPPF ->> DPP: Call /api/contract/search with "processId" + "partInstanceId" 

    rect rgba(100, 100, 100, 0.2) 
    note over DPP, EDCC: Thread1: **/api/contract/search** API
    DPP ->> EDCC: Reuse DTR negotiation, give me authorization in this callback endpoint api /endpoint/{processId}/{dtrKey}
    EDCC -->> DPP: Ok I will give you in another api
    DPP ->> DPP: Wait until "DT-READY" is available in status (wait **<<SEARCH TIMEOUT>>**) 
    end

    rect rgba(30, 60, 100, 0.3) 
    note over EDCC, DPP: START Thread2: **/endpoint/{processId}/{dtrKey}** API 
    EDCC ->> DPP: Here is the EDR with the Dataplane Endpoint and the JWT authorization token
    DPP -->> EDCC: ACK

    DPP ->> EDCD: Search DT, Call GET {DataplaneUrl}/lookup/shells?assetIds={base64Encoded with partInstanceId} with the JWT authorization in Header
    EDCD -->> DTR: Proxy Request
    DTR -->> EDCD: Response with Digital Twin Ids (if available) 
    EDCD --> DPP: Proxy Response with Ids

    DPP ->> DPP: Select Digital Twin Id (should be unique, otherwise the first)

    DPP ->> EDCD: Get DT, Call GET {DataplaneUrl}/shell-descriptors/{base64Encoded Digital Twin Id} with the JWT authorization in Header
    EDCD -->> DTR: Proxy Request
    DTR -->> EDCD: Response with Digital Twin
    EDCD -->> DPP: Proxy Response with Digital Twin

    DPP ->> DPP: Store Digital Twin in process Local Storage, and set status "DT-READY" 
    note over EDCC, DPP: END Thread2: **/endpoint/{processId}/{dtrKey}** API 
    end

    rect rgba(100, 100, 100, 0.2) 
    note over DPP, EDCC: Thread1: **/api/contract/search** API

    DPP ->> DPP: Detect "DT-Ready" and get the Digital Twin (DTR) from the process Local Storage
    DPP ->> DPP: Search for submodel in DT in the configured order "Digital Product Pass -> Battery Pass -> Transmission Pass"
    DPP ->> EDCC: Call get catalog to the indicated edc controlplane and with filter for assetId indicated in the submodel
    EDCC -->> DPP: Catalog with contracts (asset+policy)
    DPP ->> DPP: If found, filter assets with valid policies (from config)
    DPP ->> DPP: Store valid contracts in process Local Storage & generate unique signing token
    DPP ->> DPPF: Send unique signing token, contracts (with assets+policies) and "processId"
    note over DPP, EDCC: END Thread1: **/api/contract/search** API
    end

Loading

If the feature #478 could be used to remove the callback, and to make the thread1 be able to request by himself the token.
Then like this it would look:

sequenceDiagram
    participant DPPF as Digital Product Pass Frontend
    participant DPP as Digital Product Pass Backend
    participant EDCC as EDC Controlplane
    participant EDCD as EDC Dataplane
    participant DTR as Digital Twin Registry

    DPPF ->> DPP: Call /api/contract/search with "processId" + "partInstanceId" 

    rect rgba(100, 100, 100, 0.2) 
    note over DPP, EDCC: Thread1: **/api/contract/search** API
    DPP ->> EDCC: Reuse DTR negotiation {{with transferId}}, give me authorization
    EDCC ->> DPP: Here is the EDR with the Dataplane Endpoint and the JWT authorization token

    DPP ->> EDCD: Search DT, Call GET {DataplaneUrl}/lookup/shells?assetIds={base64Encoded with partInstanceId} with the JWT authorization in Header
    EDCD -->> DTR: Proxy Request
    DTR -->> EDCD: Response with Digital Twin Ids (if available) 
    EDCD --> DPP: Proxy Response with Ids

    DPP ->> DPP: Select Digital Twin Id (should be unique, otherwise the first)

    DPP ->> EDCD: Get DT, Call GET {DataplaneUrl}/shell-descriptors/{base64Encoded Digital Twin Id} with the JWT authorization in Header
    EDCD -->> DTR: Proxy Request
    DTR -->> EDCD: Response with Digital Twin
    EDCD -->> DPP: Proxy Response with Digital Twin
    DPP ->> DPP: Search for submodel in DT in the configured order "Digital Product Pass -> Battery Pass -> Transmission Pass"
    DPP ->> EDCC: Call get catalog to the indicated edc controlplane and with filter for assetId indicated in the submodel
    EDCC -->> DPP: Catalog with contracts (asset+policy)
    DPP ->> DPP: If found, filter assets with valid policies (from config)
    DPP ->> DPP: Store valid contracts in process Local Storage & generate unique signing token
    DPP ->> DPPF: Send unique signing token, contracts (with assets+policies) and "processId"
    note over DPP, EDCC: END Thread1: **/api/contract/search** API
    end

Loading

More details in here:

https://raw.githubusercontent.com/eclipse-tractusx/digital-product-pass/refs/heads/main/docs/architecture/media/dataRetrieval/searchApiFlow.drawio.svg