AIP-81 Implement Missing Bulk Insert API Endpoints will be used from CLI Commands

[bugraoz93]

Description

AIP-81

To enhance the API functionality, this umbrella ticket organizes the development of missing endpoints in FastAPI for key features like connections, variables, and more. Implementing these endpoints will enable users to manage resources more effectively through the API, improving usability and reducing the need for multiple workarounds.
The workload aims to help secure CLI by using these endpoints.

Missing Endpoints

• Import Connection(s) in single endpoint -> AIP-81 Implement POST/Insert Multiple Connections in FastAPI #43652 @bugraoz93
• Import Pool(s) in single endpoint -> AIP-81 Implement POST/Insert Multiple Pools in FastAPI #43896 @jason810496
• Import Variables(s) in single endpoint -> AIP-81 Implement POST/Insert Variables in FastAPI #43897 @jason810496 (Aims v3.1, this part will be covered from CLI to finalise AIP-81)

Code of Conduct

• I agree to follow this project's Code of Conduct

[bugraoz93]

Hey @pierrejeambrun, I need your help on AIP-81 if you have time. It is related to new set endpoints in FastAPI.

&TLDR
This is part of AIP-81. The aim is to use these endpoints from CLI. There are missing ones and one set of missing ones are these. I would like to get your help on reviews and your take on this before implementing because impacting more than one model and multiple endpoint in FastAPI.

I created these issues as file but I am not sure if we should accept files as an input in the API since they are harder to validate if the file is safe in APIs and brought more security concerns. I am thinking we can create POST endpoints that accept the models in bulks like post_connections endpoint and parse the files in CLI before sending and receiving request/response from API. What do you think?

Many thanks!

[jason810496]

Hi @bugraoz93, if I may suggest, I would +1 for the second option (parsing the files at the CLI level).
This approach would allow Pydantic data models to serve as the interface within the codebase for both the CLI and API, decoupling CLI-specific file-parsing logic from the API. Additionally, this would enable the CLI to directly utilize the public APIs.

By the way, I'm available to take on other missing APIs for AIP-81 if needed.

[bugraoz93]

Hi @bugraoz93, if I may suggest, I would +1 for the second option (parsing the files at the CLI level).
This approach would allow Pydantic data models to serve as the interface within the codebase for both the CLI and API, decoupling CLI-specific file-parsing logic from the API. Additionally, this would enable the CLI to directly utilize the public APIs.

Hi @jason810496, great to hear your +1 on this! I believe this will make both the CLI and API endpoints more secure and happy 😄

By the way, I'm available to take on other missing APIs for AIP-81 if needed.

Amazing! Once the approach is aligned, I’ll break these down further. Any help would be much appreciated!

[bugraoz93]

I've removed the export functionality, as we can retrieve all necessary data via the list endpoints and parse them to a given file from the CLI. This approach will reduce the workload and improve security for the API and the CLI. The more I looked at these endpoints and the approach, the more I felt this wasn’t even up for discussion, as this is clearly the better approach.

Additionally, I've created the remaining issues for anyone who wants to pick them up. I haven’t labelled them as good first issue yet.
cc: @jason810496 @josix if you're interested in more issues :)

[jason810496]

Sounds reasonable. I can take on both issues. Could you please assign them to me? Thanks !
By the way, I noticed a possible typo for the Pools endpoint. Should it be Insert Multiple Variables?

[bugraoz93]

Sounds reasonable. I can take on both issues. Could you please assign them to me? Thanks !
By the way, I noticed a possible typo for the Pools endpoint. Should it be Insert Multiple Variables?

Nice one! Fixed. Assigning both. Thanks for your effort! :)

[bugraoz93]

Can you comment them @jason810496? This is the only way I can see you in the issue.

[pierrejeambrun]

Most of these resources Connection Pool Variable already have a POST endpoint to create 1.

The CLI can either re-use with a loop and async calls to create multiple of them quickly as needed. (which can be enough at this point). Or we can develop a bulk insert endpoint, not sure if this is needed.

[bugraoz93]

Most of these resources Connection Pool Variable already have a POST endpoint to create 1.

The CLI can either re-use with a loop and async calls to create multiple of them quickly as needed. (which can be enough at this point). Or we can develop a bulk insert endpoint, not sure if this is needed.

In that sense, those endpoints are indeed not necessary. I agree that even hundreds of calls per second shouldn’t be an issue for FastAPI. However, for better exception handling and an improved user experience, I’d prefer to handle all inserts in a single request. Otherwise, if the request fails for multiple entries, users would need to split the file, identify each issue, and then retry each entry individually or through the UI/CLI as a single resource.

Let’s proceed with implementing the bulk insert endpoints, as it seems you’re not strictly opposed.

[pierrejeambrun]

Yes of course if we need a bulk insert for that use case we can definitely implement it.

How do we plan on handling failures ? (Even for bulk insert). Do we fail the entire transaction, or do we create those that could be created and return back the whole bunch of errors for the failing ones ?

[bugraoz93]

Awesome, thanks!

These endpoints will primarily be used when users are importing models from a file via the CLI.

My thinking is to fail the entire transaction and return a list of errors for any entries that caused issues. This way, users can quickly locate and fix specific items in the file or request, and then resend it. Creating only a portion of the entries would still be inconvenient for both API and CLI users. Not only is editing but also removing partial entries difficult if they’re spread across different sections of the list.

From the CLI perspective, we can return the response message to the user as we do with the API if it fails, so both are aligned.

What do you think?