Trying to upload large files into chunks using multipart/form-data but connexion.request.stream seems to be empty

[MajorSquirrelTVS]

Description

I'm trying to implement a POST request to upload large files (~4GB) using connexion set up with a Flask server. The physical is an embedded one so the performances are critical.

Having read the OpenAPI 3.0 File Upload specs, it suggests to use multipart/form-data to upload a file with additional data (e.g filename, user ID, ...). Here is the API I'm using :

api.yaml

paths:
  /records:
    post:
      tags:
          - records
      summary: Upload a new record to the server
      operationId: server.controllers.file_controller.records_post
      requestBody:
        required: true
        content:
          multipart/form-data:
            schema:
              additionalProperties: false
              type: object
              properties:
                record:
                  type: string
                  format: binary
                  description: Record to be uploaded to the server
                filename:
                  type: string
                  description: New filename for the record being uploaded
                  required:
                      - record
              required:
                  - record
      responses:
        201:
          description: Created, the new record has been successfully uploaded to the server
        409:
          description: Conflict, a record already exists with that name on the server

file_controller.py

def records_post():
    # Create destination directory if it does not exist
    os.makedirs(os.path.dirname(RECORDS_PATH), exist_ok=True)

    record = connexion.request.files['record']
    filename = secure_filename(record.filename)

    if 'filename' in connexion.request.form and connexion.request.form['filename']:
        filename = secure_filename(connexion.request.form['filename'])

    filepath = os.path.join(RECORDS_PATH, filename)

    if os.path.isfile(filepath):
        return {'status': 'Conflict', 'message': 'A record already exists with that name on the server'}, 409

    try:
        record.save(filepath)
    except Exception as e:
        logging.error("Failed to save '{}' to: {}".format(filename, RECORDS_PATH))
        logging.error(str(e))
        return {'status': 'Internal Server Error', 'message': 'Failed to upload record to the server',
                'error': str(e)}, 500

    return {'status': 'Created', 'message': 'Record uploaded successfully !', 'location': filename}, 201

When using multipart/form-data I can reach to the uploaded file using connexion.request.files['record'] but it seems that it has to save the file into disk when file size exceeds 500KB and I'm limited to the max available space in /tmp on my server (which is ~1.8GB).

Some people in the issues suggest to use streaming-form-data, so I've taken a look at an implementation for Flask server but it seems that connexion.request.stream is empty. Apparently, the stream would be empty if it was previously read by request.data... and if I use request.data or request.get_data(), I can't find a way to split the bytestring into chunks to save memory and write to the right destination.

If I use application/octet-stream, I can stream the file directly but I'm loosing any chance to upload additional data and I encounter the same problem as above which is: connexion.request.stream is empty anyway and I can't read connexion.request.get_data() into chunks.

Is there any way to read non-empty connexion.request.stream or read connexion.request.get_data() into chunks ?

Additional info:

Output of the commands:

• python --version 3.8.5
• pip show connexion | grep "^Version\:" 2.7.0
• OpenAPI version : 3.0.0

[RobbeSneyders]

Fixed since #1618