Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Add semantics to fileReader #583

Closed
nh2 opened this issue Apr 15, 2023 · 1 comment
Closed

docs: Add semantics to fileReader #583

nh2 opened this issue Apr 15, 2023 · 1 comment

Comments

@nh2
Copy link
Contributor

nh2 commented Apr 15, 2023

The docs at https://github.com/tus/tus-js-client/blob/beea5983cc79d6d61b65d150cef8809114873661/docs/api.md#filereader show the type signatures, but they don't explain what the values mean.

interface FileReader {
  openFile(input: any, chunkSize: number): Promise<FileSource>
}

interface FileSource {
  size: number
  slice(start: number, end: number): Promise<SliceResult>
  close()
}

interface SliceResult {
  // Platform-specific data type which must be usable by the HTTP stack as a body.
  value: any
  done: boolean
}

For example:

  1. What's the meaning of done: boolean, to what value should one set it? The code at

    tus-js-client/lib/upload.js

    Lines 784 to 791 in beea598

    return this._source.slice(start, end).then(({ value, done }) => {
    // If the upload length is deferred, the upload size was not specified during
    // upload creation. So, if the file reader is done reading, we know the total
    // upload size and can tell the tus server.
    if (this.options.uploadLengthDeferred && done) {
    this._size = this._offset + (value && value.size ? value.size : 0)
    req.setHeader('Upload-Length', this._size)
    }

    suggests it's to indicate the end of the stream when uploadLengthDeferred is used. So I guess there are 2 possible cases:
    a. uploadLengthDeferred = true, FileSource.size = Infinity -- in that case, probably my slice() function must also be prepared to receive end = Infinity, and I should set done = true exactly when I know that the streaming is now finished.
    b. uploadLengthDeferred = false, in that case FileSource.size must be finite, my slice will only be called with finite end, and it is irrelevant what I set done to because the value is not read.
  2. Is there any other contract over what the paremeters in slice(start, end) will be? For example:
    a. Is end the exclusive boundary, like in Blob.slice()?
    b. Will it be that end - start <= chunkSize (the chunkSize that was passed to openFile())?

One can guess answers from the current code, but since this is an interface, it would be nice to know what invariants will hold in the future.

@Acconut
Copy link
Member

Acconut commented Apr 16, 2023

Good point, thanks for bringing this up. The documentation could definitely be improved here.

2. a. Is end the exclusive boundary, like in Blob.slice()?

Yes, see

// The `end` option for createReadStream is treated inclusively
// (see https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options).
// However, the Buffer#slice(start, end) and also our Source#slice(start, end)
// method treat the end range exclusively, so we have to subtract 1.
// This prevents an off-by-one error when reporting upload progress.
end: offset + end - 1,

2. b. Will it be that end - start <= chunkSize (the chunkSize that was passed to openFile())?

If end is finite, yes. But be aware that end might also be Infinity.

  1. What's the meaning of done: boolean, to what value should one set it?

done indicates if subsequent calls to slice will not return any data because the stream has ended. So done can be set in the final slice call where the final data chunk was returned:

{
  done: true,
  value: [some_data]
}

Or done can be set as the only property if you notice that no more data is available:

{
  done: true,
  value: null
}

Note that in this case, tus-js-client might have to send an empty PATCH request to indicate the end of the upload if you used uploadLengthDeferred.

I am not entirely sure if done is only used if uploadLengthDeferred is enabled, but I encourage every implementation to handle it properly because we cannot guarantee that this might not change in the future.

I will make sure to add all these points to the documentation soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants