Parallel Uploads

Hello,

I have been researching and looking through the node JS server, the Golang server, and the client JS library, and I am extremely confused.

The TUS spec mentions that the concat extension supports parallel uploads. The client code seems even to support this. Yet the Golang server uses a per-file optional locker (which makes sense).

I am asking what I am missing here. I also question how, even without having a locker, it might actually handle that as a race condition to storing the metadata for each upload in parallel.

I find it even more confusing that the official client supports this but not the official server. I know a few extensions, like hashing, were not added, but this one seems a bit out of place.

Kudos.

Hello, there seems to be a misconception in your understanding of parallel uploads. With the concatenation extension, a single file is broken up into multiple chunks, each of which is uploaded as its own upload with its own locker. Thus, there is no issue with the locks. Once the partial uploads are complete, the chunks are concatenated on the server-side yielding the original file.

I am not sure what you mean here. The tusd server supports the concatenation extension for its file, S3 and GCS storages (although it’s not yet implemented for storing on Azure).

Not that I can see? The ID used by the locker is from the path tusd/pkg/handler/unrouted_handler.go at aa190c85d4cdfccd99b830b20c6b885a01685ddc · tus/tusd · GitHub.

Unless the client somehow knows to use different upload paths, which, if so, I am not sure how that works, the locker seems to be global, not per multipart instance.

Based on what I found, the JS client clearly supports parallel uploads, but it’s not clear how the server does.

The client creates one upload resource per file chunk. Each upload resource has its own ID. Therefore each file chunk upload has its own individual lock. You will not find the corresponding logic in server-side code but that logic is located in the clients.

Tusd supports its out-of-the box without any additional configuration if you are using disk, S3- or GCS-based storage.

Can you link to the client code that does this, please?

Also, a bit related, I want to bring your attention to S3 backend with TUS handler fails to handle PATCH · Issue #1067 · tus/tusd · GitHub, which I found while R&D’ing.

You can find the implementation of parallel uploads in tus-js-client at tus-js-client/lib/upload.js at 192e1d96a5cba55b398cf0957a6a4cc88e7ef72e · tus/tus-js-client · GitHub.