Un-tar:ing and un-gzip:in uploaded files on the fly on with tus java server

I’m uploading a gzipper tar file to the server and this contains giga bytes of data so I’m trying to avoid unnecessary copies. For this reason I’m trying to process the file as it comes in. I’ve found packages to do that to an input stream but I cannot figure how to access the tus server input stream. I guess this is not possible directly as the data comes in chunks in separate http requests. So my next approach would be to process the chunks myself, create some sort of in memory stream from then and use that as the input to the ungzip untar pipeline.

Does that sound feasible?

This then raises the question can the chunks be received out of order?

I haven’t used tus-java-server on my own, but it seems possible if you implement an UploadStorageService (see tus-java-server/src/main/java/me/desair/tus/server/upload/UploadStorageService.java at 1c4adf10e2ef29d86f94e572c35e7e22137ff3a0 · tomdesair/tus-java-server · GitHub and tus-java-server/src/main/java/me/desair/tus/server/TusFileUploadService.java at 1c4adf10e2ef29d86f94e572c35e7e22137ff3a0 · tomdesair/tus-java-server · GitHub). It doesn’t give you a full InputStream directly, but append is called whenever the client sends more data in another request. This is where you can process the uploaded data to your wishes.

With the contactenation extension yes, otherwise no.

Thanks. I did get this to work but I found it very awkward to pass all the relevant information from by spring controller to the UploadStorageService and had to resort to what looked like a hack. So I wrapped my own TUS server side code.

I’m glad to hear that you were able to make it work! Most tus server implementations are not focused on processing the uploaded data in “real-time” and instead store it somewhere for later processing once the upload is complete. Maybe this is an area we can support better in the future.