First time poster here so if anything doesn’t make sense, please let me know and I’d be happy to explain
I’m currently working on a Node.js application where I need to handle file uploads using tus-node-server. The frontend is using the Uppy TUS plugin. I need to be able to encrypt the files as they are uploaded and then upload the encrypted files directly to an S3 bucket.
Due to limited storage space, I can’t afford to store the files temporarily on disk. Instead, I want to process and encrypt the files as they are being uploaded and stream the encrypted content directly to S3. I’ve also opted against using Server-Side Encryption on its own in S3, as if someone gained unauthorized access to my S3 bucket, they could just download the files unencrypted.
I understand there is no native way of doing this in @tus/server and @tus/s3-store, which is fine, as I have my own encryption logic using crypto.
I have managed to successfully get everything working so that the client can upload files using TUS to my S3 bucket with both modules mentioned above.
I just need a way of pre-processing each chunk before it is sent to S3. My main question is this:
Is there any way to extract/retrieve/intercept the uploaded chunk so that I can perform my encryption logic, and then pass it back to the TUS server/S3 Datastore?
Any help, suggestions, or examples would be greatly appreciated!
Note that this is always the case with the S3 store, there will be temporary files. We need those in case a chunk is under the minimum part size or to buffer chunks until it reaches your partSize. They are deleted as soon as they’re no longer needed though.
Thanks for the reply and I understand that you don’t want to allow this.
I’m aware of S3’s server-side encryption but as I mentioned, I would like client-side encryption also as an extra layer for if someone was to gain access to my S3 bucket.
I had a method set up to encrypt files as soon as they were uploaded to the @tus/file-store which worked fine. Is there any way for me to modify the temporary file on disk before it’s uploaded to the S3 Store or will I have to use the S3 client SDK instead?
I don’t think you can target the temporary file on disk yourself without causing race conditions. Maybe we could add support for checksums, together with auto encryption in the bucket that could suffice?
That’s ok, no problem. I understand the potential issues with accessing the temporary file.
My main goal is to protect client files so that no one (not even me) can see them.
I think implementing checksums is a good idea but for now, I might leave this project to do a bit more research.
Is there a possibility in the future that you could implement support for something like this, where I can pipe the upload to an encrypt function and then to my own datastore (FileStore, S3, etc.)?
I couldn’t find any related forum topics so I understand that this is not a highly requested feature, and no worries if this is not possible.
This sounds to me like a use case for client-side encryption, where the data is already encrypted before it reaches tis-node-server. Implementing this on the client should be more straightforward and would not require a special setup on the server. Have you considered that?
Yeah, I did think about client-side encryption and that’s something I’ve been looking into.
The way I was trying to do this originally was by letting the client (web browser) upload a file, then encrypting it on the Node server, and then uploading to S3. So technically, this is still client side encryption but from AWS’ point of view.
