Hey! I’m using tus as express middleware to accept file uploads. I’m deploying my API to ECS and use EFS as storage. It goes fine when using single-instance deployment. When scalling this to multi instance I started noticing a random HTTP 409s when uploading files in chunks. It looks to me like when my requests are being routed to different instances when uploading the file in chunks it may happen the instance is not aware yet of the previous chunk upload. Client side retry helps.
Do you maybe have experience with similar setup or know whats the root cause of this?
I could “blindly” try to replace my network drive with something else but would like to know if its a known problem possibly and if you have experience with multi instance deployments of tus server.
(Since I’m in AWS I could use S3 but I initially preferred not to add any application managed networking - preferred simple drive bind to an instance)
Hi,
we have be been saving uploads on EFS in production with tusd and it works well. EFS is not a problem on its own but since it’s a shared storage between multiple instances you have to pay attention that only one instance is writing to a specific upload on the drive at any given time. Otherwise you can run into these 409 responses. Without more details, I cannot help you more, but when the upload request for a subsequent chunk arrives, the first chunk might not be fully written to EFS yet.
Tusd solves this by using upload locks to ensure that only one instance can access the upload data on storage: Upload locks | tusd documentation
What server implementation are you using? Did you build it on your own?
Thanks for a quick answer. Ok thats great to know EFS is battle tested.
I’m using nodejs @tus/server and merge it into my express REST API like in example:
After posting here I went through FAQs and found info about distributed locking. I could not find any implementation tough for nodejs. Is there any e.g. redis based locking implementation for nodejs?
For now I settled for sticky session for chunked uploads - I’m testing this now and it seems to work fine 
That’s a working solution but I would prefer to switch to distributed locking since that’s a zero “work” on client side - currently I have to rely on cookies
You can pass a locker to @tus/server.
We don’t have an official RedisLocker but if you happen to use Postgres you can use the one from Supabase: storage/src/storage/protocols/tus/postgres-locker.ts at master · supabase/storage · GitHub