Uploads don't seem to reach S3 -but info does-

Tus
1

Hey!

I have a basic setup with a uppy client and tusd server connected to a S3-like storage provider. As a simple test, I am trying to upload a less-than-2Kb file to it.

I declare the configuration required according to the docs (both env variables and tusd parameters):

AWS_ACCESS_KEY_ID=xx
AWS_SECRET_ACCESS_KEY=xx
AWS_REGION=the-region

tusd -s3-bucket=test-bucket -s3-endpoint https://storage-endpoint.com

I get the correct initialization:

2025/02/20 16:12:17.801556 Using 'https://the-expected' as S3 endpoint and bucket for storage.
[...]
2025/02/20 16:12:17.803516 Enabled hook events: pre-create, post-create, post-receive, post-terminate, post-finish
2025/02/20 16:12:17.803523 Supported tus extensions: creation,creation-with-upload,termination,concatenation,creation-defer-length
2025/02/20 16:12:17.803543 Using 0.0.0.0:8080 as address to listen.
2025/02/20 16:12:17.803548 Using /files/ as the base path.
[...]

And everything works as expected on the upload, without error messages:

2025/02/20 16:12:21.054614 level=INFO event=RequestIncoming method=OPTIONS path=8eda2f9e71e32db70ca1566baaf480c0+b7566552634573410000000001432ebc485a85ebfd2e2d68 requestId=""
2025/02/20 16:12:21.054711 level=INFO event=ResponseOutgoing method=OPTIONS path=8eda2f9e71e32db70ca1566baaf480c0+b7566552634573410000000001432ebc485a85ebfd2e2d68 requestId="" status=200 body=""
2025/02/20 16:12:21.104795 level=INFO event=RequestIncoming method=HEAD path=8eda2f9e71e32db70ca1566baaf480c0+b7566552634573410000000001432ebc485a85ebfd2e2d68 requestId=""
2025/02/20 16:12:21.497049 level=INFO event=ResponseOutgoing method=HEAD path=8eda2f9e71e32db70ca1566baaf480c0+b7566552634573410000000001432ebc485a85ebfd2e2d68 requestId="" id=8eda2f9e71e32db70ca1566baaf480c0+b7566552634573410000000001432ebc485a85ebfd2e2d68 status=404 body=""
2025/02/20 16:12:21.556533 level=INFO event=RequestIncoming method=OPTIONS path="" requestId=""
2025/02/20 16:12:21.556597 level=INFO event=ResponseOutgoing method=OPTIONS path="" requestId="" status=200 body=""
2025/02/20 16:12:21.567697 level=INFO event=RequestIncoming method=POST path="" requestId=""
2025/02/20 16:12:21.567851 level=DEBUG event=HookInvocationStart type=pre-create id=""
2025/02/20 16:12:21.574081 level=DEBUG event=HookInvocationFinish type=pre-create id=""
2025/02/20 16:12:21.817223 level=INFO event=UploadCreated method=POST path="" requestId="" id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 size=20197 url=http://127.0.0.1:8080/files/bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2
2025/02/20 16:12:21.817292 level=INFO event=ResponseOutgoing method=POST path="" requestId="" id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 status=201 body=""
2025/02/20 16:12:21.817379 level=DEBUG event=HookInvocationStart type=post-create id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2
2025/02/20 16:12:21.822731 level=DEBUG event=HookInvocationFinish type=post-create id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2
2025/02/20 16:12:21.976243 level=INFO event=RequestIncoming method=OPTIONS path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId=""
2025/02/20 16:12:21.976379 level=INFO event=ResponseOutgoing method=OPTIONS path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId="" status=200 body=""
2025/02/20 16:12:22.014607 level=INFO event=RequestIncoming method=PATCH path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId=""
2025/02/20 16:12:22.068862 level=INFO event=ResponseOutgoing method=PATCH path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId="" id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 status=409 body="ERR_MISMATCHED_OFFSET: mismatched offset\n"
2025/02/20 16:12:22.322604 level=INFO event=RequestIncoming method=OPTIONS path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId=""
2025/02/20 16:12:22.322684 level=INFO event=ResponseOutgoing method=OPTIONS path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId="" status=200 body=""
2025/02/20 16:12:22.350791 level=INFO event=RequestIncoming method=HEAD path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId=""
2025/02/20 16:12:22.403231 level=INFO event=ResponseOutgoing method=HEAD path=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 requestId="" id=bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2 status=200 body=""

On the S3 storage I get the .info file correct, with all the expected data. The file does not appear. I read the docs so I know:

The file object is not visible in the S3 bucket before the upload is finished because the transferred file data is stored in the associated S3 multipart upload
~ by docs

But nothing appears after waiting for minutes (plus no more requests appear on the console).

If I do a GET request to the upload url, in this example, http://127.0.0.1:8080/files/bd78e42ed690c8edf9280a49826c7310+b7566552634573410000000001432f24152a9170a9e5b3a2, I get the standard S3 response for object not found:

ERR_INTERNAL_SERVER_ERROR: operation error S3: ListParts, https response error StatusCode: 404, RequestID: b99bce60-efc5-4dab-a8bf-f432c08509bf, HostID: , api error NoSuchKey: The specified key does not exist.

Any suggestions?

Edit: Improve console legibility

2

Hello,

this seems odd. The server is creating the upload resource fine (see the logs for the POST request), but when the client tries to upload the data in PATCH request, it refuses with

ERR_MISMATCHED_OFFSET: mismatched offset

That’s very unexpected because directly after creating an upload, the offset should be 0, but the server seems to have calculated a different offset.

It seems you are not using AWS S3 but a different S3-compatible storage. Maybe it’s not fully compatible with S3 and inconsistencies in their API causes problem. What storage provider are you using?

3

I’m using Intercolo object storage. In their documentation the link to the official AWS documentation except for known differences.

Thanks for he help!

4

I changed the provider and still getting a similar error when trying to upload files. Error appears when doing the PutObject request and is also about a hash mismatch. Looks like s3-like services are not really supported (which makes sense bc no api is totally equal to aws api).

5

Unfortunately I haven’t been able to test Intercolo on my own yet. Time is pretty tight right now, but if you can e-mail me credentials for some test bucket at marius@transloadit.com, I should be able to have a brief look if there is some incompatibility between this provider and tusd.

That being said, I know that people are running tusd successfully with S3-compatible storages, like Minio, CloudFlare R2, Digital Ocean Spaces etc. Tusd does not only work with AWS S3, but there is always a possiblity of some interoperability issues as every S3-compatible storage is slightly different at the end of the day.

6

Yeah, that was the point of my update.

I managed to get it working with leafcloud but as said, neither intercolo nor anothers providers I checked worked successfully. Most of providers guarantee a s3-compatible API. I guess the only way to know if it will work is to give it a try, so for anyone reading this try with a trial account!

I could send you an api token but my account has only ten days left until the trial ends and i don’t plan to extend it. The whole point of requesting the trial was to check for incompatbilities with tusd :face_with_tears_of_joy:

thanks anyway!

7

Alright, glad to hear you got it working! Let me know if you run into any other problems.

8

I sent you some credentials in case you want to debug this further… let me know if you need anything! thanks!!

9

Hello, unfortunately I wasn’t able to try it out with your test credentials in time on my own. However, I just added a new flag to log all S3 API calls, which is released in tusd v2.8.0. You can just set the flag, reproduce the problem and send me the logs, so I can see if something stands out.

More details are available at:

If you are still interested in debugging this, feel free to send me the logs via e-mail.