Issue: Zero-Byte Files After Successful Uploads with `tusd`

Tus
1

Context: I’m using tusd to handle file uploads in a production environment. My setup includes custom pre-create and post-finish hooks to manage file paths, metadata, and post-upload processing. The files are being uploaded to a directory structure that includes paths with special characters and spaces, and some of the paths are relatively long (up to 328 characters).

Problem: While most files are uploaded successfully, I’ve encountered an issue where some files are uploaded but end up being 0 bytes in size. The filenames are correct, and the mv command in the post-finish hook executes successfully, indicating that the files are being moved to the correct directory with the expected names. However, these files have no content (0 bytes), which suggests that something is going wrong during or immediately after the upload process.

Details:

  • The issue does not seem to be related to the mv command itself, as the file names and paths are correct.
  • The problem occurs sporadically, with some files being uploaded successfully and others resulting in 0-byte files.
  • The paths involved are long (e.g., 328 characters), and the filenames include spaces and special characters.
  • My pre-create and post-finish hooks handle file metadata, date formatting, and path construction. I’ve checked these scripts for any obvious errors that might lead to the issue but haven’t found anything conclusive.
  • The issue might be related to the upload completion timing or filesystem handling, but I’m unsure how to diagnose further.

Steps Taken:

  1. Logging: I’ve added logging to the hooks to track the upload process, but the logs don’t show any errors related to the file handling. The mv command completes successfully, which makes it unclear why the files are 0 bytes.
  2. Path Length Check: The paths are long, but they don’t exceed the typical 4096-character limit imposed by most Linux filesystems.
  3. File Permissions: I’ve verified that the user running tusd has the appropriate permissions to write and move files in the target directory.
  4. Filesystem Check: Disk space is sufficient, and there don’t appear to be any obvious filesystem issues (e.g., no errors in dmesg or related logs).

Questions:

  • Has anyone encountered a similar issue with tusd where files end up as 0 bytes despite successful uploads?
  • Could the path length or special characters in the filenames be causing issues in ways I haven’t considered?
  • What additional debugging steps would you recommend to isolate the cause of these 0-byte files?
  • Is there a possibility that the upload completion timing (e.g., tusd finalizing the upload before the file content is fully written) could lead to this issue?

Any insights or suggestions would be greatly appreciated!

2

just looking at the logs and seen this
2024/08/26 13:05:16.789978 level=INFO event=UploadFinished method=POST path=“” requestId=“” id=f47eacb2665ff7db1b20e3c352bfd423 size=0
This log entry indicates that the upload process finished, but the file size was recorded as 0 bytes. Unfortunately, I don’t have corresponding log entries (like those from the post-finish hook) that would provide more details about this specific upload, such as the original filename or how this upload ID (f47eacb2665ff7db1b20e3c352bfd423) relates to the final file name.

This log entry suggests that something went wrong during the upload process, possibly causing the file to be saved without any content.

Has anyone encountered similar issues with zero-byte files in tusd? Any insights on how to trace this log ID back to the original filename or further investigate the cause would be greatly appreciated.

2024/08/26 13:05:16.789953 level=INFO event=UploadCreated method=POST path=“” requestId=“” id=f47eacb2665ff7db1b20e3c352bfd423 id=f47eacb2665ff7db1b20e3c352bfd423 size=0 url=https://hkauk.nalytics.net/files/f47eacb2665ff7db1b20e3c352bfd423

2024/08/26 13:05:16.789978 level=INFO event=UploadFinished method=POST path=“” requestId=“” id=f47eacb2665ff7db1b20e3c352bfd423 size=0

logs
2024/08/26 13:05:16.790010 level=DEBUG event=HookInvocationStart type=post-finish id=f47eacb2665ff7db1b20e3c352bfd423

2024/08/26 13:05:16.790077 level=INFO event=ResponseOutgoing method=POST path=“” requestId=“” id=f47eacb2665ff7db1b20e3c352bfd423 status=201 body=“”

2024/08/26 13:05:16.790630 level=DEBUG event=HookInvocationStart type=post-create id=f47eacb2665ff7db1b20e3c352bfd423

2024/08/26 13:05:16.793558 level=DEBUG event=HookInvocationFinish type=post-create id=f47eacb2665ff7db1b20e3c352bfd423

2024/08/26 13:05:16.869871 level=DEBUG event=HookInvocationStart type=post-receive id=d51bd6f013890e42ccfdd5aa7626dd70

2024/08/26 13:05:16.871301 level=DEBUG event=HookInvocationFinish type=post-receive id=d51bd6f013890e42ccfdd5aa7626dd70

2024/08/26 13:05:17.019723 level=DEBUG event=HookInvocationFinish type=post-finish id=b6766ce017c5001937bae10972e1aa66

2024/08/26 13:05:17.028150 level=DEBUG event=HookInvocationFinish type=post-finish id=512fed0f49ea029c542813844d93754b

2024/08/26 13:05:17.033289 level=DEBUG event=HookInvocationFinish type=post-finish id=58386721f723ae43b3b4a86af23f6cc3

2024/08/26 13:05:17.033431 level=DEBUG event=HookInvocationFinish type=post-finish id=f47eacb2665ff7db1b20e3c352bfd423

2024/08/26 13:05:17.106264 level=DEBUG event=HookInvocationStart type=post-receive id=b69969e072cb1f3c85b34de916253bf5

3

We believe the issue stems from OneDrive’s “Files On-Demand” feature. This functionality displays all files and folders as placeholders, even if they haven’t been fully downloaded to your local machine. While these placeholders appear as complete files, they occupy minimal disk space. As a result, when you attempt to upload these placeholder files using TUSD, the upload process generates zero-byte files because the actual content hasn’t been downloaded.

we are about to test this. have any one else had this issue?

4

Just had a call with the client and all the onedrive files were downloaded, so that wasn’t the issue

in the uppy client when the files were selected they showed zero bytes in uppy ( in file explorer they have the actual full size

#####################
another issue we just seen is some of the file name truncated
e.g 1.12.13.1.3.1 CHT-LOM-BB-08-DR-A-120070 L08 RCP Setting Out Plan (C02).pdf

changed to 112131~1.PDF

5

I have now replicated this issue which is relating to path limits

The issue occurs when file paths exceed 265 characters in length, causing the file upload to result in a zero-byte file. Has any one had this issue and have ways around it?

Path 1: Short path with a long filename (264 characters)

  • Path: C:\Test\LongFilename\
    • Path Length: 23 characters
  • Filename: This_is_a_very_long_filename_designed_to_exceed_265_characters_and_test_the_system_limitation_on_file_path_length_in_a_Windows_environment_to_see_how_it_handles_such_cases_with_different_setups_and_scenarios_including_various_fdfdfdfd.pdf
    • Filename Length: 241 characters
  • Total Characters: 23 (path) + 241 (filename) = 264 characters
  • Result: Works

Path 2: Short path with a long filename (265 characters)

  • Path: C:\Test\LongFilename\
    • Path Length: 23 characters
  • Filename: 1This_is_a_very_long_filename_designed_to_exceed_265_characters_and_test_the_system_limitation_on_file_path_length_in_a_Windows_environment_to_see_how_it_handles_such_cases_with_different_setups_and_scenarios_including_various_fdfdfdfd.pdf
    • Filename Length: 242 characters
  • Total Characters: 23 (path) + 242 (filename) = 265 characters
  • Result: Doesn’t work (zero bytes)

Path 3: Long path with a long filename (265 characters)

  • Path: C:\User1\TestUser\Documents\DeepFolderStructureTest\Level1\Level2\Level3\Level4\Level5\Level6\Level7\Level8\Level9\Level10\
    • Path Length: 123 characters
  • Filename: Long_Filename_For_Deep_Folder_Structure_Test_That_Should_Be_Over_265_Characters_Total_And_Check_How_The_System_Handles_Long_File1ffd.pdf
    • Filename Length: 142 characters
  • Total Characters: 123 (path) + 142 (filename) = 265 characters
  • Result: Works

Path 4: Long path with a long filename (266 characters)

  • Path: C:\User1\TestUser\Documents\DeepFolderStructureTest\Level1\Level2\Level3\Level4\Level5\Level6\Level7\Level8\Level9\Level10\
    • Path Length: 123 characters
  • Filename: Long_Filename_For_Deep_Folder_Structure_Test_That_Should_Be_Over_265_Characters_Total_And_Check_How_The_System_Handles_Long_File1ffd2.pdf
    • Filename Length: 143 characters
  • Total Characters: 123 (path) + 143 (filename) = 266 characters
  • Result: Doesn’t work (zero bytes)
6

What a deep dive, nice investigation! If I recall correctly, Windows has more restrictive path length limits than I am used to from Linux: Maximum Path Length Limitation - Win32 apps | Microsoft Learn

So I think this isn’t an issue/bug in tusd, but instead you have to adjust the naming approach in your pre-create hook to not generate names that exceed Window’s limits. Does that make sense?

7

fwiw i think paths can be prefixed with \\? to allow longer paths, e.g. \\?\c:\some\long\path (see Windows: Long path Path causes error · Issue #745 · mifi/lossless-cut · GitHub)