This could potentially cause a huge number of requests to the server, depending on the size of the file and the chunk size. I see in the docs you have addressed this partially with the suggestion to increase chunk size for larger files:
> The S3 Multipart plugin uploads files in chunks. Each chunk requires a signing request `prepareUploadPart()`. To reduce the amount of requests for large files, you can choose a larger chunk size, at the cost of having to re-upload more data if one chunk fails to upload.
My proposal is to instead generate the presigned URLs in batches, rather than one each time a chunk needs it. In the code above there is already the `candidates` which is controlled partly by the `limit` option. I think it would be better to generate the presigned URLs for a batch of candidates at once, something like this (pseudocode-ish):
```javascript
this.options.prepareUploadParts.then((partsWithUrl) => {
candidates.forEach((index) => {
this._uploadPartRetryable(index, partsWithUrl[index]).then(() => {
// Continue uploading parts
this._uploadParts()
}, (err) => {
this._onError(err)
})
})
});
////
prepareUploadParts (parts, { key, uploadId }) {
this.assertHost('prepareUploadParts')
const filename = encodeURIComponent(key)
return this.client.post(`s3/multipart/${uploadId}/prepare-parts`, { parts, key })
.then(assertServerError)
}
////
_uploadPart (index, presignedUrl = null) {
const body = this.chunks[index]
this.chunkState[index].busy = true
return Promise.resolve().then(() =>
if (presignedUrl) {
return Promise.resolve({ url: presignedUrl });
} else {
this.options.prepareUploadPart({
key: this.key,
uploadId: this.uploadId,
body,
number: index + 1,
})
}).then((result) => {
const valid = typeof result === 'object' && result
&& typeof result.url === 'string'
if (!valid) {
throw new TypeError('AwsS3/Multipart: Got incorrect result from `prepareUploadPart()`, expected an object `{ url }`.')
}
return result
}).then(({ url, headers }) => {
if (this._aborted()) {
this.chunkState[index].busy = false
throw createAbortError()
}
return this._uploadPartBytes(index, url, headers)
})
}
```
That way the number of round trips to Companion or the custom server that someone has set up is drastically reduced. For example, a 1GB file uploaded in 5mb chunks will normally require 200 trips to the server. With this change it would require only 50 (if the limit was 4).
I think this will necessitate longer expiry times for presigned URLs as well, which should be made known in the documentation. Would you be open to us opening a pull request to add this functionality? If so, we would add the corresponding functionality to Companion if needed.
If this is not the right channel for such a proposal, please let me know and I’ll propose it somewhere else. Thanks!
---
### Author:  [@kvz](https://community.transloadit.com/u/kvz)
#### Post date: [July 6, 2021, 6:16am UTC](https://community.transloadit.com/t/presigning-urls-in-batches-for-aws-s3-multipart-upload/15774/2 "2021-07-06T06:16:19Z")
Hi Martin, thanks for reaching out. We’re Discourse users ourselves (surprise 😄) and I’m excited about the prospect of possibly working together. I think your proposal makes sense and have asked the Uppy team to take a closer look and get back to you here.
---
### Author:  [@martinbrennan](https://community.transloadit.com/u/martinbrennan)
#### Post date: [July 7, 2021, 12:45am UTC](https://community.transloadit.com/t/presigning-urls-in-batches-for-aws-s3-multipart-upload/15774/3 "2021-07-07T00:45:59Z")
Thanks a lot! I another proposal to do with resuming multipart S3 uploads that I will flesh out at some point, but for now it’s great to hear that you are open to the contribution 🙂
---
### Author:  [@Merlijn](https://community.transloadit.com/u/Merlijn)
#### Post date: [July 8, 2021, 10:11am UTC](https://community.transloadit.com/t/presigning-urls-in-batches-for-aws-s3-multipart-upload/15774/4 "2021-07-08T10:11:06Z")
Hi @martinbrennan, I think this makes a lot of sense and we’d be happy to accept a PR for it if you are willing to work on it. Let me know if there are any questions.
---
### Author:  [@martinbrennan](https://community.transloadit.com/u/martinbrennan)
#### Post date: [July 30, 2021, 1:55am UTC](https://community.transloadit.com/t/presigning-urls-in-batches-for-aws-s3-multipart-upload/15774/5 "2021-07-30T01:55:32Z")
Hey @Merlijn I’ve opened up a PR for this functionality now. It was a fun learning experience, I haven’t used much pure JS in a while, and Jest is a great test runner:
>
>
> I'm opening this PR after discussing this on the forum, see https://community.tr…ansloadit.com/t/presigning-urls-in-batches-for-aws-s3-multipart-upload/15774
>
> \-----
>
> This PR introduces batch presigning part URLs when using the AWS S3 multipart plugin. I've put this behind a new option; by default the existing flow using \`prepareUploadPart\` is used. The new option is \`batchPartPresign\`.
>
> When \`batchPartPresign\` is true, instead of calling \`prepareUploadPart\` for each chunk we call the new function \`batchPrepareUploadParts\`, which must be filled in via an option if companion is not being used. That function takes an array of part numbers, and needs to return an object with the part numbers as the keys and the presigned URLs as the values. I've added the corresponding companion endpoint for this batch presigning too.
>
> Additionally, I added the \`minNeededForPresignBatch\` option. This is the minimum number of part candidates required before attemtping to batch presign some chunk URLs. This is used so that after the initial batch of \`limit\` presigned URLs are requested, smaller requests for one or two URLs at a time are not done.
>
> I've also done my best to add a fair amount of unit tests for the AWS S3 multipart plugin as it did not have any previously, though I haven't added tests for the companion controller because there aren't any for any of the companion routes.