How do use transloadit addStream() function in the NodeJS SDK?

Transloadit
1

Hi,

Trying out the transloadit api, the template works when I use the testing mode on the transloadit website, but when I try to use it in Node JS with the SDK I’m getting an error:

INVALID_FORM_DATA - https://api2.transloadit.com/assemblies - INVALID_FORM_DATA: The form contained bad data, which cannot be parsed.

The relevant code: (_asset.content) is a Buffer object

async function getThumbnailUrl(_assetkey: string, _asset: I.FormFile): Promise<string> {
  let tOptions = {
    waitForCompletion: true,
    params: {
      template_id: process.env.THUMB_TRANSLOADIT_TEMPLATE,
    },
  };
  const stream = new Readable({
    read() {
      this.push(_asset.content);
      this.push(null);
    },
  });
  console.log(_asset.content);
  util.transloadit.addStream(_assetkey, stream);

  return new Promise((resolve, reject) => {
    util.transloadit.createAssembly(tOptions, (err, status) => {
      if (err) {
        reject(err);
      }
      console.log(status);
      //return status;
      resolve(status);
    });
  });
}
2

Hello.

I don’t notice any immediate errors on your part, and I’m currently looking through the source code of the Node SDK to see if there’s some sort of error in handling the file.

Therefore, I think the best way to figure out where the error lies is to use a File Stream that is known to work, like reading one directly from a file. If that also fails, then it’s probably an internal issue.

However, I don’t think it is an internal issue, as addFile uses the addStream method, like so:

  addFile (name, path) {
    const stream = fs.createReadStream(path)
    stream.on('error', err => {
      // handle the error event to avoid the error being thrown
      console.error(err)

      if (this._streams[name]) {
        delete this._streams[name]
      }
    })
    this.addStream(name, stream)
  }

So what I would assume is that, the stream you are passing is invalid for some reason, and this isn’t an error with the SDK itself. I don’t know the exact reason behind this, but I’d assume that _asset.content is not a valid file to pass. Maybe try reading the stream yourself and see if it is what you are looking for.

Besides this, I don’t really know what else to recommend. Good luck. If there are any further developments on this issue or you have any more questions or problems, let me know

- Andrew

3

I’ve tested the _asset.content buffer and it writes the same file to S3 perfectly fine, so I don’t think it’s invalid.

This is also not a problem with the Nodejs SDK -> I tested directly using form-data sent to the API via http and same error.

One of my big frustrations with the transloadit documentations is that it seems very opinionated about only reading files from disk or importing from a cloud provider.

I’m writing serverless functions, so I get the response as multipart/form-data and I should just be able to use the buffer to send it along.

My current workaround is going to have to be to use the buffer to upload to S3, then get the S3 url, then import from S3 into transloadit, instead of just being able to send my file directly to transloadit through the buffer. It would work (I think, haven’t tested it yet), but just seems unnecessarily resource intensive.

4

That’s strange. It seems like it should be able to handle uploads in this way. Especially since something like uppy allows uploads from many sources with Transloadit.

I have no idea why this won’t work. It’s certainly possible to be an error within the API itself, but I don’t know the likelihood of that.

As I’m not really sure where to go from here, I’ll try relay this post to the core team of Transloadit

5

Just to provide some closure to this topic,
I just tested my workaround (upload to s3, then use import s3 robot to grab the file) and got it to work with the nodejs sdk so i should be good using that.

I have a suspicion the error I was getting was not to do with the transloadit api, but rather the form-data library for node js (https://github.com/form-data/form-data) and that’s somehow not inputting the form data in the way that the transloadit api is expecting.

But as there aren’t alternatives to that library that I could find, I wasn’t really able to test that hypothesis.

PS. Would be great if there was a tutorial on how to use transloadit with serverless functions :slight_smile:

6

Okay so I reached out to someone on the core team, and this was there response (paraphrased)

It may try to set his streams to be Tus streams which would mean that they’re not uploaded as multipart/form data.

In either case it seems like the error to his callback would be originating from the error out of _remoteJson

These could be the problem areas

  1. https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L146
  2. https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L606
  3. https://github.com/transloadit/node-sdk/blob/master/src/TransloaditClient.js#L642

It is also possible that the form-data library could be the source of the error

To really test this further we’re going to need to try using the library he was using, make sure the output of it is good, and then debug the node-sdk to see where the logic failure is in it, or if the logic failure is on the API side.

We’re gonna look further into this

1 Like