Archived from the Transloadit support forum. Originally posted on June 20, 2014; details may be outdated, see our docs for current behavior. Original thread
I am a computer consultant working on behalf of NUGS.net and Mike Elberg. We’re trying to solve an issue that just started happening last Friday. When they do exports of TransloadIT jobs to their server (cryptical.nugs.net/209.234.243.31 - hosted in an Equinix colo) via FTP, the uploads go very slowly (30kbps - see attachment). The FTP server is running on a 1Gb connection at Equinix and all other uploads to that server are very fast. Also, FTP exports from the TransloadIT servers to their other FTP server (in a DataPipe colo) are going at normal speeds. So the problem seems to be specific to FTP exports going from TransloadIT to Equinix.
We think there is some kind throttling happening on the Equinix side or possible some peering issues happening between the Amazon EC2 network and Equinix, but we need more information.
What EC2 zone are the TranscodeIT servers located in?
Would it be possible for you to do a traceroute from one or more of your EC2 servers to cryptical.nugs.net so we could see the path the traffic takes?
Do you have any other customers reporting similar export speed issues?
Do you see any errors in the FTP exports that might indicate some kind of network transmission or FTP protocol issues?
Sorry for the late reply. Depending on the endpoint you use (or if you are really close to one particular cluster), you’re serviced from Ireland, Switzerland, or (most likely) US east.
Here’s a traceroute to the host you requested from a US east machine
$ traceroute cryptical.nugs.net
traceroute to cryptical.nugs.net (209.234.243.31), 30 hops max, 60 byte packets
1 ip-10-46-240-2.ec2.internal (10.46.240.2) 0.401 ms 0.426 ms 0.492 ms
2 100.64.145.224 (100.64.145.224) 0.961 ms 100.64.145.0 (100.64.145.0) 0.771 ms 100.64.145.96 (100.64.145.96) 0.988 ms
3 100.64.139.117 (100.64.139.117) 0.725 ms 100.64.138.35 (100.64.138.35) 0.977 ms 100.64.138.105 (100.64.138.105) 0.700 ms
4 100.64.142.146 (100.64.142.146) 1.056 ms 100.64.142.100 (100.64.142.100) 0.988 ms 100.64.128.30 (100.64.128.30) 1.187 ms
5 ip-10-1-173-104.ec2.internal (10.1.173.104) 0.458 ms ip-10-1-173-86.ec2.internal (10.1.173.86) 0.474 ms ip-10-1-3-12.ec2.internal (10.1.3.12) 0.527 ms
6 100.64.39.139 (100.64.39.139) 2.908 ms 100.64.37.209 (100.64.37.209) 1.191 ms 100.64.39.219 (100.64.39.219) 0.886 ms
7 216.182.224.74 (216.182.224.74) 1.366 ms 216.182.224.94 (216.182.224.94) 1.458 ms 216.182.224.80 (216.182.224.80) 1.369 ms
8 100.64.14.11 (100.64.14.11) 12.292 ms 100.64.8.213 (100.64.8.213) 16.736 ms 100.64.14.167 (100.64.14.167) 17.172 ms
9 100.64.0.62 (100.64.0.62) 1.374 ms 100.64.0.84 (100.64.0.84) 1.606 ms 100.64.0.64 (100.64.0.64) 1.430 ms
10 100.64.16.203 (100.64.16.203) 0.703 ms 100.64.16.9 (100.64.16.9) 0.523 ms 100.64.16.5 (100.64.16.5) 0.727 ms
11 72.21.220.225 (72.21.220.225) 0.905 ms 72.21.222.130 (72.21.222.130) 0.939 ms 72.21.222.34 (72.21.222.34) 0.681 ms
12 72.21.220.28 (72.21.220.28) 1.134 ms 72.21.220.44 (72.21.220.44) 1.043 ms 72.21.222.34 (72.21.222.34) 0.647 ms
13 xe-7-3-0.was14.ip4.tinet.net (173.241.128.49) 1.261 ms 72.21.220.36 (72.21.220.36) 1.112 ms 72.21.220.44 (72.21.220.44) 1.222 ms
14 xe-7-1-2.was14.ip4.tinet.net (77.67.68.153) 1.415 ms xe-9-1-0.lax20.ip4.tinet.net (89.149.181.166) 70.752 ms xe-7-2-0.was10.ip4.tinet.net (216.221.156.185) 0.747 ms
15 highwinds-gw.ip4.tinet.net (77.67.68.30) 63.858 ms 63.989 ms 65.483 ms
16 209.197.4.6 (209.197.4.6) 68.277 ms highwinds-gw.ip4.tinet.net (77.67.68.30) 65.409 ms 65.407 ms
17 209.197.4.6 (209.197.4.6) 64.441 ms te8-2-3580.bbr1.lax1.bandcon.com (69.16.190.214) 66.791 ms 209.197.4.6 (209.197.4.6) 64.720 ms
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
$
We haven’t had other customers complain. One thing to keep in mind is that since our platform is distributed and jobs are processed by many machines, as soon as their required input files are available, in parallel.
It’s possible some hosts consider that unnatural traffic and start rate-limiting, or even rejecting. We can’t really throttle this without implementing global locks / SPOFs, which we want to avoid.
So how we ended up dealing with rejects, is retries with exponential backoffs. This however seems not to be the case. So the traceroute, host throtteling, or bandwidth issues are more likely.
Thanks for all of this information - it’s been helpful, but we’re still experiencing the issue. One additional question - do you require passive or active FTP on the destination server?
For FTP imports you can configure passive_mode (true or false) to control passive behavior. For exports that is not implemented yet.
I just checked the code and we don’t have passive turned on atm. If you need that we can implement it via a -p flag which we could make available under the same passive_mode configuration option.
Thanks - let’s leave passive mode disabled for the moment.
Would you please do one more traceroute from one of your servers in Virginia to cryptical.nugs.net? I am gathering information for a conference call with Equinix for further troubleshooting.
Thanks for all your earlier help with this. The problem ended up being a peering issue between Amazon and Equinix. They have a direct peering connection but, for some reason, a temporary route was put in place by an Amazon peering engineer that circumvented that direct peering connection. Once they removed the temp route, everything went back to normal.
Unfortunately, the problem has reappeared as of a few days ago and Transloadit FTP uploads are going slow again. Would you please do another traceroute and send it along at your earliest convenience? I will send this information to Equinix so they can use it for troubleshooting.