Archived from the Transloadit support forum. Originally posted on January 14, 2014; details may be outdated, see our docs for current behavior. Original thread
Hello,
I’m sysadmin and we are working with Transloadit.
We have a production environment and an staging environment. But recently we found that many people can access the staging environment and be able to see things we don’t want to be seen.
Our site runs an Apache so I worked with the “Allow from” directive to restrict the access to the staging environment. Most directives are IP based.
The source code of our app, sends files to Transloadit with an URL in the metadata. When Transloadit finishes the convertion, hits that URL. So the app registers the “ready” state of the convertion.
The problem is: The Transloadit IP is not allowed by Apache. We don’t know if these IPs are statics or dynamics. (right now notifications are from 54.237.26.210). If we allow a group of IPs, our app couldn’t be operative when you change it or create a new one.
A temporary solution is to allow access from everywhere to that notification URI, but it’s not a correct solution.
The best way, is to allow an IP, or a range, or a domain, or a subdomain.
I noted that TransloadIt uses Amazon’s services . So I suggest that you may send the notification from a determined URL. IE: server14.responses.transloadit.com (with an architecture without an ELB and in a way that your platform may change without major modifications)
In that case, we could Allow access from *.responses.transloadit.com
You could set the Route53 register very easily
Please let me know the resolution of this issue and if I can help in any way.
This question sometimes surfaces and we have covered some possible solutions, each with their own caveats here: https://transloadit.com/docs/faq#can-i-whitelist-transloadits-ips-in-my-firewall. Most firewalls we’re dealing with are IP based so having correct reverse dns records set up for all our outgoing traffic would still not solve our problem in most cases.
Additionally, having domains in an .htaccess would require dns lookups for every incoming request, that can quickly build up. If your dns server ever is unresponsive server load will soar. Something to keep in mind.
Just thought of something that could work in your case.
I guess if you make the notification url run under HTTPS and you include a hash in it that people won’t be able to guess, you could safely whitelist just that url while traffic for other urls/people will still be denied except for your office/co-workers.
Well… the problem is that we don’t want to whitelist our own URLs. Just IPs. Because.. you KNOW how it is… In a while, somebody knows that and ask for “another”. When you whitelist one URL, you start an era of destruction and desolation with humongous and unmaintainable sheets.
Perhaps you could take advantage of the Amazon infrasrtucture and set an instance as “instances monitor” that publish which EC2 instances are running as a simple text file hosted in Apache.
Every user could read it and update their whitelist in an automated process
(we use something like that made with Bash. Actually I made it work )
May be my proposal looks complex but it’d be as simple as a crontab with an SH running a line like this: ec2-describe-instances | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}"|grep -v "^10\." > whitelist.txt
Explained:
ask Amazon for every instance
take (grep) only (-o) the part of the string that looks like an IP
remove (grep -v) the lines starting with “10.” because they are internal IPs
write the result in a file (whitelist.txt)
Again, PLEASE let me know what you think.
Kind regards,
Juan
Thanks for your suggestion and fleshing out the actual commands! Our platform is very volatile however so the list will never be up to date in realtime and you’re guaranteed to miss out on communication, so not an airtight solution.. What do you think?
What do you think about all the other options provided in the f.a.q.?
Well, I think the “volatile situation” could and should be controlled.
As a better idea (I’m thinking while I type)… I don’t know how you manage the infrastructure, but … for example, if you use AutoScalling, you can take advantage of the --user-data-file in the creation of the launch configuration. To put a “registration procedure” inside ensures the service registered before start the operation.
The registration procedure could be as simple as a wget command hitting an internal web server. The webserver having (for instance) PHP, can register the $_SERVER[‘REMOTE_ADDR’] in a whitelist.txt
Another way could be to setup al Alarm that triggers when an EC2 instance starts (I’m not sure if exists this alarm in CloudWatch). May call an SNS service in a webservice that register the instance in the whitelist.
If none of this (brilliant) solutions can be made, I think I’m gonna suggest the “hashing url” method to the development department. But I’m pretty sure we cannot do it because of the internal architecture
Allowing the entire range from Amazon is a security breach by itself. Have in mind that, generaly, most attacks are from the inside of Amazon. Users knows that Amazon have great power and most clients doesn’t protect their instances from other computers in the cloud that doesn’t belong to them.
I’ll wait for your answer before proceed
Kind regards,
Juan
By volatile I mean we are running on 100 server today and tomorrow there will be 8, different ones. We control that.
Even if we have servers push their curl wgetip.com to s3 and we aggregate that into one file and you download that, you will be to late. Catch my drift? We cannot make this realtime.
As the faq mentions whitelisting amazon is not secure, but can be a compromise for some people. If you are really are concerned about security why not host a machine outside of your DMZ and pull updates from that? (or whitelist that one ip and forward traffic)
Hi Kevin,
Yes. I understand that the service could answer before we read the whitelist file and therefore we would not allow the communication and finishing by losing the notification. I got you now.
The DMZ could be a solution. I confess I saw that paragraph in the FAQ just after send the previous message. I stoped reading before reach that point. I’m very sorry about that.
)