Ask HN: How to build a resilient, scalable REST api that is open to the web?

You should probably consider the process design more than the technology (for now).<p>Goals:<p>1. Whenever receiving a request for processing, you should return a task identifier that can be used to check for progress (ex. you could simply store this in redis as 'TaskId', 'Url', 'FileId', 'Type', 'Status', 'OutputId' - with Type being the type of input: URL or FileId)<p>2. Once the client received the TaskId, it can do a request to get the status, which would could be 'Pending' or 'Complete' with an URL to the output.<p>Process:<p>===================<p>Request Processing:<p>-------------------<p>1. Receive the request with parameters (can users upload files or simply pass through an URL?)<p>2. Create a token based on the hash of the input file and the parameters supplied (ex. MD5 hash of the video, from time, end time of the video to capture).<p>3. Check if a task identifier has already been assigned to the token, if it has, return it. This means the client will then do a request to check for the status and would receive { 'Status': 'Complete', 'URL': '...' } and can immediately access the output via the URL.<p>4. If the video is uploaded, check whether it has been saved via hashing the file (ex. MD5), if it has then use the existing FileId, otherwise, save the file and generate a new FileId.<p>5. Generate a new task identifier and create a task entry to store in redis that contains metadata about the task.<p>6. Create a command containing the task identifier and publish it to the message queue for processing (which is asynchronous).<p>7. Return the task identifier to the client.<p>Task Processing:<p>-------------------<p>To process the video, you would simply grab the metadata from redis and update the progress as you go along. If the task identifier doesn't exist (no metadata entry on redis), simply abort the task processing.<p>Although this is pretty simple, you should focus on optimizing file storage (ea. don't store the same file twice and reference them by MD5 hash or the like and perhaps have a cleanup routine to remove rarely/once-off used files) and processing time (if the task has already been processed, simply return the output). You should also decide on limitations to prevent abuse (DOS/DDOS attacks, large file uploads, what the service can be used for etc.) and limit your liability.<p>Is this the answer you were looking for or are you asking about the specifics of implementation (technology wise)?

As soon as as a user types that into the browser, a few things will happen of course at that exact moment an http request is received by ________.<p>and then it will call iron.io ironmq or something like that, put in a request to the queue, fire off some ffmpeg scripts in the ironworker asynchronously, receive message that it's successful, send back to the browser animated gif that has been rendered.<p>During this time, the user will just have to wait, until the task completes. I'm not sure if I would want them to time out because what if (edge case), someone decides to convert one of those "Nyan cat loop for 10 hours" videos, the server is stuck in a long task.<p>Then comes the question of whether the same long process will continue if some other user wants to test out this exploit. That's where I think persistent storage of the processed file would come in handy. the request url should first be scanned if it's existing in one of our previous url requests.<p>Redis could be used replacing rabbitmq or ironmq. but I'm think for spike cases, not in a sense that the app will become hugely popular overnight (although making it on the first page of news ycombinator would awesome), but certain users either being passionate users making lot of requests and malicious individuals abusing it.<p>I really don't want to block the api behind the http auth dialog or a web based login.<p>I envision it to just work like you would use google, except you can convert youtube videos to animated gifs, just by entering in the url.<p>Of course in the backend, I'd need extensive monitoring and realtime analytics would be cool.