TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Serverless File Uploads – Netlify

51 pointsby peterdeminover 8 years ago

12 comments

mnuttover 8 years ago
This article makes it sound like upload requests are restricted to your domain via CORS, but that is one of the pitfalls of thinking about everything through a serverless lens: while another site could not use javascript to directly upload files to your s3 bucket, a malicious user could absolutely make a backend request to receive signed tokens for uploading to your s3 bucket. Additionally, it doesn't look like the policy is locked down so they could overwrite your existing files with malicious ones.
评论 #13639797 未加载
评论 #13647171 未加载
mamispover 8 years ago
How is using a node.js server to generate signed requests to upload files onto S3 servers, "serverless"?
评论 #13640302 未加载
评论 #13639629 未加载
评论 #13641524 未加载
评论 #13639857 未加载
评论 #13639690 未加载
评论 #13639849 未加载
评论 #13639626 未加载
chrisballingerover 8 years ago
We built something similar to this for our Kickflip.io HLS&#x2F;S3 live video streaming service. The original version is closed source but we rewrote a generic open source Python library called storage_provisioner [1] for a client. It&#x27;s minimal and super simple.<p>We also made a Django-Rest-Framework module that wraps storage_provisioner called django_broadcast [2]. Working with AWS&#x2F;S3 can be a pain, hopefully these tools can help.<p>1. <a href="https:&#x2F;&#x2F;github.com&#x2F;PerchLive&#x2F;storage_provisioner" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;PerchLive&#x2F;storage_provisioner</a><p>2. <a href="https:&#x2F;&#x2F;github.com&#x2F;PerchLive&#x2F;django-broadcast" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;PerchLive&#x2F;django-broadcast</a>
ameliusover 8 years ago
Silly nomenclature. This isn&#x27;t &quot;serverless&quot; at all.<p>Suggestion: &quot;File uploads using 3rd party server&quot;.<p>As an added benefit, this title shows that the data might not be safe from the curiosity of other entities.
评论 #13639843 未加载
评论 #13639880 未加载
评论 #13639813 未加载
评论 #13640197 未加载
shakeel_mohamedover 8 years ago
What&#x27;s wrong with doing all of this on the front-end? I recently did just that after generating the signature and policy locally.<p>See this guide: <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;articles&#x2F;1434" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;articles&#x2F;1434</a>
评论 #13639902 未加载
评论 #13639858 未加载
mrmondoover 8 years ago
I really wish people would stop using the term serverless, it&#x27;s not useful and it&#x27;s highly misleading.
danielrhodesover 8 years ago
You do need a server to create a token from your access key and secret. However, this doesn&#x27;t really go very far in protecting your bucket, as somebody could just grab that token and upload whatever they want.<p>So an additional layer of security is creating an upload bucket with a policy where all objects over 24h old are deleted. When somebody finishes uploading a file, you ping your server and move the file from the upload bucket to the real bucket.<p>Another trick is putting Cloudfront in front of that bucket. You can then upload to any Cloudfront server, which will then put the file in your bucket -- the reduced latency to a Cloudfront (vs S3) will increase the speed at which you upload by quite a bit.
cyberferretover 8 years ago
Nice work. I have several web apps that run on EC2 instances, that have my users uploading via Browser -&gt; EC2 -&gt; S3. This can cause high latency on some of my smaller EC2 boxes, which is annoying, it else forces Elastic Beanstalk to spool up more instances unnecessarily when it thinks traffic is being flooded when large files are being uploaded.<p>I&#x27;ve always wondered about the best strategy to go &#x27;serverless&#x27; with the file uploads and have the user&#x27;s browser essentially upload direct to S3, and this tutorial gives a great insight into that - thanks.
curiousAlover 8 years ago
&quot;Each returned URL is unique and valid for a single usage, under the specified conditions.&quot;<p>Where and how is the url actually invalidated after it is used? (or are you relying on expiration as invalidation?)
jest3r1over 8 years ago
&quot;Serverless&quot;<p>So is this 100% client-side, or is there a (server) dependency?
asciimikeover 8 years ago
What I&#x27;ve found people really want when they say &quot;serverless&quot; in this context is, &quot;direct file upload without a proxy server&quot;, which is basically what BaaS&#x27; like Firebase and Parse do...<p>&lt;pitch&gt;<p>Firebase Storage (<a href="https:&#x2F;&#x2F;firebase.google.com&#x2F;docs&#x2F;storage&#x2F;" rel="nofollow">https:&#x2F;&#x2F;firebase.google.com&#x2F;docs&#x2F;storage&#x2F;</a>) provides clients that perform secure, serverless uploads and downloads. Instead of doing the dance with a server minting signed URLs, it uses a rules engine that lets developers specify declarative rules to authorize client operations. You can write rules to match object prefixes, restrict access to particular user or set of users, check the contents of the file metadata (size, content type, other headers), or check a current file with the prefix to not overwrite it.<p>If HackerNews formatting were more forgiving of code snippets, I&#x27;d post one here, but instead have to link to the docs (<a href="https:&#x2F;&#x2F;firebase.google.com&#x2F;docs&#x2F;storage&#x2F;security&#x2F;secure-files" rel="nofollow">https:&#x2F;&#x2F;firebase.google.com&#x2F;docs&#x2F;storage&#x2F;security&#x2F;secure-fil...</a>).<p>We&#x27;ve found that this model is more performant and less expensive (no need for a proxy server), as well as lower cognitive load on developers, as they think about what they want the end result to be, rather than how they need to build up the end result.<p>And since I know people will bring it up: there are definitely limitations in flexibility (you&#x27;re using a DSL), and a steeper learning curve for the very complicated use cases. The goal here is make it trivial for 90% of use cases and possible for 9%; rather than making it possible for 100% and equally difficult for everyone. Tradeoffs...<p>&lt;&#x2F;pitch&gt;<p>And if you want other examples, Parse did a similar thing with role based access control to a Parse File, allowing direct client upload and access by only a set of users. S3 and GCS can do this as well, assuming their (relatively coarse) IAM models are granular enough for you (and you&#x27;re a authorized principle in their systems, which is often the harder thing).<p>Bringing this full circle, &quot;serverless&quot; typically involves a switch from writing code (imperative) to writing config (declarative). You&#x27;re not validating JWTs signing URLs, or writing middleware, you&#x27;re letting services know how to configure those primitives for you. In some ways the Serverless framework does abstract this for you (hey look, I didn&#x27;t provision a VM), and in some ways it doesn&#x27;t (you still wrote code to generate a signed URL).<p>Disclosure: I built Firebase Storage
评论 #13640585 未加载
madmodover 8 years ago
I use this technique extensively in several production systems.<p>As others have mentioned having an expiration policy is a good idea. Also you can mitigate charges from malicious activity by using rate limits on the signing endpoint. (API gateway supports this.) Using infrequent access or reduced redundancy storage might also be a good idea if you expect a lot of traffic. It&#x27;s also good to limit the CORS policy on the bucket to the needed domains and headers.<p>Signed metadata headers are very useful when combined with S3 event handlers (SQS or straight to Lambda.) using a HEAD request on the uploaded objects. This is a great technique for post processing an upload without requiring client trust or an external data store. (With a separate falliable request which could lead to consistency issues.)<p>Edit: It is also critically important to have some randomness in each key path so it is ungessable. Otherwise user files would be overwriteable by an attacker. (Many file names are easily guessable and an attacker with many tries could eventually stuff malware in for example.) I used guids for this because they are both URL and S3 key safe. If keeping the original file name is needed I put it in a metadata value and rename the file on download using a Content-Disposition header. Making the S3 headers work with symbols in file names can be tricky but encoding it as a JSON string works around most issues.<p>In order to overcome the 30 second request limit in API gateway for longer post processing while still offering realtime client feedback you can set up an S3 event handler to trigger the post processing lambda which then updates a DynamoDB record with the S3 key as it&#x27;s id. A status endpoint lambda is then polled by the client with the S3 key for status events.<p>For more complex post processing and client side workflows I have used key prefixes (folders) each with seaprate event handlers or CORS configurations. IAM polices with conditions including S3 key prefixes are used to restrict access. Using the S3 API copy command can move large objects quickly between workflow steps.<p>Also enabling server side encryption is a must imo. Be sure to specify AWS signature version 4 in the S3 constructor so that all parts of the request are signed. (Otherwise some older regions may not sign metadata headers.)<p>Also the S3 API copy command has an interesting append feature which can be used to build objects iteratively. I once toyed with the idea of using it to create large zip files of many S3 objects efficiently but ended up not needing it. Someday I would like to try that because it could be great for a lot of web apps where users can select a random list of files to download.<p>Also I (re)implemented most of the above this week using CloudFormation and the newer AWS Serverless Template (not the serverless.com project but the actual AWS feature.) which allows for really easy deployment.