Deleting an S3 Bucket Costs Money

You can use lifecycle policies to delete it for free, but its best to confirm it via support. Not saying this is the great way, maybe its intentionally hidden, but at least there is a way.<p><a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;59170391&#x2F;s3-lifecycle-expiration-do-object-expiry-deletes-cost-money-for-sia-objects" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;59170391&#x2F;s3-lifecycle-ex...</a>

Not only is a problem that deleting a bucket costs money, but if you have a big bucket with many deeply nested files, it can take a really long time to clean it up using the AWS command line.<p>I ran into this with a bucket full of EMR log files a few years ago and had to figure out some pretty crazy command line hackiness, plus running on a EC2 machine with lots of cores to figure it out. This a write-up I did if anyone else ever runs into this issue.<p><a href="https:&#x2F;&#x2F;gist.github.com&#x2F;michael-erasmus&#x2F;6a5acddcb56548874ffe780e19b7701d" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;michael-erasmus&#x2F;6a5acddcb56548874ffe...</a>

Per-object costs can be tricky with S3 -- it&#x27;s easy to mentally round costs less than 1&#x2F;10th of a penny to zero, and then look up a few years later and realize you have hundreds of millions of things and can&#x27;t afford to do anything with them.<p>When this bit us on a project I made a tool to solve our particular problem, which tars files, writes csv indexes, and can fetch individual files from the tars if need be.[1] Running on millions of files was janky enough that I also ended up scripting an orchestrator to repeatedly attempt each step of the pipeline.[2] Not tested on data other than ours but could be a useful starting point.<p>[1] <a href="https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;s3mothball" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;s3mothball</a> [2] <a href="https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;mothball_pipeline" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;harvard-lil&#x2F;mothball_pipeline</a>

And deleting your AWS account will keep billing you [1] if you don’t delete all resources first.<p>AWS is designed to extract dollars from big enterprise contracts.<p>Also interesting from the article, this poor soul on StackOverflow was trying to figure out how to delete a bucket that would cost him $20,000 [2]. Can’t delete, can’t close.<p>[1] <a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;aws&#x2F;comments&#x2F;j5nh4w&#x2F;ive_deleted_my_account_but_amazon_keeps_billing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;aws&#x2F;comments&#x2F;j5nh4w&#x2F;ive_deleted_my_...</a><p>[2] <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;54255990&#x2F;cheapest-way-to-delete-2-billion-objects-from-s3-ia" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;questions&#x2F;54255990&#x2F;cheapest-way-to...</a>

Pricing of AWS services makes me uneasy in general, just take the S3 as an example - you go to the pricing page and you have several tabs with dozens of entries which makes calculating how much exactly will you pay difficult. I might be simple minded but I prefer a clearly defined plans with predetermined limits - you know exactly what it costs you each month and what you get and if you need more, just switch to a higher plan, no risk of nasty (and often expensive) surprises like mentioned in the article.

Yup. And uploading &#x2F; downloading large objects from S3 incurs tons of requests because S3 client does parallel chunking with a small number of other control requests. That client works on the same premise as SFTP client.<p>It’s amazing how often it retries.<p>Example from go sdk: <a href="https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-go&#x2F;blob&#x2F;main&#x2F;service&#x2F;s3&#x2F;s3manager&#x2F;download.go#L303" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;aws&#x2F;aws-sdk-go&#x2F;blob&#x2F;main&#x2F;service&#x2F;s3&#x2F;s3man...</a>.

The first thing everyone who tries using cloud services should learn: <i>everything</i> costs money. Even the service that tells you how much it costs: <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;aws-cost-management&#x2F;pricing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;aws-cost-management&#x2F;pricing&#x2F;</a>

This post finally got my ass in gear to cancel an account that I thought I had closed but was still charging me a few dollars a month.<p>I spinned up an AWS instance to practice, and once I was done I thought I closed everything down.<p>Turns out I had just stopped my micro instances, and I didn&#x27;t terminate them. I also hadn&#x27;t released the my IP address. There was also a snapshot of the tiny db I had created still floating around. The documentation was a little confusing, so after I went through it I spent half an hour chatting with a support rep to make sure everything was completely good. After next month my last bill should go through and I should be free and clear. Unfortunately I have to wait for next months bill to go through as I can&#x27;t just pay it all now.<p>This was mostly my fault for letting it go on for so long, but I hate how if you don&#x27;t do some very specific steps you can still be charged. And I think if an account is closed, it should absolutely terminate all services that are still running on that account, and then send you the final bill.

How much of this is a problem in practice?<p>I think in practice, S3 data is often indexed using other DBs e.g DynamoDB, Postgres, MySQL etc. Can&#x27;t this index be used to enumerate all S3 URLs? I am off-course simplifying this a lot.

Stories like this make me extremely hesitant to try AWS. I was about to try S3 for a static site I was working on this weekends but I think I am gonna stick with netlify or digital ocean instead after reading this.

&gt; .5¢ per 1000 items LISTed seems insanely expensive considering how cheaply you can transfer terabytes of data with S3.<p>Correction: I misread - .5¢ per 1,000,000 items LISTed<p><pre><code> .5¢ per 1000 LIST operations LIST operations max out at 1000 items </code></pre> Still a little pricey, but way less so than I&#x27;d imagined.<p>Do they make a lot of money off of charging for basic operations? It seems like you could make the whole pricing structure a lot more friendly by only charging for bandwidth use. I guess when you&#x27;re as dominant as S3, you don&#x27;t need to care about friendly pricing structures.<p>Charging for basic operations like that is weird, it&#x27;s akin to a service charging people per number of clicks on a website.

&gt; In 2021, anyone who comes across this question may benefit to know that AWS console now provides an empty button.<p>source : <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;67834172" rel="nofollow">https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;67834172</a>

&gt; you can also get an export of all objects in a bucket using S3 Inventory and run the output through AWS Batch in order to delete those objects<p>&quot;S3 Batch Operations&quot; sends S3 requests based on a csv file, which can but does not have to be from S3 Inventory. But S3 Batch Operations supports only a subset of APIs and this does not include DeleteObject(s). [0]<p>An AWS Batch job could run a container which sends DeleteObjects requests but only when triggered by a job queue which seems redundant here.<p>If I can&#x27;t use an expiration lifecycle policy because I need a selection of objects not matching a prefix or object tags, I would run something with `s5cmd rm` [1]. Alternatively roll your own golang which parses the CSV and sends many DeleteObjects requests in parallel goroutines.<p>0. <a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AmazonS3&#x2F;latest&#x2F;userguide&#x2F;batch-ops-operations.html" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AmazonS3&#x2F;latest&#x2F;userguide&#x2F;batch-...</a><p>1. <a href="https:&#x2F;&#x2F;github.com&#x2F;peak&#x2F;s5cmd#delete-multiple-s3-objects" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;peak&#x2F;s5cmd#delete-multiple-s3-objects</a>

They have an example of some person almost paying $20k on transition fees. In my early days of AWS, I racked up $90k on S3 transition fees. Thankfully, AWS forgave it.

Would the S3 inventory help here? That would allow you to get the list of all files (albeit on a delay similar to the lifecycle rule approach), which you could process offline to generate the DELETEs.<p><a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AmazonS3&#x2F;latest&#x2F;userguide&#x2F;storage-inventory.html" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;AmazonS3&#x2F;latest&#x2F;userguide&#x2F;storag...</a>

DELETE is free LIST is not<p>I guess one could spam DELETE calls while bruteforcing filenames to make it free.

Ok pretty obvious, but if you don&#x27;t know what you are storing inside your bucket, how are you accessing your objects in the first place ?<p>If your use-case is storing random things you don&#x27;t know the path of, maybe it&#x27;s the wrong product to use.

&gt; AWS is &quot;eventually consistent&quot; within most services, and S3 is no exception<p>Nowadays, it (¿almost?) is. <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;consistency&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;s3&#x2F;consistency&#x2F;</a>:<p><i>“After a successful write of a new object, or an overwrite or delete of an existing object, any subsequent read request immediately receives the latest version of the object”</i><p>I think that says that deletes are immediately visible, too, but they phrase it weirdly, as, after a delete, there is no latest version of the object.<p>Also, I don’t think buckets are objects in this sense, so the caveat in the article stands.

&gt;<i>The wait is often hours until AWS released a bucket name (since bucket names are globally unique, not just within your account).</i><p>I think last time I did this, the wait time was pretty much exactly 60 minutes.

This kind of thing is the best argument for your own bare metal hardware.

Anyone have suggestions for S3 alternatives for storing many files sized 50-500mb each? They are mostly long audio files and there is an external index as well.

Its silly that they wont just let you delete the whole bucket but this actually pretty cheap tho.<p>Based on some quick maths, deleting a million files would only cost you like $5.<p>P.S. Again its silly they do this and I&#x27;m probably greatly underestimating how these costs can add up for mid to large orgs.

&gt; Deleting a bucket won&#x27;t let you re-create that bucket immediately.<p>This is partially incorrect. I can recreate it immediately in the same account, but in different account, I need to wait for ~1 hour

Another reason to switch to R2.

If s3 was any service the general public use, it would be banned for sure.

Most things on AWS cost money and AWS makes pricing incredibly complex and opaque...where the monthly bill is usually the first way people find out about these things. While it is likely no consolation, S3 is by far one of the most complex AWS products pricing-wise with different object storage types each with their own rates, request costs with different rates for GET&#x2F;PUT&#x2F;POST&#x2F;etc which this post mentions, and transit&#x2F;egress fees.<p>I work on <a href="https:&#x2F;&#x2F;www.vantage.sh&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.vantage.sh&#x2F;</a> which helps teams get visibility on their cloud costs which may be helpful to folks here as well on this topic.