TE
TechEcho
Home24h TopNewestBestAskShowJobs
GitHubTwitter
Home

TechEcho

A tech news platform built with Next.js, providing global tech news and discussions.

GitHubTwitter

Home

HomeNewestBestAskShowJobs

Resources

HackerNews APIOriginal HackerNewsNext.js

© 2025 TechEcho. All rights reserved.

Google App Engine's Datastore Admin is Terribly Inefficient

48 pointsby marramover 13 years ago

8 comments

peterknegoover 13 years ago
A GAE datastore delete takes multiple operations because it also updates indexes:<p>1 entity delete = 2 Writes + 2 Writes per indexed property value + 1 Write per composite index value<p>All from this page: <a href="http://code.google.com/appengine/docs/billing.html#Billable_Resource_Unit_Cost" rel="nofollow">http://code.google.com/appengine/docs/billing.html#Billable_...</a> And more about why it is so: <a href="http://code.google.com/appengine/articles/life_of_write.html" rel="nofollow">http://code.google.com/appengine/articles/life_of_write.html</a><p>Well, the OP is just another coder who can't read docs, but can write a blog.
评论 #3394797 未加载
评论 #3394301 未加载
stickfigureover 13 years ago
This is something that a lot of GAE developers misunderstand: put()ing a datastore entity is not a single write operation. There are indexes to update - in your case, lots of them - and updating these indexes can require several write operations. A simple delete is one write per index but changing a value can be two operations; one to delete the old index value and one to write the new one. And since each property has two indexes (ascending and descending), these numbers are X2.<p>If you create your own bulk delete method, you will find that it takes exactly as many write ops as the admin console tool.<p>You probably have defined more indexes on your entities than you need to - you will likely be able to make your app cheaper by removing unnecessary indexes. Managing indexes carefully is a critical part of making apps affordable on GAE.
评论 #3394515 未加载
latchkeyover 13 years ago
Should Google refund developers when they make an uninformed decision that costs them money?<p>One could argue it is a bug in GAE that allows developers to make an expensive mistake when they don't fully understand how something (fairly complicated) works.<p>Someone else could argue that we are all developers and we should know the costs associated with the systems we are building. There is a real cost associated with PaaS systems like GAE.<p>What do you think?
cr4zyover 13 years ago
I'm pretty sure this uses the map reduce API which has a lot of overhead in the datastore. In principle map reduce is nice because it could make very large jobs fast. But since Google engineers don't pay for anything, they optimized for time, not cost.<p>And with regards to your script, you can't just delete 3k keys in one request. If you want I'll send you the script I've adapted for jobs that make large changes to the datastore.
评论 #3394097 未加载
评论 #3394048 未加载
ch0wnover 13 years ago
I ran into the same issue. If you want to purge all data from an app, it's much cheaper (and sometimes even faster) to start over and create a complete new app with an empty data store than to use the data store admin and delete the data from there.
ecksorover 13 years ago
number of writes also depends on the number of indexes you have on the data
Maven911over 13 years ago
Can somebody explain the article in laymen terms ? For those not too familiar with GAE...
评论 #3394532 未加载
tnucover 13 years ago
There are plenty of things that are wrong with Google App Engine. And there are plenty of bugs that exist that have cost me money.<p>Why don't you try filing a bug report/suggesting a warning and send an email requesting something of a refund. They tend to be a friendly bunch who give refunds to obvious problems.<p>Moving to AWS will of course save you lots of money in the longer term, depending on what your hosting requirements are.