IOS 5's "Cleaning" Behavior

Lots of people are focussing on Marco's particular use case in the comments, and I think it's a valid one, but this extends beyond simple documents.<p>There is a category of data that is aimed at offline use. Streaming apps like Spotify, that let you download playlists for offline use. GPS apps that download hundreds of MB of map data. You get the idea.<p>On one hand, this data is a form of cache. The data <i>is</i> always available elsewhere (on the content provider servers) and it can be restored if necessary in a worst case scenario. But the key word here is "offline". This is the kind of data that, by definition depends on being around if the user is offline and therefore cannot be easily restored on demand, when the user needs it.<p>Obviously, having all of this stuff backed up to iCloud and using up GBs of people's capacity is not feasible or even logical. So this kind of data does not belong anywhere that iCloud will back up. But it must be stored somewhere that is safe from being purged.<p>Yes, a users GPS maps can be restored eventually but that doesn't help them when they are stranded in the middle of nowhere with nothing but a weak GPRS signal and all of their maps gone.<p>Apple have made an almighty cockup in overlooking the "offline data" use case.<p>In Marco's case, I'd agree that the articles represent user data that should be stored somewhere like Application Support, which will be backed by iCloud but I think that's probably fine in this case.

I'm sympathetic to Marco's blog post here, but I wonder if he's failing to interpret the two paragraphs from the documentation within the context of Instapaper's purpose.<p>With respect to point #1, and in the context of what Instapaper is for, the user is intending to read an article offline, and the local copy of the article was generated by the user's intent (and can, therefore, be considered user-generated even though the user is not the author of the article.)<p>With respect to point #2, the articles fail the "can be downloaded again" test in light of the app's purpose of making the articles available to the user when the network is not available. When the network is unavailable, the articles cannot be downloaded again. <i>Edit: JeffDClark makes another excellent point here that the article may have originally been behind a paywall and therefore cannot be guaranteed to be re-downloadable. The same is true if the author removed the original article from their webserver.</i><p>Ergo, put them in the Documents folder. Whether this would satisfy the app reviewer is another question, but it's worth a shot to carefully explain to them how your app is not violating the letter of the law.

I write several offline mapping apps, and this is totally throwing us for a loop. We're recommending our power users not upgrade to iOS 5. Users download gigabytes of maps to their cache directory, they don't want to eat their iCloud allotment with that, or their slow their iTunes sync. But they also don't want to have to download those maps again, or find themselves in the middle of the woods without the maps they downloaded.

<i>Apple - iCloud - Your content. On all your devices.</i><p>That is the title of Apple's main iCloud page at <a href="http://www.apple.com/icloud/" rel="nofollow">http://www.apple.com/icloud/</a>.<p><i>iCloud is seamlessly integrated into your apps, so you can access your content on all your devices.</i><p>This is Apple's definition for the iCloud service. It doesn't matter what the data is, it's <i>your</i> data and Apple is promising to sync it between your devices, to preserve your experience.<p>In the case of Instapaper, the solution is obvious: Put the files in Documents. That is Instapaper's content and part of the experience that users want synced between devices.<p>If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready.

Apple's in a tight spot here - either they leave things as they are and piss off developers (and, by fiat, piss off users), or they have to start forcing users to more proactively manage their remaining space.<p>One can probably easily imagine an interface for showing the user how much data a particular app is using and allow them to nuke the temporary stuff. It might even look beautiful. It might even be fun to use. But it's going to introduce a lot of hand-wringing and micro-managing and lots of mental overhead that Apple really, <i>really</i> wants to avoid.

Add .nosync to the file/folder name path and keep it in the home or documents directory. Problem solved.<p>edit: I'm shocked this thread is so long and no body mentioned this. It's been on the apple developer forums for months as a solution.

There are a number of possible scenarios for file storage, the problem is a lack of clarity or documentation about the properties of the various locations as they stand now. As a developer, I could imagine desiring the following choices:<p>1. Temp: No backup, cleared regularly<p>2. Cache: No backup, cleared when space is tight<p>3. Local: Local backup only, never cleared<p>4. Documents: Local/cloud backup, never cleared<p>5. Cloud: Cloud backup, cleared when space is tight<p>The problem seems to be that #3 doesn't exist. Yet you'd think it would be a common requirement for stuff like in-app purchases of large and essential content packs, for example, turn-by-turn navigation maps.<p>I'd hate to be on holiday and have a 10 megabyte podcast download automatically trigger the erasure of 1000 megabytes of navigation data.

I don't understand Apple's logic here. You can't reconcile the idea of "cleaning because you can redownload" with "available offline". As soon as you clean up, you are going to break offline uses.<p>I would guess the idea is to help enable these 500 MB per issue magazine downloads. You download a new issue, you nuke some old issue, no one cares. As long as that wasn't an issue you cared about.

The whole idea of an app like Instapaper (or any of the other examples presented) is that the "stuff" that is saved for later is all user-generated content. Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache.<p>It seems that the argument that only the list of metadata is user-generated can apply to any type of media (music, movies, etc...). Technically speaking all of the music on my phone could be downloaded on demand. Of course this requires an always connected, fat, and cheap network connection. Which is pretty much the opposite of what most folks have.<p>This also breaks the user's expectations. I was annoyed/surprised when I upgraded to iOS 5 and Instapaper had to re-download everything.

The first time that one of my several hundred megabyte foreign language dictionary files isn't available and needs to be re-downloaded when I need it in say, a meeting will trigger a severe re-evaluation of my use of the phone.

I'm definitely at risk of running into this predicament as a user. I'm more worried about the hundreds of megabytes of podcasts I download when WiFi is available for use throughout the day when WiFi isn't available.<p>To avoid this issue and enjoy the benefits of iOS 5, I'm going to have to clear out a bunch of apps, music, and photos and ensure that I always have a sufficient amount of "buffer" space so the cleaning is never triggered. I cannot be alone in this, and the fact that Apple is making this kind of thinking necessary for end users is kind of ridiculous. I really can't see this behavior lasting for very long, and I'm sure Apple will address it soon; this is the antithesis to the traditional iPhone experience. The only scenario where I could see this being purposeful is if Apple is really trying to hurt offline apps to increase data usage and appease carriers (maybe for pissing them off with iMessage?).

Seem like the Dropbox app will have this quandary but on an even larger scale.

It seems fair to say that Instapaper's version of an article can't be "redownloaded" for various reasons (offline, paywall, article removed, etc.) so it would be OK to put it in the Documents folder.<p>The argument against that is that you're now syncing that article with iCloud in addition to Instapaper.<p>But I wonder whether the correct answer is instead to eliminate Instapaper's sync feature, and just let iCloud do it. Once you have system-level cloud sync, don't you want to let Apple do the work? Sync is hard, and it isn't really the core value of Instapaper.<p>Edit: I was wondering about iCloud only for iOS 5/MacOS 10.7.2 devices with iCloud accounts. But that story does fall apart for people with mixed devices. So never mind.

Isn't NSApplicationSupportDirectory for stuff that isn't "Documents" and yet needs to be managed as app state (not in 'Caches')? Why not just use that instead? That's what my app does and it seems to be OK with iOS 5.

I wouldn't say that the articles' metadata (URL, title, date of download, maybe thumbnail of the 1st page) is downloaded content. It's clearly user-generated, and should be in the "home" of the app IMHO.<p>The articles themselves, yes, send them to the cache. If the user needs to reclaim the storage used up by the articles, let the OS delete them. Then, when the user needs to read the article again, it will take some time to download. But don't get into an "all articles gone" situation. Just my 2 cents.

Has anyone figured out where the high water mark is? When will the cleaning behavior kick in?<p>This is an interesting twist especially with the 16GB devices which tend to actually be above 50%.

What about apps like spotify and rdio? Where do they store music if those two directories are constantly cleaned?

I develop Definition (<a href="http://definitionapp.com" rel="nofollow">http://definitionapp.com</a>) and I store the database in the Caches directory as well, after receiving the email from Apple to move. This is bad news for me, the entire offline dictionary could be wiped out.

Maybe the solution is to make it a choice for the user? An option to "clean up documents when space is low on the device" in the instapaper options. If checked, stuff gets stored in cache. If not, in documents.<p>That way, the default behavior is that "download something = want to keep it on device", but users can do the other one too if they want. I don't think the option is particularly useful, but it might make the app reviewer happy?

Is it possible that a user could use Instapaper to save a document that they wouldn't be able to download later, even if they had access to an internet connection? If that's possible, I think Instapaper would have every right to store things in the Documents folder, since you're asking it to create something more akin to an archive than a temporary cache of data.

I think the problem is that Marco sees Instapaper users as his customers. I sympathize with this, but I think iOS is an assertion by Apple that App Store customers are Apple customers. App Store Developers are providing a service to Apple's customers. But everything that happens must point back to Apple's servers, not their own.<p>In the same way Apple doesn't want every developer operating their own independent payment system, they also don't want developers operating their own cloud storage services. If Apple holds the data they can guarantee its security. They likely see these problems as temporary, until developers and customers learn to adjust to the fact that iOS data is controlled by the iCloud service.

It sounds like what Marco is looking for is something equivalent to Windows's AppData\Local, machine local, doesn't get nuked, but doesn't sync over roaming profiles either

manage it yourself: put things that need to be persistent in Documents. put the rest in Cache. move 'em as needed. do it automatically by download and access dates and/or provide an interface for people to manage it.<p>your app absolutely needs tons and tons of data to function? doesn't seem like your day. it's their device, their cloud, their decision. Apple doesn't give a shit about your day; they're going cloud. they may be wrong, but i'd guess they're going to have to find that out for themselves.<p>i'd assume they acknowledge this may kill some apps. i don't think they ever promised anyone a business; on the contrary, they seem to remind developers that they are there at their good grace all the time. as someone that built Facebook apps since '07, trust me, i know what this is like. start coding and start calling. best of luck.

DHH declared the solution in 2007 [1] - apparently nobody needs offline applications!<p>[1] <a href="http://37signals.com/svn/posts/347-youre-not-on-a-fucking-plane-and-if-you-are-it-doesnt-matter" rel="nofollow">http://37signals.com/svn/posts/347-youre-not-on-a-fucking-pl...</a>

I must be missing something. What is the problem with backing up pages stored in InstaPaper to iCloud? That's exactly the behavior I would want if I were using InstaPaper.

That's pretty interesting. So, the three non-backed locations are tmp, Caches and the Application Bundle. tmp and Caches are now swept. So, if you store your real cache stuff in the Application Bundle, it won't be backed up or auto-deleted. But maybe it gets trashed when you update the app.<p>I wonder if OS X, in line with its trend to be more like iOS, is going to start automatically clearing the ~/Library/Caches directory as well.

<i>There needs to be a file storage location that behaves the way Caches did before iOS 5</i><p>I'm confused. If the goal is for documents to be available in the near future in offline form, why not keep documents in /Documents until the user has read them (or some sane amount of time has passed), and then move them to /Caches or some temporary storage?<p>I've never used Instapaper, so perhaps documents are only stored until read anyway?

This is also particularly problematic for my own apps, which are offline photo browsers that sync your collection of photos. Keeping GBs of data in the Caches folder was the only way to have iPhone backups occur reasonably fast. Sure, they can be re-synced, but that will be a very time-consuming process for thousands of photos.

Clearly, apps that cache data need to be modified to show the difference between having no data and having no data cached. IMAP clients have dealt with this, for example; they show a message like "the contents of this folder are not available offline".<p>Perhaps Apple could have made cache cleaning opt-in on a per-app basis until iOS 6, though.

Could you use NSUserDefaults? I don't think there is a limit on the data stored there.

So, people (developers) are going to have to put their stuff in Documents, and instruct users to disable the iCloud sync for their particular app, unless the user really wants to have it eat into their iCloud storage.<p>Would that work?

Or worse: <a href="http://blog.foreflight.com/2011/10/14/flying-with-foreflight-and-ios-5/" rel="nofollow">http://blog.foreflight.com/2011/10/14/flying-with-foreflight...</a>

What about offline web apps which are saved onto the home screen? Are they also "cleaned"? If we can't depend upon an offline web app to be there when offline then that would fundamentally defeat the purpose.<p>Of course, one could argue that the cleaning behaviour fundamentally defeats the purpose of a lot of apps, as already covered above (Instapaper, Offline Maps, etc.).<p>My iMac is packed up for building work, so I can't upgrade my iPad and check this out for myself. Sorry.

"Prior to iOS 5, the system never deleted the contents of Caches and tmp, so they were safe places for apps to put data that should always be available"<p>Not exactly.

He says he knew about this behavior, and he also explicitly said (on Twitter, can't be bothered looking for it) that he was not going to report bugs to Apple during the beta.<p>I guess Marco just prefers venting after the fact.

I dont get it.. he get furios because he saves files he needs in a folder called /cache , and /cache gets purged when the system runs low on space.... What did you think cache meant?<p>Why dont you just update your app?