Hi all, Arash from Dropbox here. We understand the concern that the government could try to guess whether a particular file has been uploaded to Dropbox based on processing times and then request that Dropbox identify a user who has access to that file. However, to seek user content information, the government needs to comply with the provisions of the Electronic Communications Privacy Act by obtaining a warrant supported by probable cause (or in some cases a court order from a judge). Those safeguards protect user privacy. De-duplication does not make users any more vulnerable to intrusive government actions. Today, a government agency could ask any online service to provide the names of all users who have a particular file, whether or not the service employs de-duplication. And in that case, the government would also need to support its request with a warrant or court order. The rules that provide a check against unwarranted government snooping apply to online services equally, regardless of their back-end architecture.
I think most of us, if we were developing Dropbox, would have made the same decisions. De-duping at the cost of complete privacy, even in the face of law, is a sound technical and business decision for a service such as Dropbox.<p>When I think Dropbox, I think sharing, I think convenience, I don't think backup and security. For backup I need more space, for security I need to use my own private key (not a password that one can change/recover). Neither of these things is offered by Dropbox. And this is the reason why I never confused Dropbox with, say, CrashPlan. One is a way to share and collaborate, the other is a place to send my private key encrypted bits to.<p>My individual privacy is not compromised by somebody being able to say if a certain file is stored by the entirety of Dropbox user base. The other claim, that, given a court order, Dropbox can be forced to turn over your files or tell the court if you store a certain file _may_ be true, but I don't think Dropbox, the company, has ever promised that level of security.
My take-away is this:<p>"What this means, is that from the comfort of their desks, law enforcement agencies or copyright trolls can upload contraband files to Dropbox, watch the amount of bandwidth consumed, and then obtain a court order if the amount of data transferred is smaller than the size of the file."<p>This, I think is significant, especially if Dropbox is advertising security, privacy and encryption. As the author mentions, the ToS are being updated to reflect the above possibility ("if Dropbox receives a warrant, it has the ability to remove its own encryption to provide data to law enforcement").
Bottom line - you should assume that you cannot trust ANY cloud-based service to keep what you upload safe from hackers (if they're determined enough), and especially from governments. I'm not sure why anyone would be under the illusion that this is the case. Even assuming that hosting such a service would be legally possible today (not sure if it is, IANAL), it could be illegal tomorrow and the service may be compelled to hand over any data.<p>If you want something to stay secret you MUST either:<p>a. Not put it on the internet/cloud.<p>b. Encrypt it yourself before uploading. Yes, there are trust issues with commodity encryption software as well, but these may be mitigated somewhat.
It's not clear to me why Dropbox would need your keys to de-dupe. He says so in the article, but doesn't say why.<p>Why not compute the file hash on your local machine before encryption, and check that hash against a master dupe list (hash, dupe_count) of all hashes from all users' pre-encrypted local files?<p>Secondly, I cannot see how this requires there to be an index of users hashes. Surely one could store hashes with reference count, increment when a user adds, decrement when a user deletes. The user ID isn't necessary for a reference counter.<p>Not saying Dropbox isn't doing what he says. But he says de-duping proves they can decrypt and proves they have a list of who has the same files. I don't see it from de-dupe alone.
Without knowing the internals of how Dropbox operates, my empirical observations are that they employ block-level deduplication, i.e. when you change bits in the middle of the file, the whole thing doesn't get re-uploaded. Which means they keep pointers and have an algorithm that's similar to LBFS (and Rabin fingerprints)<p>This means it's theoretically possible for parts of the file to come from different sources, which means contraband files are 'built' from parts of otherwise legal files.
With pure file encryption where the user's password serves as the key you lose...password recovery features, public links for files, shared folders, web access, mobile access (unless you want your phone doing the decryption)<p>All other syncing services do things a pretty similar way
Rather than avoid deduplication (which is technically sound and benefits everyone), perhaps the solution to this is to make it impossible for DropBox itself to know who owns which files.<p>E.g. right now I assume a dropbox user owns a list of file ids with some metadata (e.g. that user's name for those files). If follows that if the government decides file XYZ is illegal then anyone with XYZ in their list is in trouble.<p>The user account could keep track of the total size of all the user's files and use arithmetic to keep it up-to-date, but not actually store the size of individual files except when they are "looked at".<p>So then the user's password (say) which is not itself stored is used to unlock stuff in the user's file table on a per request basis -- i.e. the actual file ids are only computed as needed. The actual mechanism doesn't need to be terribly secure, it just needs to be <i>deniable</i>. In other words without the user's password we simply cannot unambiguously determine which files are his or hers.
Of course Dropbox knows the keys. If they didn’t, you wouldn’t be able to access your files on so many platforms (web, desktop, iPhone) and you wouldn’t be able to easily share folders with others.<p>Even though it isn’t spelled out, I’ve always suspected that many actual backup services such as Backblaze don’t know the key if I decide to encrypt my backups.
Why doesn't DropBox just stop doing de-duplication? They must have the money for the storage? The bandwidth savings for users isn't that big of a deal in most cases. I expect that if I have a 2GB file that I'm uploading a 2GB file. I don't cross my fingers that you already have big chunks of it.<p>This just seems like the type of thing that someone much smarter than I can and will exploit in the future.
This makes me worried about hash collisions as well. The article implies that a file whose hash matches something they already have will never even reach their servers - so presumably I just have to keep my fingers crossed that the file they're synchronising to all my machines is the one I uploaded, and not some other user's completely different file that happens to have the same hash?
Early optimization is the root of all evil, so I understand that an up-and-coming company might do this, but Dropbox has the traction and userbase to make this a very relevant concern.<p>Popularity is also proportional to chance of being targeted by hackers and approached by government or corporation representing intellectual property owners.
If you knew a target file's hash, it might be possible to modify the dropbox client to report that file as added, then dropbox would download that file onto your computer. Of course, 10^77 possible hashes makes it unlikely.
Previous HN discussion of SpiderOak's (very different) approach to this same topic: <a href="http://news.ycombinator.com/item?id=1640074" rel="nofollow">http://news.ycombinator.com/item?id=1640074</a>
Could be worse. Last year when I tried ZumoDrive (a similar service), I noticed it marks the web-browser login cookies as safe for HTTP, and defaults to open session pages via HTTP. All it takes it checking your ZumoDrive once from an unsecured WiFi and your account may be compromised.<p>At least Dropbox gets the endpoint-to-server encryption right.
Its not obvious to me this is a price based decision for Dropbox (although the benefit there is obvious).<p>Arguably the best feature of Dropbox for me is binary diffs. If you encrypt the Dropbox this goes out the window, or at least becomes significantly harder to pull off. Am I wrong?
tl;dr
Dropbox use their own encryption keys to encrypt your data rather than encrypting each user's data using a user provided key. This helps them dedup files and save space/money. This implies that a court could ask to analyse your data. Dropbox will update their privacy policy to say this clearly.
I've been traveling for most of the last six months, mostly dependent on slow & unreliable hotel wifi. Dropbox's implementation has saved me a ton of time backing up files that would have taken forever to upload in their entirety.
Here's a startup idea: A background service that runs on a PC/IPad/Phone, checks for new media files (pictures/mp3s/videos) and automatically re-encodes them, such that the quality, etc. is preserved but the file hash changes and cloud services can no longer deduplicate it.<p>It will be quite valuable for users of services like Dropbox, Amazon Cloud Player, etc.