I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc.<p>I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.
<a href="https://towardsdatascience.com/black-box-attacks-on-perceptual-image-hashes-with-gans-cc1be11f277?gi=cfb3a66600e5" rel="nofollow">https://towardsdatascience.com/black-box-attacks-on-perceptu...</a><p>I mean, who's to say that my hash is actually what they think it is...<p>...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?<p>"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"<p>"DEFENDANT: I'm not going to give you my phone's password."<p>"THE COURT: Bonk: Stay in jail for contempt."<p>I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
If it matches, am I guilty until found innocent?<p>I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching.<p>I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic.<p>PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).
This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on <i>uploaded</i> user content since it is actually voluntary.<p>Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.<p>Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
Reminder that even accusations of abuse imagery will get you in cuffs and your children taken away (see article at the end)<p>Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation".<p><a href="https://jonathanturley.org/2009/09/18/arizona-couple-sues-wal-mart-after-store-calls-the-police-on-them-for-developing-pictures-of-this-children-in-a-bathtub-and-the-children-are-taken-by-the-state/" rel="nofollow">https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...</a>
In a world where all police were incorruptible and all laws were just, maybe I'd be ok with this. But since we live in <i>this</i> world, I feel like it's only a matter of time until the "think of the children" technology is coopted to scan for other "bad" images.<p>I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
Scan for child abuse today; scan for wrong-think tomorrow. I'm liking the idea of Graphene OS more and more each day... or just going phone-free entirely.
> ...ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography.<p>I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
When it comes to Apple and privacy, it's important to differentiate two different types of privacy: (1) privacy against snooping by the state, and (2) privacy against snooping by all others. As we had seen with Apple and China, it's only the second type of privacy that Apple cares about. Sharing your information with the state is not out of character for Apple. In fact, had they done the opposite, it would have been out of character.
WhatsApp automatically saves photos sent to the phone to the Photo Album. Does this mean they will get a knock on their door if someone sent a bad image?
Apple should perhaps dogfood their system on their own employees first: <a href="https://www.losaltosonline.com/news/hills-man-arrested-on-child-porn-charges/article_9bbe51f7-aaa4-5814-98eb-a9a6b0f51a42.html" rel="nofollow">https://www.losaltosonline.com/news/hills-man-arrested-on-ch...</a>
> Apple intends to install software on American iPhones to scan for child abuse imagery<p>> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system<p>Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation.<p>Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?
As any of the lawyers here can attest, the legal definitions here open to huge debates. Some material is unambiguous, other images reside extensive grey areas. The legality of material can even change based on context. Something that is perfectly legal in one area can be absolutely illegal in another. Example: If I had to list the most iconic photographs in world history, one of them is of a totally nude child (Vietnam). That isn't an illegal image, but try explaining that to a computer. At the moment these nuances are handled by cops, prosecutors and in extreme cases the courts. Is Apple going to put a marker down on a particular universal definition? Will that definition become the de facto world standard?
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.<p>It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?<p>If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.<p>> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities<p>Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
Things that would reduce child sexual abuse more than scanning everyone's phones on a daily basis:<p>- Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household<p>- Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos)<p>- Enforce obscenity laws against incest and other extreme forms of pornography<p>- Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly<p>Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.
<a href="https://news.ycombinator.com/item?id=28068741" rel="nofollow">https://news.ycombinator.com/item?id=28068741</a><p>This topic already made it to the current frontpage.
> Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.<p>Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
Isn't Apple going to start auto scanning the text in your images too for searching/easy copying? How many years until the hash of text in your images is compared to a database of... god knows what. Sensitive information? Dangerous text? I honestly can't think where it's most likely headed but I don't like it.
I have yet to deep-dive into the techical details of Pegasus so this is pure speculation, but if Pegasus demonstrates the feasibility of deploying and executing arbitrary payloads on remote devices without the owner's knowledge or consent wouldn't that raise a blackmail vector with this new program? Again I'm not familiar with the specific exploits used by Pegasus, but is there anything stopping another malicious actor from relying on the same exploits to say install a bootloader that downloads images that would trigger a positive scan?
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.<p>This doesn't really make any sense. So it's not a Neural Engine thing but merely hash matching, and it also only applies to iCloud Backup? That is already non-e2e encrypted, so Apple can already access those files whenever it wants.
This will enable Apple to send hashes of matching fingerprint and face scans to their online database. This also completely disabled their argument that their product is safe because the data remains encrypted on your device. Instead of passing the data to the spy service they are sending a report on data to the cloud which will allowed to be much more opaque and misleading.<p>I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.
This just means child molesters will seek out PinePhones, Purism Librems, etc. And those phones will acquire reputations as "pedophones", and anyone buying or using one will be immediately considered sus. People with nothing to hide won't mind an OS with surveillance features.
I'm too old for this, but I really wonder what it's going to be like for high-school kids and nudes 10 years on.<p>I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person?<p>Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.
While they are at it they should up the ante and also add COVID misinformation filters, so that a phone refuses to save it and/or doesn't send it anywhere. Would be very topical. There is also racism and other almost universally recognized wrongthink. If you decide to police offline activity as if it's a cloud platform, why not implement all the online platform's automatic content filtering/censorship? /s
I had a very sick 8 month old baby who I obviously had a million photos of. Unfortunately, my lil girl had 9 surgeries and her little body had gone through more than people can think of. She passed in May, and it's really difficult to look back at old pictures because of how much stuff was going on, various levels of critical ICU care, pre/post surgery, etc. But it wasn't until a few months after she was born I had gone to do something in Google and realised my little girl's very severe diaper rash, open sternum, chest tubes, etc, etc, etc photos were backed up and tagged in Photos.<p>This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI.<p>How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.
Hopefully they are using phash to match the phone's images to known child abuse image hashes, because if they are using ML, I can only imagine what a false positive will do to someone's life.
On one hand, I applaud this - CASM is a massive issue, and the internet community has never taken it as seriously as it should have.<p>As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police.<p>Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache.<p>Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach.<p>This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.
Imagine if Microsoft does this with Windows, lots of people would get arrested rightly so.<p>But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
Do we know why did Apple put the effort to create this? Who forced them to do it? Like it is a move that is bad for PR and also does not make them money, I suspect some big government is demanding it but maybe I am to simple in my thinking and this makes sense.
so you’ll be able to ruin someone’s life by sending them a live photo where one frame has child abuse. i find the law enforcement part of these well intentioned anti child abuse initiatives to be absolutely terrifying because the standard is not that you were proven to have been a party to child abuse but that an offending file was found on your device.
Along with this: <a href="https://www.reuters.com/technology/exclusive-facebook-tech-giants-target-manifestos-militias-database-2021-07-26/" rel="nofollow">https://www.reuters.com/technology/exclusive-facebook-tech-g...</a><p>Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".
This article raises serious privacy concerns. Just building the infrastructure for on-device scanning and reporting is extremely troubling. A slippery slope, or a break in the dam, as others have said. My view is that we've been on that slope for a long time, and this changes little. We just have to trust Apple, as has always been the case.<p>When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else.<p>Two technical reasons for this:<p>- The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (<a href="https://news.microsoft.com/on-the-issues/2018/09/12/how-photodna-for-video-is-being-used-to-fight-online-child-exploitation/" rel="nofollow">https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...</a>). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.<p>- As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow.<p>- It sounds like it doesn't flag a single image, but requires multiple hash hits.<p>So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud.<p>This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it.<p>Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
To everyone here who wishes we do nothing about this: you should rethink your position. There is a difference between “let’s be careful and design the right solution” and “let’s do nothing to maximize individual liberty”.<p>In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow.<p>In raw percentages the data clear: too many children are being abused and the numbers are growing.<p><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b6555a1018a750f39028005bfdb9f35eaee4b947.pdf" rel="nofollow">https://storage.googleapis.com/pub-tools-public-publication-...</a>
Related to this, I love Canva.com but then I noticed they were scanning my designs. Some warning popped up.<p>This idea of perpetually renting services than monitoring the use is something that should be criticized.
To laugh, I just imagine the society of 30 years ago and what would have happened if someone would have tried to suggest to do something like that:<p>ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal.<p>It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore...<p>But sadly, it looks like that intelligence and common sense has generally decreased in the population.<p>Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature!<p>Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!
What if under a secret order Apple is instructed to insert non-CSAM photos, to match whatever a relevant juridiction wants to match? Effectively, you now have a black box operating on your own computer, this is deeply disturbing. This is not going to end well. And by the way this is like all the DNA stuff, you're not only at risk from your own computer but also the all the libraries your picture happened to be in.
> The proposals are Apple’s attempt to find a compromise between its own promise to protect customers’ privacy and ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography.<p>The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.
So, how long before the same technology is used to fight "misinformation" and "domestic terrorism" (e.g. memes that disagree with official point of view)? How long until it's used to ban "hateful speech", or any speech unapproved by the gov^H^H^HReal Government - the Almighty Apple?
Wow, this seems like a terrible, terrible idea on so many counts. It would certainly keep me from buying an iPhone -- the last thing I need is to get a visit from the cops because of false positives.<p>It also means that you can't have any trust about your data safety even if you never put it in the cloud.
Will Google by some miracle be the company that offers freedom from forced snooping?<p>It seems not, so we have a choice between this and Google's always-track-you setup.<p>I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?
This is extremely alarming, I don’t care what the reason, I don’t want apple or anyone looking through my personal photos. It’s none of their business, and the sheer chance of false positives and ruining peoples lives is far too great to let this go forward.
> the screening is done on the phone.<p>Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity?<p>Did I agree to this already?<p>(Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)
Does this paywalled article have any further details? Aside from a cryptic twitter post that mentions CSAM, are there any further details?<p>CSAM also means Cyber Security Asset Management.
Whose to say what is abuse? Are Caesar Milan tactics on a child abuse? This is outright dangerous beyond belief. Nobody is gonna wanna be a parent if you have to worry about your kid being mad at you and then posts something like that.<p>IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
Apple: An Apple NeuralMatch Agent has positively matched images on device "Apple iPhone 12 (Red™)" geotagged at the coup attempt against Prime Minister Trump occurring on January 30th, 2024. Images and documents have been submitted to the Thump Bureau of Investigation for further evaluation. Your Apple account has been permanently suspended.<p>You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account.<p>Sorry for any inconvenience.
It’s 2035, a major terrorist attack occurs. Apple scans all phones for location and images near the area over the last 12 months, sends the list over since the government declared Marshall law.<p>You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.<p>Edit:<p>Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this?
____________<p>Second Edit:<p>Wanted to reply to another comment but being rate limited:<p><i>As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.</i><p>The number of Muslims in America:<p><i>A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.</i><p>Lol. Jesus Christ, you do the math.