TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

Extracting training data from diffusion models

163 点作者 ericwallace_ucb超过 2 年前

33 条评论

dang超过 2 年前
See also <a href="https:&#x2F;&#x2F;twitter.com&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624</a>. (Thanks to all who posted that. We merged the threads now.)
saurabh20n超过 2 年前
The last author&#x27;s tweet thread and replies have some interesting tidbits: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624</a><p>* &quot;We propose to extract memorized images by generating many times with the same prompt and flagging cases where many of the generations are the same.&quot;<p>* &quot;- Diffusion models memorize more than GANs - Outlier images are memorized more - Existing privacy-preserving methods largely fail&quot;<p>* &quot;Stable Diffusion is small relative to its training set (2GB of weights and many TB of data). So, while memorization is rare by design, future (larger) diffusion models will memorize more.&quot;<p>* &quot;It only memorizes a very small subset of the images that it trains on.&quot;<p>* &quot;our goal is to show that models can output training images when generating in the same fashion that normal users do.&quot;
评论 #34608766 未加载
FeepingCreature超过 2 年前
100 images out of 350,000 that they looked at were memorized.<p>This seems to mostly happen when an image appears frequently (more than 100 times) in the training data, and&#x2F;or the dataset is small relative to the model.
评论 #34612344 未加载
评论 #34612508 未加载
评论 #34610427 未加载
评论 #34613320 未加载
评论 #34612223 未加载
评论 #34610124 未加载
评论 #34614823 未加载
评论 #34611351 未加载
6gvONxR4sf7o超过 2 年前
It&#x27;s work like this that makes me frustrated at the popular discourse around generative models (especially here). There&#x27;s a ton we don&#x27;t know about these models, and yet you get tons of people arguing that these models absolutely don&#x27;t memorize, or that they learn like we do and so their learning should be treated like ours (legally and ethically). Then you get work like this showing that yes they actually do some memorization and regurgitation. There&#x27;s still so much we don&#x27;t know here.<p>My fear is that when things like this come up for lawsuits, overconfident experts are going to talk out of their asses about how these models do or don&#x27;t work, and that&#x27;s going to determine how automation affects our society.<p>On a technical level, I&#x27;d love to see a patch-wise version of this investigation. This shows whole images being regurgitated near-exactly rarely. I expect that small part-of-the-image patches are regurgitated even more often. But is it simple stuff like edges being regurgitated or are larger parts regurgitated frequently too? Given the architectures generally used, I&#x27;d guess that it&#x27;s significant.
评论 #34606921 未加载
评论 #34605797 未加载
评论 #34605491 未加载
评论 #34605754 未加载
yetanotheruser8超过 2 年前
This study was organized by Google (Technically DeepMind).<p>I wouldn&#x27;t be surprised if Google is wanting the lawsuit to lose. It would block open-source models like these from existing and give them potentially a competitive advantage to be able to afford whatever compliance is mandated. They&#x27;d be able to offer services that comply, but open-source models would only have access to lower quality data and would be stunted.
评论 #34612630 未加载
mxwsn超过 2 年前
Their extraction: (1) assumes the attacker knows the caption for some training images, and (2) primarily works on images duplicated 100x-3000x in the training dataset. Their attack does not succeed for any singleton images. Deduplicating can be challenging on internet-scale datasets, but their work as presented does not appear to be a major concern for releasing diffusion models trained on other smaller datasets.<p>On memorization - I suspect this is a great thing for downstream performance, and a positive indicator that diffusion models are actually better generative models than prior methods (VAEs, GANs, etc). This mirrors the finding that feedforward neural networks can memorize randomly labeled data very well. Intuitively it feels like memorization is a quantifiable behavior that is a foundational activity in information processing - it is one type of optimal usage of observed data - that superpowers downstream performance.
评论 #34601483 未加载
Imnimo超过 2 年前
Only 109 retrievable images out of the 350,000 most-duplicated is fewer than I expected. Maybe it&#x27;s just the stringent definition of retrieval, but I would have expected many famous works of art like the Mona Lisa and Girl with a Pearl Earring to be readily extractable. Maybe these just aren&#x27;t quite pixel-perfect enough?
GaggiX超过 2 年前
So wait they only found 109 matches after generating 175 milion images using the prompt from the most duplicated samples from the dataset and SD v1.4? Also almost all of them have more than 300 copies in the dataset, so with a model with the same size and trained on a dedup dataset like SD 2.0&#x2F;2.1 there will be almost no matches, even after generating 175 mln images and knowing the prompts used in the dataset. Finally Google at el need to explain how an attacker that want to extract images from a trained model somehow has the prompts for the top X duplicated images in the dataset but not the images themselves, and thus will going to spent an incredible amount of money to generate something like 175 mln samples and test them together to find the matches.<p>Edit: I also want to add that google seems to try really hard to show themselves as the good guys by not releasing its models because it&#x27;s not safe enough, but in this paper they used an incredible amount of computation and show me otherwise.
评论 #34612193 未加载
评论 #34614460 未加载
babel_超过 2 年前
The tweet&#x2F;paper co-author posted the paper (<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2301.13188" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2301.13188</a>) yesterday on HN (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34596187" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=34596187</a>) and ironically the top comment there is referencing to this exact tweet thread (which was posted yesterday as well). Evidence for the metacircular evaluation of HN comments?<p>I think the paper is well worth the read, it&#x27;s not particularly long (much is references and appendices), and nicely written, with at least a quick bit on most things I would think to test as part of something like this. Good stuff.
评论 #34614423 未加载
danaris超过 2 年前
I know it&#x27;s not cool to say &quot;I told you so&quot;, but...<p>This was entirely predictable, and is one prong of the primary arguments that these ML models, trained on datasets including copyrighted images taken without permission, infringe on the copyright of those images&#x27; creators.<p>Train the damn things on public domain images and images you have <i>explicit</i> permission for, and you&#x27;ll be fine. Stop acting like you have a right to just vacuum up every image ever created because it&#x27;s &quot;AI&quot;.
评论 #34611861 未加载
评论 #34610391 未加载
评论 #34610397 未加载
评论 #34610470 未加载
ericwallace_ucb超过 2 年前
The paper shows that Stable Diffusion and Google&#x27;s Imagen regenerate individual images from their training sets. They show it is very rare, but can be found reliably.
评论 #34599825 未加载
评论 #34599838 未加载
jjcon超过 2 年前
Is there any reason we shouldn’t view diffusion models as any other tool? I can infringe copyright with photoshop too… even accidentally. If I generate original work with either that seems like fair game.<p>I imagine with the right prompt one could coax out a copywritten image even if it hadn’t ever seen it before
评论 #34600840 未加载
Glyptodon超过 2 年前
To me this is kind of like being shocked that people who&#x27;ve seen the Starry Night can remember what it looks like.
评论 #34612921 未加载
评论 #34612740 未加载
评论 #34611818 未加载
评论 #34611964 未加载
6gvONxR4sf7o超过 2 年前
I’m disappointed in all the anthropomorphizing in this thread. Time and time again, we make analogies for how black box ML algos must work like people, only for researchers to come along and show that they actually just use shortcuts that don’t remotely resemble human learning&#x2F;thinking.<p>When will we learn to stop being overconfident about how these things work? Just say “we don’t know yet.” Anthropomorphism and overconfidence are dangerous in that we could set the wrong precedents (culturally and legally) for how these are used and how automation affects society.
评论 #34613697 未加载
评论 #34614677 未加载
评论 #34613712 未加载
Lerc超过 2 年前
Figure 2 doesn&#x27;t fill me with confidence as to the ability to detect similar images. The best example is the bottom right match which hits because the collar is in the same position and a bunch of white being in the same place outweighs a lot more meaningful data.<p>This probably means there are far more matches to be found that would be considered clearly copies to humans. SSIM might be a bit heavy for the task but a simple comparison of the gradients from neighboring pixels might match quite a lot more.
Archelaos超过 2 年前
When I experiment with Stable Diffusion, I quite often come across blurred &quot;Getty Images&quot; labels.
评论 #34610381 未加载
评论 #34610339 未加载
评论 #34610341 未加载
jxy超过 2 年前
I don&#x27;t understand what is so surprising here. The training of the model consists of adding noise to training samples and denoising the resulting random samples to reproduce training samples. If you have one training sample, you can find the optimized random sample that reproduce the training sample.
评论 #34600080 未加载
评论 #34600988 未加载
评论 #34599988 未加载
评论 #34604846 未加载
usrbinbash超过 2 年前
So some images were overrepresented in the dataset, and subsequently, the network overfitted. Known problem, known solution.
评论 #34611165 未加载
评论 #34611135 未加载
larsiusprime超过 2 年前
Some interesting commentary by AI expert Alex J. Champandard: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;alexjc&#x2F;status&#x2F;1620466058565132288" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;alexjc&#x2F;status&#x2F;1620466058565132288</a>
评论 #34603434 未加载
rvz超过 2 年前
Also another previous study that has Stable Diffusion (SD) emitting images in the training set [0]<p>It is now clear that SD is treading on thin ice: training on watermarked and copyrighted images without their author&#x27;s permission, then attempting to commercialize it even when the model emits images that resemble a high similarity of the original training data including watermarks or copyrighted images: (Mickey Mouse, Getty Images watermarks, Bloodborne art cover, etc).<p>This weakens their fair use argument, especially with Getty Images also threatening to sue SD for the same reason. If OpenAI was able to get permission to train on shutterstock images [1], then SD could have done the same, but chose not to.<p>Perhaps SD thought they could get away with it and launch their grift (DreamStudio) on digital images and artists. It turns out that now SD creates an opt-out system afterwards but artists can already find out if their images are in the training set. [2].<p>[0] <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2212.03860.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2212.03860.pdf</a><p>[1] <a href="https:&#x2F;&#x2F;www.prnewswire.com&#x2F;news-releases&#x2F;shutterstock-partners-with-openai-and-leads-the-way-to-bring-ai-generated-content-to-all-301658310.html" rel="nofollow">https:&#x2F;&#x2F;www.prnewswire.com&#x2F;news-releases&#x2F;shutterstock-partne...</a><p>[2] <a href="https:&#x2F;&#x2F;haveibeentrained.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;haveibeentrained.com&#x2F;</a>
评论 #34610018 未加载
评论 #34610080 未加载
评论 #34621555 未加载
yorwba超过 2 年前
I wonder whether the &quot;data dimension&quot; from <a href="https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;toy-double-descent&#x2F;index.html#comment-mnist" rel="nofollow">https:&#x2F;&#x2F;transformer-circuits.pub&#x2F;2023&#x2F;toy-double-descent&#x2F;ind...</a> could be used to identify the model parameters involved in memorization and remove them without having to retrain from scratch on a cleaned-up dataset.
mshake2超过 2 年前
I expect to see this paper in many lawsuits soon as evidence of copyright infringement.
diimdeep超过 2 年前
Already there are hundreds &#x27;fine-tuned&#x27; or merged models, made with base models from Stable Diffusion and easy to use inference and training tools like this[2]<p>I wonder whether extraction attacks easier if you have many ancestral models?<p>[2] <a href="https:&#x2F;&#x2F;github.com&#x2F;AUTOMATIC1111&#x2F;stable-diffusion-webui#stable-diffusion-web-ui">https:&#x2F;&#x2F;github.com&#x2F;AUTOMATIC1111&#x2F;stable-diffusion-webui#stab...</a>
ornornor超过 2 年前
In case you’d rather not suffer twitter’s abysmal UI on mobile web browser: <a href="https:&#x2F;&#x2F;nitter.net&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624" rel="nofollow">https:&#x2F;&#x2F;nitter.net&#x2F;Eric_Wallace_&#x2F;status&#x2F;1620449934863642624</a>
adrianmonk超过 2 年前
Ironically, this almost makes it more human.<p>It&#x27;s a surprisingly common experience for music students to excitedly tell everyone they know about a new piece of music they&#x27;ve been composing, saying it is probably the best thing they&#x27;ve ever written, and then a friend or teacher has to say, &quot;I don&#x27;t know how to break it to you, but you&#x27;ve &#x27;composed&#x27; the Xth movement of Beethoven&#x27;s Yth symphony.&quot;<p>And sometimes they will say, &quot;I have? I don&#x27;t think I&#x27;ve ever heard Beethoven&#x27;s Yth symphony.&quot; But of course they have, just without realizing it. It was in the background of some movie they watched or something like that.<p>Unlike humans, I don&#x27;t think AIs have any belief about whether their work is original or not, but it&#x27;s the same type of error. And with similar legal consequences: people have been sued for stealing a melody (presumably not always consciously). The difference with AIs is they can produce much more output than humans, and it&#x27;s muddier what is actually doing the creating (AI authors? users?).
评论 #34614349 未加载
xiphias2超过 2 年前
Looks like it was funded by Deepmind for the purpose of fighting more open models on the legal field. I don&#x27;t think they are just ,,protecting the artists&#x27;&#x27;.
sschueller超过 2 年前
What will happen if SD looses the court case? The cat is out of the bag and the data set can be downloaded by anyone today.
评论 #34611296 未加载
评论 #34612743 未加载
评论 #34614044 未加载
评论 #34611122 未加载
singularity2001超过 2 年前
This is really bad news for the community, especially in the context of the Copilot lawsuit. Soon lawyers will terrorize network creators, startups and users.
评论 #34611232 未加载
评论 #34611897 未加载
评论 #34612121 未加载
评论 #34611266 未加载
bethecloud超过 2 年前
Person enters &quot;Ann Graham Lotz&quot;, image of Ann Graham Lotz appears. Why does this upset people and google image search doesn&#x27;t
评论 #34611235 未加载
评论 #34611214 未加载
fab1an超过 2 年前
The way the author summarizes his own study in this thread borders on misinformation. You could actually take their findings and write the opposite headline, which would more accurately reflect their actual research results:<p>&quot;Critics claim that models such as Stable Diffusion act like modern collage tools, recreating copyrighted and sensitive material.<p>Yet, our new paper shows that this behaviour is exceedingly rare, recreating copies in less than 0,00006% of 175M test cases.&quot;
KHRZ超过 2 年前
Copyright holders screeching again? Maybe I should copyright a black image and sue anyone who turns off their screen.
seydor超过 2 年前
Eigenimages
alexb_超过 2 年前
Anybody who knows what the pigeonhole principle is should know that a lot of these fears are complete bunk.
评论 #34611752 未加载
评论 #34611537 未加载