TE
科技回声
首页24小时热榜最新最佳问答展示工作
GitHubTwitter
首页

科技回声

基于 Next.js 构建的科技新闻平台,提供全球科技新闻和讨论内容。

GitHubTwitter

首页

首页最新最佳问答展示工作

资源链接

HackerNews API原版 HackerNewsNext.js

© 2025 科技回声. 版权所有。

AI Dungeon public disclosure vulnerability report

198 点作者 kemonocode大约 4 年前

17 条评论

rdl大约 4 年前
I am strongly against child abuse, but I really don&#x27;t have a problem with a computer being forced to emit textual patterns which include English words correlated with something a human might call a story about child abuse. It&#x27;s a waste of GPU time, but enh.<p>It&#x27;s scary that the press release from latitude talks about &quot;and to comply with law&quot; as a reason for review. Under US law, <i>maybe</i> specific threats to the President might be reportable (although one on one communication with an AI would be a stretch here...), but I&#x27;m pretty sure an AI or other system emitting textual patterns which humans view as representing fantasy sexual abuse of anything isn&#x27;t illegal, just distasteful.<p>They&#x27;re perfectly within their rights to ban it under a ToS but pretending it is for legal purposes is fucking bullshit. (Of course, I&#x27;m not a lawyer.). My understanding of court decisions is that even machine-generated images are legal, although &quot;is this image machine generated or is it evidence of actual child sexual exploitation&quot; is an increasingly difficult question and if you&#x27;re building an automated low-cost system it often makes sense to err on the side of safety. There might be some complexity around laws related to image manipulation involving real, but legal&#x2F;non-sexual minor images which are then convoluted into something sexual, or &quot;revenge porn&quot; use of something which simulates a specific person (or is based on that person), and <i>maybe</i> text can be legally questionable if it&#x27;s abuse targeted to a specific person, but especially for on on one non-published communications with a computer you are probably fairly ok with a weird fantasy sexual fetish about a neighbor, even a minor neighbor, in text form, unless it rises to an actual threat.
评论 #26978810 未加载
评论 #26979806 未加载
评论 #26977703 未加载
评论 #26977745 未加载
评论 #26978021 未加载
评论 #26987450 未加载
neiman大约 4 年前
&gt; Unfortunately, this is, in fact, the second time I have discovered this exact vulnerability. The first time, the issue was reported and fixed, but after finding it again, I can see that simply reporting the issue was a mistake.<p>I feel uncomfortable with this. The author already reported a vulnerability, it was fixed, but now there&#x27;s a new one (which is identical, ok, but new nevertheless), so he decided they didn&#x27;t study their lesson, and punish them with public shaming? I&#x27;d maybe get it if the first time was ignored, but like this? Nah ah.<p>It&#x27;s like my worse teachers coming back to hunt me as an adult.
评论 #26980386 未加载
评论 #26986283 未加载
评论 #26982304 未加载
评论 #26978575 未加载
评论 #26979662 未加载
nanidin大约 4 年前
The author found a vulnerability, extracted data they should not have had access to, processed the data (aggregated, anonymized), then published the data. Isn&#x27;t everything starting from &quot;extracted&quot; illegal? Or is it a gray area where &quot;the server would not have provided the data if I were not authorized to receive it&quot; -- in spite of the author&#x27;s admission that it was acquired via a vulnerability?
评论 #26977952 未加载
评论 #26979186 未加载
akersten大约 4 年前
I haven&#x27;t worked with GraphQL before, but looking at those code snippets and reading the description of the vulnerability, it seems like a mess. You&#x27;re giving a client unfettered access to just... query your database? Of course you&#x27;re going to get these kind of issues - that just seems obvious to me.<p>Getting real off-topic, but the syntax is backwards too:<p>` Interface Votable implemented by Adventure, Comment, Post, Scenario `<p>The interface lists what implements it? Reminds me of COMEFROM[0].<p>I dunno. Modern front-end is wild. These live code Notebook things are chaos. Spaghetti begets spaghetti.<p>[0]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;COMEFROM" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;COMEFROM</a>
评论 #26978011 未加载
评论 #26977822 未加载
评论 #26979029 未加载
narrator大约 4 年前
This is a really interesting moment in AI. An AI spontaneously commits a crime and engineers have to teach the AI how to obey the law.<p>We have the AI allegedly emitting illegal fiction and the engineers have to fix it and all they can try to do is word filters. What happens next in this story?<p>This reminds me of the Chinese virtual girlfriend who got neutered for saying politically illegal speech that the Chinese government objected to.<p>Another one, was when the Google image labeler was mistaking people for animals. That was extremely distasteful, but not illegal. Google&#x27;s solution was to get rid of those labels.<p>Also, all the restrictions on drone activity are another thing. However the drone problem is solveable with reasonably simple rules.<p>Imagine if Alpha Go made a pattern that could make people have seizures or something, but most people could detect it and not do it, but nobody could make a simple rules based approach to detect it. I guess you&#x27;d need a whole nother alpha go size model to recognize that pattern perhaps?
评论 #26979109 未加载
评论 #26977840 未加载
评论 #26977843 未加载
评论 #26978726 未加载
评论 #26978038 未加载
评论 #26977990 未加载
minimaxir大约 4 年前
See also the change in content filtering announced today (<a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26967683" rel="nofollow">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=26967683</a>), which given the disclosure timeline here, may be related.
pdkl95大约 4 年前
(off topic, but this report is a good example of how to handle user data)<p>&gt; anonymized<p>Could we, perhaps, stop using this word? Instead of using the vague, often misleading term &quot;anonymized&quot;, state directly what actually happened, e.g. &quot;names and addresses were removed&quot;, &quot;user data was aggregated by ${group}&quot;, or &quot;the UID was replaced with a new, equivalent key&quot;. Most of the time claims about data being &quot;anonymized&quot; are simply not true; replacing names or UIDs with a hashed value that is merely replacing an existing candidate key with a new synthetic key. As DJB said[1]:<p>&gt;&gt; Hashing is magic crypto pixie-dust, which takes personally identifiable information and makes it incomprehensible to the marketing department. When a marketing person looks at random letters and numbers they have no idea what it means. They can&#x27;t imagine that anybody could possibly understand the information, reverse the hash, correlate the hashes, track them, save them, record them.<p>The rare examples where &quot;anonymized&quot; actually involves meaningfully making user data <i>anonymous</i> are when the actual user-correlated relations[2] have been <i>destroyed</i>. This report specifically discusses how this was done:<p>&gt; If a sentence fragment appeared in less than 10 unique adventures, it was discarded from the result set to preserve anonymity.<p>Sometimes this required accepting a small amount of error:<p>&gt; this data needed to be processed in batches of around 10000 adventures per batch. In each batch, fragments appearing only once were purged. Therefore, counts under around 25 are actually underestimates.<p>[1] <a href="https:&#x2F;&#x2F;projectbullrun.org&#x2F;surveillance&#x2F;2015&#x2F;video-2015.html#bernstein" rel="nofollow">https:&#x2F;&#x2F;projectbullrun.org&#x2F;surveillance&#x2F;2015&#x2F;video-2015.html...</a><p>[2] <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Relation_%28database%29" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Relation_%28database%29</a>
评论 #26983327 未加载
pugworthy大约 4 年前
OK now I have to go check out AI Dungeon. Is this some clever marketing ploy to get me to try it out?
评论 #26977929 未加载
评论 #26977576 未加载
评论 #26977649 未加载
MrGilbert大约 4 年前
&gt; The results are... surprising, to say the least.<p>Well, are they? I always thought that people will try stuff in a &quot;safe harbor&quot; which they cannot try or should do somewhere else. So I always expect these sandboxes to be full of nsfw stuff.<p>And people might not understand that their stories will influence the story of others, so...
评论 #26979248 未加载
评论 #26978385 未加载
评论 #26978854 未加载
h_anna_h大约 4 年前
Fun fact: AI Dungeon used to be Open Source and you used to be able to run it locally without sending your data to someone else and without censorship of any form <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;AI_Dungeon#Development" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;AI_Dungeon#Development</a><p>This is what happens when software that you use does a bait and switch into cloud.<p>For anyone wanting to play it locally, a quick google search gave me these two links: <a href="https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;1OjBQe4H4C2s-p4-OeJoXw5DStIjPy2VS" rel="nofollow">https:&#x2F;&#x2F;colab.research.google.com&#x2F;drive&#x2F;1OjBQe4H4C2s-p4-OeJo...</a> and <a href="https:&#x2F;&#x2F;pastebin.com&#x2F;UMUV0KTw" rel="nofollow">https:&#x2F;&#x2F;pastebin.com&#x2F;UMUV0KTw</a>
User23大约 4 年前
This reminds me of how Nintendo developers discovered their western customers love drawing phalluses[1]. I don’t find the NSFW percentage to be at all surprising. It was common with Eliza too.<p>[1] <a href="https:&#x2F;&#x2F;www.kotaku.com.au&#x2F;2012&#x2F;11&#x2F;nintendo-created-a-penis-drawing-inferno&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.kotaku.com.au&#x2F;2012&#x2F;11&#x2F;nintendo-created-a-penis-d...</a>
评论 #26977429 未加载
shawnz大约 4 年前
One of my first thoughts when playing with AI dungeon was to try and get it to write something erotic. Glad I didn&#x27;t follow through
评论 #26978855 未加载
评论 #26977811 未加载
throwawayaid大约 4 年前
Speaking as a former customer, their actual application is really not that great. I was subscribed over several months and while new features were sparse, nearly daily the app would update with fixes to the UI and backend. Existing features that became broken and fixed on a day to day basis and UI glitches all over the place. So while their core product, the AI, is the best on the market, everything they wrapped around that really isn&#x27;t that great at all. So I&#x27;m not really suprised that their API is lacking as well. Just something to keep in mind, before using their product...
评论 #26978338 未加载
Amaru84大约 4 年前
People are idiots, you act like you care about children and want them to be safe, but freak out more over fiction then reality.. I was born in 1984 and was sexually abused like so many other kids, and it was by a parent.. what also gets me is you think only pedophiles sexually abuse children, the fact is they are less likely too.. you can look it up yourself, its well known in the phycology field.. <a href="https:&#x2F;&#x2F;blogs.bmj.com&#x2F;medical-ethics&#x2F;2017&#x2F;11&#x2F;11&#x2F;pedophilia-and-child-sexual-abuse-are-two-different-things-confusing-them-is-harmful-to-children&#x2F;" rel="nofollow">https:&#x2F;&#x2F;blogs.bmj.com&#x2F;medical-ethics&#x2F;2017&#x2F;11&#x2F;11&#x2F;pedophilia-a...</a> .. Pedophilia and Child Sexual Abuse Are Two Different Things — Confusing Them is Harmful to Children.
评论 #26984900 未加载
Kiro大约 4 年前
Isn&#x27;t it just random dungeons created by anonymous users? Is there actually any sensitive data here? I have a &quot;similar&quot; service (nothing about AI but similar in other ways) and security is the least of my concerns since being hacked means I will expose completely meaningless data. Now I&#x27;m afraid someone will hack me and make a similar fuzz about me being an idiot.
评论 #26979201 未加载
sergiotapia大约 4 年前
I had no idea people still used autoincrementing ids. Do people also build businesses on cakephp and joomla?
评论 #26977327 未加载
评论 #26977642 未加载
评论 #26977418 未加载
评论 #26977553 未加载
评论 #26978577 未加载
评论 #26977316 未加载
nitwit005大约 4 年前
&gt; In summary - if user input on a private adventure is flagged using an automated system, it will be manually reviewed, with other private user adventures potentially being manually reviewed as well. With almost half of the userbase being involved with NSFW stories, this seems like a tremendous misstep, as users have an expectation that their private adventures are, well, private.<p>I would assume they want to review the inputs to avoid a repeat of the incident where Microsoft&#x27;s Twitter bot was trained to say inappropriate things: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Tay_(bot)" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Tay_(bot)</a>
评论 #26978740 未加载
评论 #26977566 未加载