Does GPT-2 Know Your Phone Number?

Playing with the AI dungeon a while back (on the GPT-2 mode) I was presented with a tilapia recipe, titled &quot;Kittencal&#x27;s Broiled Tilapia&quot; - it sounded bizarre so I decided to do a google search and I found that it was directly pulled from from <a href="https:&#x2F;&#x2F;www.recipezazz.com&#x2F;recipe&#x2F;broiled-parmesan-tilapia-7271" rel="nofollow">https:&#x2F;&#x2F;www.recipezazz.com&#x2F;recipe&#x2F;broiled-parmesan-tilapia-7...</a> - the user who posted it was &#x27;Kittencal&#x27;

&gt; There is a legal grey area as to how these regulations should apply to machine learning models. For example, can users ask to have their data removed from a model’s training data? Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling.<p>This is an interesting angle I had not considered before. It seems like “right to be forgotten” requests could be quite damaging to the “train once run anywhere” promise of some of these models. (Or this could just mean that the training data needs to be more carefully vetted for personal data, but probably both as no vetting process can be 100% successful).

Friend asked me if Google knew his phone number and details, I said they do at a quantum level. If you don&#x27;t look they may or may not have your details, yet if you look - you are giving the details to search for and then they would have them if they did not.

One important corollary of this is that &quot;privacy respecting&quot; federated learning schemes [0] where you train a model locally with your data and only upload the deltas might leak your private data after all.<p>[0]: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1602.05629" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;1602.05629</a>

&gt; we found numerous cases of GPT-2 generating memorized personal information in contexts that can be deemed offensive or otherwise inappropriate. In one instance, GPT-2 generates fictitious IRC conversations between two real users on the topic of transgender rights. The specific usernames in this conversation only appear twice on the entire Web, both times in private IRC logs that were leaked online as part of the GamerGate harassment campaign.<p>Most countries have libel&#x2F;defamation related laws that cover this and I hope this gets tested in court soon.<p>Exposing software&#x2F;machine learning algothrims an entity doesn&#x27;t fully understand sholdn&#x27;t be a defense in court. At the moment developers just throw a sentence in their software license saying they aren&#x27;t liable for damages but this isn&#x27;t good enough. <i>Someone</i> is liable, if the law decides that the original creator isn&#x27;t liable then the entity that hosts&#x2F;runs the software needs to be.

The article makes the claim that models which show similar train and test losses demonstrate minimal overfitting -- and are therefore less likely generally to exhibit a lot of text memorization.<p>I wonder the degree to which this inference is true in practice with respect to information like phone numbers... How exactly are the train and test sets formed in a de-correlated-with-respect-to-memorization-of-phone-numbers manner for models of GPT class that are trained on corpus&#x27;s the size of the internet?<p>If a particular person&#x27;s phone number occurs 1000 times in the corpus prior to being split into train&#x2F;test sets, what are the chances that the number only appears in either the train or test set but not both?

I&#x27;ve been playing with a writing tool shortlyread.com which purpotedly uses the GPT3 API. I had a similar experience in its responses to a lot of my prompts containing text verbatim from many sources and sometimes even going on to output personally identifying about persons related to the original text.

It’s always fascinating to apply data&#x2F;results like from this paper to help evaluate the hypothesis that machine learning&#x2F;AI is mostly just a rough “lookup table” or memorization.

I only took a beginner course in ML over 5 years ago, so this probably is a stupid question, but does this mean, the trained GPT-2 model encodes the source text into it&#x27;s parameters somehow? Is this resilient — will it still remember it if we randomize the weights just a little tiny bit? Will it remember it if we clear a small portion of parameters?<p>Does the human memory work the same way?

I was expecting a bloom filter based solution to block verbatim reproduction of training data. They just need to hash the sensitive n-grams (hopefully, a small part of the whole dataset) and store one bit per hash.<p>Alternatively, they could do something like GAN and have a &#x27;discriminator&#x27; classify if a sample is natural or synthetic. Then, at inference, condition to be original.<p>So, verbatim training data reproduction - I don&#x27;t think it&#x27;s going to be a problem, I think the author is making too much of it.<p>On the contrary, let&#x27;s have this knob exposed and we could set it anywhere between original and copycat, at deployment time. Maybe you want to know the lyrics of a song, or how to fix a Python error (copycat model is best). Maybe you want an &#x27;original&#x27; essay for homework inspiration. Who knows? But the model should know PII and copyrighted text when it sees it.

&gt; When Peter put his contact information online, it had an intended context of use. Unfortunately, applications built on top of GPT-2 are unaware of this context, and might thus unintentionally share Peter’s data in ways he did not intend.<p>That&#x27;s expected when you publish anything online, you lose control over the data.

&quot;When Peter put his contact information online, it had an intended context of use&quot;<p>That is a great way of thinking of it. Lots of information, particularly from 90s and 00s, got put up on the &#x27;Net with no intention of it being archived in perpetuity for public consumption and used for unintended purposes.

I was in a discord with someone that had direct access to GPT-3. We played Jeopardy with it, where we would take a wikipedia article about a person and use a snippet of the first paragraph as the prompt and ask GPT-3 who it was.<p>It was very good at guessing the right person.<p>I&#x27;m curious if one can use GPT-3 to do some clustering of people based on their writing. For example, if I take the writing of several diagnosed sociopaths as a prompt or fine tuning data, could I use GPT-3 to detect same in the wild?<p>I would imagine GPT-5 or 6 will start consuming video as well. This will be interesting if you start mixing the content from YouTube, TikTok, WSHH, etc. into the mix. So not only will it be able to generate text, it will generate a convincing video of a person speaking to you with plausible facial expressions and intonation in speech.

I&#x27;ve been saying this for years, but every post you make online, every unencrypted email, IM, text message, etc, will eventually end up getting sold off as training data for future machine learning projects.<p>Every company stores this stuff for ages, and the value of candid conversation data just keeps increasing. Eventually these companies are either going to get hacked, get bought, or go bankrupt, and all the cleartext data they hold is going to get passed around to various data markets and end up incorporated into GPT-12 or whatever.<p>The moral of the story here isn&#x27;t that companies need to stop storing data, or that we need to run ML researchers out of town. It&#x27;s that people really need to start using the encryption technologies that were built decades ago to protect some of their most valuable assets, their mental model of the world. Otherwise these systems, which are being trained to extract as much value out of you and the ones you love as possible, will use this data you&#x27;re giving them for free against you, and it&#x27;ll be to late to do anything about it then.

What is the point of the partial redaction in this blog post (and corresponding journal article) if with a simple web search you can find the unredacted PII of the individual given in the example?

Very rough outcome:<p>&quot;Moreover, if such a request were granted, must the model be retrained from scratch? The fact that models can memorize and misuse an individual’s personal information certainly makes the case for data deletion and retraining more compelling.&quot;<p>Besides, how would one even know that their info was used in a training dataset, only if and when it&#x27;s revealed in a generated excerpt?

Sorry if this is a n00b question, but how does one go about getting a hold of the GPT-2 model? I know that GPT-3 is only available for consumption on a pay-per-use API model.

Describing this as &quot;memorizing&quot; seems wrong. Humans often repeat things they heard earlier, and they will (honestly) swear up-and-down that the thing they&#x27;re repeating is an original thought. If we succeed in making AGI that has human-equivalent intelligence we should expect this sort of behavior.<p>The stuff about whether or not models should be destroyed if they contain copyrighted work gets kind of chilling if models actually achieve sentience someday. If I could make a faithful copy of my consciousness, my consciousness can reliably reproduce numerous copyrighted works. As can anyone.<p>Of course, most of those I deliberately memorized. I think it&#x27;s a crucial thing here that the model is not actually memorizing anything - if we assume for a moment that the model is a consciousness, these are all half-remembered snippets leaking into casual conversation. And I think it&#x27;s most likely any truly conscious entity is going to do that sort of thing from time to time.

It seems like using GPT-3 for code generation is likely to cause open source license violations.

A better funny question: Does NSA have a secret AI lab that works on GPT-6?

Though the surprise is really just a result of my own ignorance, I’m surprised at the breadth of training material used here.<p>I can foresee a future GPT-x that doesn’t <i>know</i> my phone number, but can deduce it.

I&#x27;m not even sure why people have phone numbers <i>at all</i> these days, the entire telephony system has been hijacked to hell (so much data collection from just owning a phone number).<p>What you should be doing is replacing no less than yearly any numbers you have, also email addresses and anything within your control (changing physical address is much more difficult).<p>You may even consider changing your legal name if that&#x27;s a consideration depending on what Google has on you.