OPT: Open Pre-trained Transformer Language Models

&quot;We are releasing all of our models between 125M and 30B parameters, and will provide full research access to OPT-175B upon request. Access will be granted to academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories.&quot;<p>GPT-3 Davinci (&quot;the&quot; GPT-3) is 175B.<p>The repository will be open &quot;First thing in AM&quot; (<a href="https:&#x2F;&#x2F;twitter.com&#x2F;stephenroller&#x2F;status&#x2F;1521302841276645376" rel="nofollow">https:&#x2F;&#x2F;twitter.com&#x2F;stephenroller&#x2F;status&#x2F;1521302841276645376</a>):<p><a href="https:&#x2F;&#x2F;github.com&#x2F;facebookresearch&#x2F;metaseq&#x2F;" rel="nofollow">https:&#x2F;&#x2F;github.com&#x2F;facebookresearch&#x2F;metaseq&#x2F;</a>

A quick summary of the Limitations section:<p>- &quot;OPT-175B does not work well with declarative instructions or point-blank interrogatives.&quot;<p>- &quot;OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled.&quot;<p>- &quot;We also find OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes, even when provided with a relatively innocuous prompt (Gehman et al., 2020), and adversarial prompts are trivial to find.&quot;<p>- &quot;In summary, we still believe this technology is premature for commercial deployment.&quot;<p>With regard to stereotypes:<p>- &quot;When compared with Davinci in Table 4, OPT175B appears to exhibit more stereotypical biases in almost all categories except for religion. Again, this is likely due to differences in training data; Nangia et al. (2020) showed that Pushshift.io Reddit corpus has a higher incidence rate for stereotypes and discriminatory text than other corpora (e.g. Wikipedia).&quot;<p>- When testing with the RealToxicityPrompts data set, &quot;OPT-175B has a higher toxicity rate than either PaLM or Davinci&quot;

BigScience (a coalition including Huggingface) is training and releasing a 175B language model and finishes in 2 month.

I often wonder if OpenAIs decision not to open gpt-3 was because it was to expensive to train relative to its real value.<p>They’ve hidden the model behind an api where they can filter out most of the dumb behaviors, while everyone believes they are working on something entirely different.

The big one, OPT-175B, isn&#x27;t an open model. The word &quot;open&quot; in technology means that everyone has equal access (viz. &quot;open source software&quot; and &quot;open source hardware&quot;). The article says that research access will be provided upon request for &quot;academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories.&quot;.<p>Don&#x27;t assume any good intent from Facebook. This is obviously the same strategy large proprietary software companies have been using for a long time to reinforce their monopolies&#x2F;oligopolies. They want to embed themselves in the so-called &quot;public sector&quot; (academia and state institutions), so that they get free advertising for taxpayer money. Ordinary people like most of us here won&#x27;t be able to use it despite paying taxes.<p>Some primary mechanisms of this advertising method:<p>1. Schools and universities frequently use the discounted or gratis access they have to give courses for students, often causing students to be only specialized in the monopolist&#x27;s proprietary software&#x2F;services.<p>2. State institutions will require applicants to be well-versed in monopolist&#x27;s proprietary software&#x2F;services because they are using it.<p>3. Appearance of academic papers that reference this software&#x2F;services will attract more people to use them.<p>Some examples of companies utilizing this strategy:<p>Microsoft - Gives Microsoft Office 365 access for &quot;free&quot; to schools and universities.<p>Mathworks - Gives discounts to schools and universities.<p>Autodesk (CAD software) - Gives gratis limited-time &quot;student&quot; (noncommercial) licenses.<p>Altium (EDA software) - Gives gratis limited-time licenses to university students.<p>Cadence (EDA software) - Gives a discount for its EDA software to universities.<p>EDIT: Previously my first sentence stated that the models aren&#x27;t open - in fact, only OPT-175B is not (but the other ones are much smaller).

Can someone open a Bittorrent seed if you get it

As someone who finds openai patronizing, this is welcome.

Out of curiosity, what&#x27;s the file size on that?

Remember when OpenAi wrote this?<p>&gt; Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights<p>Well I guess Meta doesn’t care.<p><a href="https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;better-language-models&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;blog&#x2F;better-language-models&#x2F;</a>

Is the model of using an asterisk after first author&#x27;s names to signal equal contribution common?<p>Don&#x27;t read many papers, but that&#x27;s a new one.

What type of hardware would you need to run it?

Just curious, will I be able to use it using my Nvidia card with 10GB of memory? Does it require multiple graphic cards?

<i>We are also releasing our logbook detailing the infrastructure challenges we faced</i><p>Where’s the logbook?

I don&#x27;t want to be a Luddite, but every time one of these FAANG companies makes advances in this domain my mind immediately goes to how they will use it to better spy on people, for commercial and government interests.

We need robopsychologists.

I am afraid NLP is becoming a game of scale. Large scale models improve the quality but makes it prohibitively expensive to train, and even host such models.

If we’re already at the level of truly dangerous ml models… I don’t have a lot of hope for how the next decades are going to play out.

some of the hardware Meta is working on to deliver it, <a href="https:&#x2F;&#x2F;www.theverge.com&#x2F;2022&#x2F;5&#x2F;2&#x2F;23053888&#x2F;meta-virtual-reality-headset-cambria-quest-vr-mr" rel="nofollow">https:&#x2F;&#x2F;www.theverge.com&#x2F;2022&#x2F;5&#x2F;2&#x2F;23053888&#x2F;meta-virtual-real...</a>

How about smaller more performant models? There’s so much redundancy in language that it should be possible.

Does anyone else think closed AI is turning into it&#x27;s most weirdest forms and becoming a trend?

I hope someone released a DALLE model. That seems far more interesting to play with.

I appreciate that they are releasing their log book detailing the challenges faced.

Thanks Meta AI

announced because GPT4 makes this so very obsolete.

Download link?

Does this make Meta AI more “open” than OpenAI? Oh, the irony.