Visual ChatGPT

The &quot;memory usage&quot; section of the README highlights the surprising fact that image generation models need <i>much</i> less memory than text-based language models. ChatGPT itself is by far the most resource-hungry part of the system.<p>Why is that so? It seems counterintuitive. A single picture snapped with a phone takes more space to store than the text of all the books in a typical home library, yet Stable Diffusion runs with 5 GB of RAM while LLAMA needs 130 GB.<p>Can someone illuminate what&#x27;s going on here?

Meta will probably soon release a competing technology. It will be called &quot;DALL-E LLaMA&quot;.

This feels like it owes more to LangChain than a link at the bottom of the page.<p>Compare their prompt:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;visual-chatgpt&#x2F;blob&#x2F;main&#x2F;visual_chatgpt.py#L51">https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;visual-chatgpt&#x2F;blob&#x2F;main&#x2F;visual...</a><p>With that of the LangChain ReAct conversational agent:<p><a href="https:&#x2F;&#x2F;github.com&#x2F;hwchase17&#x2F;langchain&#x2F;blob&#x2F;master&#x2F;langchain&#x2F;agents&#x2F;conversational&#x2F;prompt.py">https:&#x2F;&#x2F;github.com&#x2F;hwchase17&#x2F;langchain&#x2F;blob&#x2F;master&#x2F;langchain...</a><p>Also it seems appropriate to cite the original ReAct paper (from Google mainly)<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.03629" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.03629</a>

Man, Microsoft is kicking ass at AI. Maybe the others have great AI models too but haven’t seen any large company release product after product with AI.

Very cool. It&#x27;s almost as if that chat session is a terminal, but instead of running commands you run prose. Very much a new HCI paradigm.

That&#x27;s neat, but it&#x27;s not doing anything in the latent space of ChatGPT, is it? As I understand, it basically teaches the assistant to use SD for generating images&#x2F;descriptions, but comes with all the limitations of the image model being used (as opposed to a leap in results quality such as GPT 3.5 itself was). Teaching it to use tools is of course an interesting concept itself, though.

i have been trying for an hour and am completely unable to run this project. currently facing a &quot;Building wheel for numpy (pyproject.toml) did not run successfully.&quot; error.<p>the state of python dependency management and project distribution is just abjectly horrible.<p>---<p>update: perhaps spoke too soon. just made it work! <a href="https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;visual-chatgpt&#x2F;issues&#x2F;37">https:&#x2F;&#x2F;github.com&#x2F;microsoft&#x2F;visual-chatgpt&#x2F;issues&#x2F;37</a>

We&#x27;re at the point where these generative AIs are good enough that they&#x27;re doing things which are really surprising and unexpected and kind of exciting, but they&#x27;re bad enough that almost everything they create falls somewhere between mediocre and dogshit.<p>I really hope, if these this stuff is going to be ubiquitous, that there are big strides made in improving the quality of the output, very soon. The novelty of seeing fake screencaps of Disney&#x27;s Beauty and the Beast directed by David Cronenberg is wearing off fast, and aside from some very niche use cases (write some boiler plate code for this common design pattern in this very popular language) I haven&#x27;t found much it&#x27;s actually useful for

I think GPT is super useful but can&#x27;t seem to eke any value out of DAL-E. Yes, it can draw a bear in a business suit on the beach well, which is impressive but I can&#x27;t think of how to utilize this.<p>As an example, I&#x27;ve tried to get it to draw architecture diagrams, it draws a few boxes but then places the strangest text on those boxes.

Wow, this is very timely! I just finished up a script that uses ChatGPT (via openAI APIs) to read my customer support messages on Etsy and generate a response. Since I often send and receive images via Etsy support (my customers can customize the product with images) I have been searching for a way to let ChatGPT &quot;know&quot; what the image is. Current the script just inserts the text &quot;&lt;uploaded image&gt;&quot;, but I was just hacking together something using stable-diffusion-webui&#x27;s API (interrogate using CLIP), but was struggling with a few things. I took a break to browse HN and this pops up!<p>I will definitely be taking a look to see how this works and will try to get it integrated with my script.

This reminds me of Christina&#x27;s workstation in Westworld Season 4

Linked paper is available here: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.04671" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2303.04671</a>

I think the chat interface is a bit restrictive when it comes to multimodal models. A much cleaner interface would be an &quot;AI notebook&quot; where the user can move, compare, rerun blocks. Also sharing, versioning and collaborating with others on notebooks is more straightforward.

&quot;ChatGPT, I meant a desk with legs.&quot;<p>For a second, I thought this was a Visual Studio-related plugin.

ChatGPT now is not only a simple standalone AI model, but a powerful AI core engine, and more and more people or companies will develop more and more interesting things based on ChatGPT. Like this awesome visual ChatGPT.

Microsoft is releasing second toy while Google had trouble launching its first.

The most incredible thing about this system is that it uses Stable Diffusion (the open source AI art generator), rather than DALL-E (the proprietary closed art generator owned by OpenAI).<p>The fact that even Microsoft, which partially owns OpenAI, is giving up on DALL-E shows the power of building an open-source community around models with published, downloadable weights.

If you are trying to run this on a single GPU, please be aware the models take up a lot of memory. You can reduce the number of tools by modifying the self.tools portion of the python script

I guess one of the advantage of being early is that Microsoft get to pick all the low hanging fruit first.<p>All of these products are very useful and interesting by itself but it is still too early to know if MS can continue to refine and maintain a competitive edge. Dall-E basically died in a few months, unable to compete. Hopefully these other stuff will have better fate.

I think they ate using StableDiffusion and not Dall-E? Which makes it kind of funny

I have tried to use my Macbook pro(M2 pro) to run it out, but failed to download the massive file。

There are more examples in the paper:<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2303.04671.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;2303.04671.pdf</a>

The shit has an MIT license... then requires an API key. Open source all the way, guys! Microsoft loves Open Source!

The comprehension thrown around in this thread is beautiful. Love the passion.

Are there any recommendable resources for learning about designing these kind of system architectures?

hmmm can I use this to see how far away we are now<p><a href="https:&#x2F;&#x2F;karpathy.github.io&#x2F;2012&#x2F;10&#x2F;22&#x2F;state-of-computer-vision&#x2F;" rel="nofollow">https:&#x2F;&#x2F;karpathy.github.io&#x2F;2012&#x2F;10&#x2F;22&#x2F;state-of-computer-visi...</a>

Happy that this is &lt;8gb vram. Neatly fits into medium&#x2F;highish consumer GPUs

future AI systems based on LLMs and other foundation models might think less like individuals and more like companies. Ironically, LLMs might finally make symbolic AI possible! The way I see it, symbolic AI was always missing a small sprinkle of &quot;general intelligence&quot; too amooth things out, to grease the gears and connect interfaces. I feel like LLMs have that little bit of magical &quot;generality&quot; so we can start building &quot;symbolic&quot; AI systems which produce work by managing a number of black box models. It is like a company: protocols and management structures are a sort of symbolic AI that connects black box humans to eachother.

Ive created a little api to grab images from pages to embed in chats. Was surprisingly easy to control with natural language.<p><a href="https:&#x2F;&#x2F;aimgsrc.com" rel="nofollow">https:&#x2F;&#x2F;aimgsrc.com</a>

the pace of all this is astonishing, this is amazing

Endless new possibilities for online scammers. Bright times ahead.

Now is a really good time to make a start up called skynet.