Hugging Face Releases Agents

I&#x27;m not 100% sure that AGI is guaranteed to end humanity like Yudkowsky, but if that&#x27;s the course we&#x27;re on, seeing news like this is depressing. Can anyone legitimately argue that LLMs are safe because they don&#x27;t have agency, when we just straight up give them agency? I know current-generation LLMs aren&#x27;t really dangerous -- but is this not likely to happen over and over again as our machine intelligences get smarter and smarter? <i>someone</i> is going to give them the ability to affect the world. They won&#x27;t even have to try to &quot;get out of the box&quot;, because it&#x27;ll have 2 sides missing.<p>I&#x27;m getting more and more on board with &quot;shut it all down&quot; being the only course of action, because it seems like humanity needs all the safety margin we can get, to account for the ease at which anyone can deploy stuff like this. It&#x27;s not clear alignment of a super-intelligence is even a solvable problem.

If you want an overview, scroll down to this part of the page: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;transformers_agents#whats-happening-here-what-are-tools-and-what-are-agents" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;transformers_agents...</a><p>In short:<p>- they&#x27;ve predefined a bunch of <i>tools</i> (e.g. image_generator)<p>- the <i>agent</i> is an LLM (e.g. GPT-*) which is prompted with the name and spec of each tool (the same each time) and the task(s) you want to perform<p>- the code generated by the agent is run by a python interpreter that has access to these tools

Asking for help from those that are smarter than I am ;;<p>-<p>One of the very common things for Martial Arts Books in the past, was the fact that one were presented with a series of pics, along with some descriptions about what was being done in the pics.<p>Sometimes, these are really hard to interpolate between frames, unless you had a much larger repetoir of movements based on experience (i.e. a white belt vs another higher belt... e.g. a green belt will have better context of movement than a white belt...)<p>--<p>So can this be used to interpolate frames and digest <i>lists</i> (lists are what many martial arts count as documentation for their various arts...<p>Many of these have been passed down via scrolls with either textual transmissions, paintings and then finally pics before vids existed...<p>It would be really interesting to see if AI can interpret btwn images and or scroll text to be able to create an animation of said movements.<p>---<p>For example, not only was Wally Jay one of my teachers, but as the inventor (re-discoverer) of Small Circle JuiJitsu - his pics are hard to infer what is happening... because there is a lot of nuanced feeling in each movement that is hard to convey via pics&#x2F;text<p>But if you can interpolate btwn frames, and model the movements, its game changing because through such interpolations on can imagine that you can get any angle of viewership -- and additionally, one can have the precise positioning and translucent display of bone&#x2F;joint&#x2F;muscle articulation such that one may provide for a deeper insight into the kinematics behind each movement.

I&#x27;ve been thinking lately of the two tiered reasoner + tools architecture inspired by LangChain, simonw&#x27;s writing[0] and this is right along those lines.<p>We&#x27;re trying too hard to have one model do it all. If we coordinate multiple models + other tools (ala ReAct pattern) we could make the systems more resistant to prompt injection (and possibly other) attacks and leverage their respective strengths and weaknesses.<p>I&#x27;m a bit wary of tool invocation via python code instead of prompting the &quot;reasoning&quot; LLM to teach it about the special commands it can invoke. Python&#x27;s a good crutch because LLMs know it reasonably well (I use a similar trick in my project, but I parse the resulting AST instead of running the untrusted code) so it&#x27;s simpler to prompt them.<p>In a few iterations I expect to see LLMs fine tuned to know about the standard toolset at their disposal (eg. huggingface default tools) and further refinement of the two-tiered pattern.<p>[0] <a href="https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;Apr&#x2F;25&#x2F;dual-llm-pattern&#x2F;" rel="nofollow">https:&#x2F;&#x2F;simonwillison.net&#x2F;2023&#x2F;Apr&#x2F;25&#x2F;dual-llm-pattern&#x2F;</a>

Follow up Guide that explains how to create your own tools: <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;custom_tools" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;custom_tools</a>

Cool! The DX is tricky to nail, when combined with LLM&#x27;s tendency to hallucinate.<p>I asked it to extract some text from an image, which it dutifully tried to do. However the generated python kept throwing errors. There&#x27;s no image -&gt; text tool yet, so it was trying to use the image segmenter to generate a mask and somehow extract text from that.<p>It would be super helpful to:<p>1) Have a complete list of available tools (and &#x2F; or a copy of the entire prompt given to the LLM responsible for generating python). I used prompt injection to get a partial list of tools and checked the Github agent PR for the rest, but couldn&#x27;t find `&lt;&lt;all_tools&gt;&gt;` since it gets generated at runtime (I think?).<p>2) Tell the LLM it&#x27;s okay to fail. E.g.: &quot;Extract the text from image `image`. If you are unable to do this using the tools provided, say so.&quot; This prompt let me know there&#x27;s no tool for text extraction.<p>Update: per <a href="https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;custom_tools" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;docs&#x2F;transformers&#x2F;custom_tools</a> you can output a full list of tools with `print(agent.toolbox)`

Whoa this is super awesome, kind of makes a ton of sense since HF pretty much dominates the market for model hosting and interfacing. The documentation actually looks about as complex as langchain. Gonna give it a go to query the docs with an agent to get an example (going full circle).

Kinda what people are asking for, I mean people are really attracted to &quot;describe a task&quot; as opposed to &quot;create a training set&quot;.

They also released today StarChat, their code model fine tuned as an assistant<p>Might be good to try with CodeGPT, AutoGPT or BabyAGI

From the documentation, HF Agents are much better explained than LangChain but not easier to use, and due to multimodality it may actually be more arcane to use.

Could use LocalAI to get around this: “The openAI models perform better (but require you to have an openAI API key, so cannot be used for free);”<p><a href="https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;selfhosted&#x2F;comments&#x2F;12w4p2f&#x2F;localai_openai_compatible_api_to_run_llm_models&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.reddit.com&#x2F;r&#x2F;selfhosted&#x2F;comments&#x2F;12w4p2f&#x2F;localai...</a>

If you are like me and you tried to copy paste the python commands and it did not work, you need to generate an access token. Here is what you should do:<p>1. Sign up (<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;</a>) to hugging face.<p>2. Setup access tokens (<a href="https:&#x2F;&#x2F;huggingface.co&#x2F;settings&#x2F;tokens" rel="nofollow">https:&#x2F;&#x2F;huggingface.co&#x2F;settings&#x2F;tokens</a>)<p>3. Install or Upgrade some dependencies `pip install huggingface_hub transformers accelerate`<p>4. From the terminal run `jupyter lab`<p>5. Then, if I did not forget any other dependencies you can just copy paste<p>```python<p>from huggingface_hub import login from transformers import HfAgent<p>login(&quot;hf_YOUR_HUGGING_FACE_TOKEN&quot;)<p>agent = HfAgent(&quot;<a href="https:&#x2F;&#x2F;api-inference.huggingface.co&#x2F;models&#x2F;bigcode&#x2F;starcoder" rel="nofollow">https:&#x2F;&#x2F;api-inference.huggingface.co&#x2F;models&#x2F;bigcode&#x2F;starcode...</a>&quot;)<p>agent.run(&quot;Is the following `text` (in Spanish) positive or negative?&quot;, text=&quot;¡Este es un API muy agradable!&quot;)<p>```

This is beautiful, but is there a decent way to plow through say, 20TB of text and put that into a vector database (encoder only)? It would be quite a great addition, especially if the vectors could then be translated into other forms (different language, json representation, pull out names&#x2F;NER, etc) by just applying a decoder to the database.

If a typical LLM has decent representation of the languages in question (and you&#x27;d be surprised how little decent is with all the positive transfer that goes on during training) then outsourcing translation is just a downgrade. a pretty big one in fact.<p><a href="https:&#x2F;&#x2F;github.com&#x2F;ogkalu2&#x2F;Human-parity-on-machine-translations">https:&#x2F;&#x2F;github.com&#x2F;ogkalu2&#x2F;Human-parity-on-machine-translati...</a><p>T5 seems to be the default so i get why it&#x27;s done here. Just an observation.

This seems to be an interpretation similar to that of langchain.

As this LLM agent architecture continues to evolve and improve, we will probably see a lot of incredible products built on top of it.

How does this compare to langchain agents?