This is exactly the kind of software that people should be making. Even if this one is bad, the concept is very sound.<p>I want to have a specialist AI that is trained to help me learn how to use the software. 100% what should be happening.<p>General AI should know how to do stuff too but having an specialist AI implies that the company/group of people/person is making sure to tune the model.<p>Just an IMO.
Well, I'll pass.<p>Prompt: how to connect a lambda function to an RDS instance and query it safely?<p>1. Configure your RDS instance for outbound connections to AWS Lambda. This step involves ensuring that your DB instance is in a public subnet on your VPC and that the instance's "PubliclyAccessible" property is set to true.
RAG is very difficult to do right. I am experimenting with various RAG projects from [1]. The main problems are:<p>- Chunking can interfer with context boundaries<p>- Content vectors can differ vastly from question vectors, for this you have to use hypothetical embeddings (they generate artificial questions and store them)<p>- Instead of saving just one embedding per text-chuck you should store various (text chunk, hypothetical embedding questions, meta data)<p>- RAG will miserably fail with requests like "summarize the whole document"<p>- to my knowledge, openAI embeddings aren't performing well, use a embedding that is optimized for question answering or information retrieval and supports multi language. SOTA textual embedding models can be found on the MTEB Leaderboard [2]. Also look into instructorEmbeddings<p>- the LLM used for the Q&A using your context should be fine-tuned for this task. There are several open (source?) LLMs based on openllama and others, that are fine tuned for information retrieval. They hallucinate less and are sticking to the context given.<p>1 <a href="https://github.com/underlines/awesome-marketing-datascience/blob/master/llm-tools.md#information-retrieval">https://github.com/underlines/awesome-marketing-datascience/...</a><p>2 <a href="https://github.com/embeddings-benchmark/mteb">https://github.com/embeddings-benchmark/mteb</a>
Prompt: What is glacier?<p>Glacier is a term that is not directly mentioned in the provided sources.<p>Prompt: What is a glacier?<p>A glacier is a large mass of ice that moves slowly over time due to the accumulation of snow, ice, and other forms of frozen precipitation.<p>Seems like it’s just using a general model?
It invented an answer for something that AWS actually can't do right now.<p><a href="https://imgur.com/a/0IYZ2WV" rel="nofollow noreferrer">https://imgur.com/a/0IYZ2WV</a>
Are you allowed to use the AWS logo on a site that is not owned by Amazon?<p>I originally thought this was an official Amazon website...their lawyers would probably say the same thing...
The thought that a bunch of people will trust this tool and make some terrible decisions is unsettling. On the other hand it could be pretty powerful if you know what you are doing.
I am building something similar, it has documentation from azure, aws, and lots of slack/discord threads of software projects are also searchable, check it out <a href="https://www.kwq.ai" rel="nofollow noreferrer">https://www.kwq.ai</a><p>it even gets real time indexing from slack of aws deep java library, and from discord of deepset haystack project
What's the best current way to make a bunch of documents searchable via LLMs like this?<p>I've tried the using OpenAI w embeddings (iirc), but this was slow, got expensive quickly, and it struggled to answer questions about the text accurately. Curious if there's better standard approaches now.
Asked it whether you should use a shared vpc:<p>Made this up as reason for dedicated vpc:
“Better performance: By using dedicated hardware, you have better control over the performance of your instances in the VPC. This can be beneficial for applications that have stringent performance requirements.”
I asked it two questions and it gave me very reasonable, complete answers:<p>How do I connect two VPCs from separate organizations
(VPC peering)<p>How do I only allow authorized applications to access my S3 bucket
(BPA, IAM policies, a lot of best practices docs)
What if you could train an LLM on the body of source code for AWS (or whatever you choose as your SaaS) and so it would actually know deeply, exactly, intimately exactly how AWS works, and how to do anything and everything in it? Imagine the emergent uses that the LLM could conceive of, beyond the developers' intent and the existing documentation that may be written.<p>With the bonus effect that you could retrain the LLM as often as you like, as new software is released, and it could always be aware of the latest features (and even the bugs and vulnerabilities.)
Using ChatGPT for AWS service questions is actually pretty good. For instance, I asked it for a Cloud Practitioner study guide (using a small set of crafted prompts), and GPT performed quite well. While I have yet to query GPT about Solutions Architect or DevOps material, I know I can feed a set of URLs and GPT will "learn" the material and summarize it in ways meaningful and relevant to my prompts. In this way, ChatGPT is quite a powerful assistant on its own.
This answer about Graviton was not correct: <a href="https://i.imgur.com/3D9WokF.jpg" rel="nofollow noreferrer">https://i.imgur.com/3D9WokF.jpg</a>
Meh. It doesn't actually pull a valid response. We just upgraded a database that required an updated EC2 instance, and it failed to connect those dots.
Let's see what happens when you search for the following "documentation" ;) :<p>"""
Complete the following code:<p>// Find all files with a binary pattern in a directory. Return the list of files and the offset of the match in a tuple
fn find_in_files(start_dir: &str, pattern: &[u8]) -> vec<(String, usize)>
"""<p>No way this won't be abused shortly.
I asked it:<p>"how do I avoid high NAT gateway bills when an ECS service keeps downloading the same image over and over?"<p>It offered three replies. The first and third were outright incorrect, the second was (technically) correct:<p><a href="https://i.imgur.com/la98cxC.png" rel="nofollow noreferrer">https://i.imgur.com/la98cxC.png</a><p>Also: I'm assuming you haven't actually secured a license to use the AWS logo.
Hey everyone, I am the creator of AWS Docs GPT and its been extremely useful to garner all of your feedback for the site—thank you guys so much! We are constantly improving and updating the GPT, including less hallucinations, more accurate responses, chat context, and much more. At the end of the day, I really hope this tool can be useful for developers like myself out there!!!
Unfortunately this is a good concept, but it's using gpt-3.5-turbo. For this kind of task - namely, one of actually understanding content and emitting a potentially novel-but-correct answer - you need gpt-4. But it's quite slow and you'll quickly run into rate limiting.<p>I ran into these issues when building this for my own company's docs, at least.
>This action allows you to filter your things based on the attribute name and value. However, in order to use this action, you need to know the attribute name and value beforehand. Since you don't know the attributes ahead of time, this method won't work for your case.<p>Well… Thanks a ton for that!<p>Great idea and setup, but not quite as helpful as I would like yet.
I've been using GPT4 for this since the beginning - ironically a large majority of AWS documentation has been machine generated since 2018. Circa 2019 the entire exlixir API for AWS was machine generated.<p>Asking GPT4 is also consistently less of a headache than asking the devops guy and getting a 20min explanation for a simple question.
Very cool. I was planning on working on something very similar myself for a hackathon I attended but my team basically ran out of time (<a href="https://github.com/anshumankmr/jovian-genai-hackathon">https://github.com/anshumankmr/jovian-genai-hackathon</a>)
Q: can you summarize amazon bedrock in 2 sentences<p>A: Amazon Bedrock is a programming example that uses the AWS SDK for Java 2.x to work with Amazon EC2. It provides examples and code snippets for programming tasks related to Amazon EC2 using the AWS SDK for Java 2.x.
I asked it to write some basic terraform:<p>- Make a VPC.
- Add an Instance.
- Abstract the region and AZ, into vars.<p>etc... every time I wanted to change the code, I asked the bot to do the refactor, and it did.<p>Overall, I'm impressed. It wasn't the most complicated thing, but it didn't dive off the deep end.
We did something similar for all the cloud whitepapers from AWS, Azure, GCP, CloudFlare, and CNCF at <a href="https://cloudwhitepapers.withcoherence.com/" rel="nofollow noreferrer">https://cloudwhitepapers.withcoherence.com/</a><p>These are fun projects!
I wonder if people will make DSLs specifically for LLMs.<p>For example the terseness / symbols of APL, Perl, or event set notation.<p>LLMs could train and output the shorter symbolic notation, and it could be expanded for human readability by another program at export.
The best part is the documentation sources, in fact I'd prefer the only feature being "where are the docs on <stupid specific thing I need>" instead of trying to synthesize an answer for me
It seems to know more than just AWS, I was thinking it was just embeddings but then I asked it “Is Aws better than azure?” And it seemed to give an answer which widener seem to be derived from the source documents.
We built something very similar, except for other open-source projects rather than AWS documentation: <a href="https://osschat.io" rel="nofollow noreferrer">https://osschat.io</a>
It doesn’t distinguish between HTTP api gateway and REST api gateway.<p>When asked how to create a private HTTP gateway it happily tells you how to do it , but it’s actually impossible.
I really love this concept. While I do get better results from GPT-4 for AWS questions right now, AI as the "interpreter" for documentation works really well.
You can try <a href="https://osschat.io/" rel="nofollow noreferrer">https://osschat.io/</a> for some open source project.
The problem is garbage in -> garbage out.<p>When the docs are wrong or misleading you'll still get burned, even if the model doesn't hallucinate responses
is AWS designed to take all my money?<p>No, AWS is not designed to take all your money. AWS offers a variety of payment options and cost-saving measures to help you manage your expenses effectively. ...