I’m curious if anyone has seen a project involving LLM’s that takes a series of user actions and creates integration test code for them, something like:<p><pre><code> I goto to example.com
I click the search bar
I type in “fruit”
Expect to see a list of fruit
</code></pre>
Then the LLM would execute something like playwright to achieve each of these steps and report back if any steps failed…
Are you looking for a web service that does the whole thing? Otherwise asking ChatGPT gives you<p>convert this free text<p><pre><code> I goto to example.com
I click the search bar
I type in “fruit”
Expect to see a list of fruit
</code></pre>
to playwright script<p>const { chromium } = require('playwright');<p>(async () => {
const browser = await chromium.launch();
const context = await browser.newContext();
const page = await context.newPage();<p><pre><code> // Navigate to example.com
await page.goto('https://example.com');
// Click the search bar
await page.click('input[type="search"]');
// Type in "fruit"
await page.type('input[type="search"]', 'fruit');
// Press Enter to submit the search
await page.press('input[type="search"]', 'Enter');
// Wait for the page to load
await page.waitForLoadState('networkidle');
// Expect to see a list of fruit
const fruitList = await page.$$('ul li');
if (fruitList.length > 0) {
console.log('List of fruit found!');
} else {
console.error('Failed to find the list of fruit.');
}
// Close the browser
await browser.close();</code></pre>
})();
I have created a project for easily writing this type of test with YAML:<p><a href="https://github.com/hitchdev/hitchstory">https://github.com/hitchdev/hitchstory</a><p>I dont think that this type of task is really appropriate for an LLM though. It is better to use hard, prewritten abstractions for the truly deterministic stuff (e.g. click).<p>For other situations where you may need to do subtle trade offs (e.g. choosing an HTML selector for the search bar) an LLM will generally do a bad job.
I have never heard about the LLM-based solution, but the tests written with help of Cucumber (<a href="https://cucumber.io/tools/cucumber-open/" rel="nofollow">https://cucumber.io/tools/cucumber-open/</a>) framework looks very similar to what you're asking about.