I am less than lukewarm on those sort of "Natural-Language-Like" DSLs that seem to mostly come out of the QA space.<p>When I worked together with a QA team for automated testing, they used Cucumber, which uses Gherkin as an english-but-not-really DSL. Now the idea is obviously, that someone who doesn't know how to code can still describe and read test scenarios. But what ended up happening eventually is that I spent a lot of time writing code to support different Gherkin expressions, but then had to take over writing the Gherkin expressions as well, because even though Gherkin -looks- like english, it still behaves like a programming language. In the end, it would've been much easier for me to just write the tests in a programming language from the start, instead of using an abstraction layer that turned out to serve no purpose.<p>What I took away from that frustrating experience is that a lot of people think that learning to code is learning a syntax of a programming language, and if you could just do away with that pesky syntax, everybody could code. But, obviously, the syntax ist just the initial barrier, and actually coding is crystallizing and formulating intent - something which programming languages are much better at than natural languages.<p>But for some reason, this low/no-code idea never seems to die down, especially in some spaces, and I have the feeling I'm just missing something here.
A QA consultancy used this tool at my previous gig. They were extremely unproductive, as in several months to deliver a few tests.<p>Honestly if you know your Python you are better off with Selenium, which is probably what Robot uses under the hood anyway.
Used it extensively for a while. I think all in all, it's a nice, mature framework. The Gherkin stuff is completely optional, I'm not fond of it and never used it. Using Python, you can create a custom DSL tailored to your needs, so that you can write tests in a tabular form. All your tests will need to adhere to this rigid tabular structure. Whether this makes sense, entirely depends on the circumstances. Compared to just using pytest, I think the main advantage is that tests can also be written and extended by people who are not familiar with Python. So if you have a separate QA team with people who are not programmers, this might make sense. Also, I did like that this rigid tabular structure made all tests very much alike, whereas when using pytest, you often end up with a hodge-podge of different ways to test things and people re-inventing the wheel all the time.
I remember this from a gig I did a while ago. Customer wanted to establish their own automated QA and thought Robot Framework was a good fit, because the language „looks easy“. The goal was to enable IT personnel that lacked a technical background, with no programming experience, to write tests.
I was sceptical from the start, and was eventually proven right (I did not enjoy this, it created more not-so-enjoyable work for me).
Test code was flaky to the point of being useless, and contained alot of redundancy.
IMHO it is really hard to write good tests without having a bit of a „programmers mindset“.
I develop tooling around Robot Framework as my main job, personally I feel biggest tradeoff of RF in Python world is that it competes against Pytest, which is my personal favourite testing framework out of any I have tried.<p>In JavaScript projects I don't have such a favourite, Jest and Mocha have been okayish when I have tried them but didn't really spark joy. In multilanguage projects (like the Playwright based robotframework-browser which I have been developing) I have enjoyed writing integration tests with Robot Framework.<p>I don't have excessive experience with other comparable QA tools (like cucumber), but I would say Robot Framework's main advantage and disadvantage are both the fact that it is constantly so close to python. (E.g. you can easily do in-line python expressions, but if you were to have a team working with Robot Framework where some are lacking Python competence those might become very confusing pretty fast.) Also for writing advanced libraries Python usage is pretty much mandatory. But I guess it's pretty rare for a DSL to support writing advanced extensions with the same DSL.
I have used RF for many years now at different companies. We use it for automated integration tests, it does not matter in which language your application is written. I do admit that it requires some programming skills, especially for the keywords. Choose the keywords wisely and ensure that the QA people know how to use them. We also use it as part of regression testing and even for acceptance tests with the customer.
From QA perspective Robot Framework is by far the best tool there is.<p>First of all, it's not web testing tool, it's not mobile testing tool, it's not REST API testing tool. Or any other specific testing tool. You can automate pretty much any testing activity using RF, including web, mobile and REST, but not in any way limited to those.<p>The killer feature from quality assurance point of view is test tagging combined with really powerful way of selecting what to include in test run and getting good reports which are supported by many many tools. Another killer feature is dead simple test instrumentation.<p>Regarding tagging. Let's take this example:<p><pre><code> *** Settings ***
Force Tags feature-xyz
Suite Setup Initialize Tests
*** Variables ***
${test_environment} dev
*** Test Cases ***
Foobar
[Tags] jira-id-001 jira-test-id-001 smoke
No Operation
Lorem
[Tags] jira-id-002
No Operation
Ipsum
[Tags] jira-id-003 bug-in-jira-001
No Operation
*** Keywords ***
Initialize Tests
Connect To Environment ${test_environment}
</code></pre>
I can select any combination of those test to be included in given test run, specify which environment to connect and send results automatically to Jira. This allows me to run only "smoke" tests against "Pull Request" environment when ever PR is opened. This also allows me to automatically run all tests every hour against "Dev" environment and submit results to Jira.<p>Like this:<p><pre><code> robot --include smoke --variable test_environment:pr
</code></pre>
That would only run the one test tagged smoke and Connect To Environment would get the value "pr".<p><pre><code> robot -i feature-xyz .
</code></pre>
Would run all tests with tag feature-xyz (and in the example file that would be all tests) against dev environment. And then I could just `curl` the XML result file from the run to Jira (given it has XRAY installed) and Jira would automatically update all the Jira tickets in mentioned in the tags with the test results. If there is no Jira Test tagged in RF test, Jira would automatically create new Jira Test for me.<p>And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc.<p>That way, when you need to know what is you test coverage, just open Jira and see it yourself.
<p><pre><code> *** Test Cases ***
Write my test using a DSL
Read examples of the DSL
Write a basic test and see that it works
Write a complex test and find something missing
Learn quirks about DSL
Implement missing things on the language that the DSL was implemented on
Write part of the test in said language
Write part of the test in DSL matching regexes
Run the test
And it works 9 out of 10 times
Install flaky extension
Add flaky tag to test
And it is green
[Teardown] Reflect about my life as a software developer</code></pre>
I test embedded devices with RF. You write most of your test drivers in python, RF simply orchestrates the tests and collects the results into a report, which works well in simple QA scenarios.<p>Robot lacks features you'd need to support larger-scale embedded test, such as with a device farm, where you need the concept of tests leasing resources. If you build a test stand with 10 testable devices and need to run a suite of 100 tests, it's ideal to run the tests in parallel as devices become available. Some test setups might require more than one instance of a testable device, or instances of more than one type of device (for example, to ensure that a V2 product can still interoperate with a V3 product.) Robot doesn't really support this.<p>While a feature to support this could be made extremely general (resource classes, instances, and leases) the RF developers have been uninterested in incorporating this aspect of test into their framework. The result is everyone who does even mid scale embedded device testing has to writes their own.<p>Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's "suite scope" is actually a file scope" for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups.
When I started writing tests for my current project, I looked at what was available and decided forgo existing testing frameworks. I haven't been disappointed.<p>Writing code to call a bunch of test functions then generate a report is really not hard and having control over the whole thing is nice.
Robot Framework is a general purpose automation framework.<p>You can use it to automate
- Web Applications (with Selenium or Playwright)
- Rest APIs
- Desktop Applications (Java Swing, WPF, SAP, ..)
- All kinds of hardware<p>It offers control-structures like most programming languages (IF/ELSE/FOR/WHILE/TRY/EXCEPT)<p>The Extensions for VS Code or PyCharm/IntelliJ offer Code-Completion, step-wise Debugging, Linting.<p>It is very hackable and can be extended using Python.
It has a great API, allowing you to connect it to all kind of other tools.<p>I guess a lot has happened since some of the people here used it.<p>But I guess, if somebody is focused on using a single programming Language (like JavaScript) and is not open to learn Python or the RF Syntax I recommend to look into another native test framework
Stop calling things that are not robots, robots.<p>Does this framework stream sensor data? No. Does this framework control actuators? No. Does this framework represent an actuated device’s configuration space? No. Does this framework perform collision detection? No. Does this framework keep an obstacle map? No. Does this framework have any planners? No.<p>Just stop already. Stop calling things a robot when they are not.
Fun to see this is still around! Used this in 2016, at a big bank. I did not enjoy it, but that can largely be attributed to the way management wanted us to use it. The tests had to be written in our native language (not English) which caused a lot of weird mixed language use behind the scenes. Then a manager would sign off on the test report, which we would deliver to him on paper.
I've been using RF for almost 8 years now. I came in touch with it, in the beginning with my C/C++ developer mind: "...what an overhead of work...".<p>But soon a realize I was wrong and start to notice the power of writing tests in robot. And how it allowed to easy the understand of what operation I did on code in the past... like 6 months before, 1 year before... and also the ability to understand more clearly how other teams' features worked and to fix their test and features many times, because we more clearly wrote test using Gherkins in RF... and see how new team members could ramp MUCH more easily inside our environments, by reading tests, most of time.<p>I was even able to implant RF inside some other business after that, bringing together that culture of writing clear tests with RF. Some of those places had the heavy culture of writing top-down requirements precisely. On those places, we integrate RF documentation and test case procedure on our process, generating final documentation using RF test case information.<p>Since them I'm helping to disseminate and evangelizing for the RF use. I saw system analysts embracing the RF, writing the first version of test cases for requirement in matter of minutes. RF provided us with much faster peace in maintaining some very complex environments.<p>We could grab requirements, and write down cases, that latter could be implemented, edited, removed or adjusted. Whatever is the test environment, testing it is a 'living' environment, and need catering from developers. RF had helped greatly maintaining such places, to produce great reports, to create this live ecosystem easy for uses.<p>Is perfect? Surely No.<p>But is much better that all other tools available around.
Is open source, easy to expand, allow use of natural language. And when well used, allow us to greatly improve our QA and DEV lives :D
I loved this tool :) ! Few years ago, as a junior I joined one global corporation to my first job and I started to use it with zero knowledge about the python or even testing at all. But learning curve was very easy & fast and as of time, we created more than 3000 E2E & integration tests for our business application, running through the Jenkins. The tests were written in a very human-readable format, so anybody at the team could easily understand the test and identify the issues. Reporting had nice (but quite slow) html & jquery page. Also, Robot editor was a recommended tool to write tests.<p>After a time, when I was starting a startup with a few junior developers which lacked the expertise, we used this tool in combination with Selenium to crawl data. It was fast & easy.<p>Always glad to see this tool is still alive. Good job Pekka Klarck!
I wrote my bachelor thesis about Robot Framework. The biggest issue i had with Robot Framework was the lack of support in IDEs. Initially you were suppose to work on Robot Framework in RIDE (dedicated IDE build on top of Eclipse) or use a Eclipse extension. I believe both of these are not supported anymore. I like Vim, so i wrote a package, which helps users when developing with Robot Framework. I also created some pull requests for Vim ALE to also support rflint.<p>We also used Robot Framework at work for automated testing, worked like charm.
Robot has it's peak maybe a couple of years ago. Although my company still uses it extensively there are other Automation frameworks that are proving to be better, like , say Cucumber, Cypress, etc.<p>But why is Robot Framework being mentioned and getting attention now? It's not like it's something new and doing anything different from the existing frameworks.
We use Robot at my current job and it works flawlessly. It can test the web front-end (Browser, wrapper for Playwright), the REST APIs (REST, RestInstance), CLI stuff and much more.<p>We selected it specifically for its versatility and the DSL is easy enough so even less technically adept testers can pick it up quickly.
The developer experience is fairly nice when it comes to Robot Framework. Nice thing is, that Robot Framework was originally designed as automation tool, so also people who do not know a lot about coding can get up to speed fairly easily.
If you use Robot Framework and have a lot of tests, which can be executed in parallel. Take a look at pabot [1]. It is a parallel executor for Robot Framework.<p>[1] = <a href="https://pabot.org" rel="nofollow">https://pabot.org</a>
My only experience with Robot was at Juniper Networks, where it was a poorly maintained bastard QA org tech. Would have been infinitely better if DEV’s and QA had a shared repo of python scripts.
As an interesting note that it is not surprise that half of showcased business logos are from Finland.<p>Robot Framework is (was?) heavily used by Nokia and then spread from there to its subcontractor network.