[hpaavola]

From QA perspective Robot Framework is by far the best tool there is.

First of all, it's not web testing tool, it's not mobile testing tool, it's not REST API testing tool. Or any other specific testing tool. You can automate pretty much any testing activity using RF, including web, mobile and REST, but not in any way limited to those.

The killer feature from quality assurance point of view is test tagging combined with really powerful way of selecting what to include in test run and getting good reports which are supported by many many tools. Another killer feature is dead simple test instrumentation.

Regarding tagging. Let's take this example:

    *** Settings ***
    Force Tags    feature-xyz
    Suite Setup    Initialize Tests


    *** Variables ***
    ${test_environment}    dev    


    *** Test Cases ***
    Foobar
        [Tags]    jira-id-001    jira-test-id-001   smoke
        No Operation

    Lorem
        [Tags]    jira-id-002
        No Operation

    Ipsum
        [Tags]    jira-id-003    bug-in-jira-001
        No Operation


    *** Keywords ***
    Initialize Tests
        Connect To Environment    ${test_environment}

I can select any combination of those test to be included in given test run, specify which environment to connect and send results automatically to Jira. This allows me to run only "smoke" tests against "Pull Request" environment when ever PR is opened. This also allows me to automatically run all tests every hour against "Dev" environment and submit results to Jira.

Like this:

    robot --include smoke --variable test_environment:pr 

That would only run the one test tagged smoke and Connect To Environment would get the value "pr".

    robot -i feature-xyz .

Would run all tests with tag feature-xyz (and in the example file that would be all tests) against dev environment. And then I could just `curl` the XML result file from the run to Jira (given it has XRAY installed) and Jira would automatically update all the Jira tickets in mentioned in the tags with the test results. If there is no Jira Test tagged in RF test, Jira would automatically create new Jira Test for me.

And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc.

That way, when you need to know what is you test coverage, just open Jira and see it yourself.

[djvdq]

As someone working as QA Automation I have to admit I hate RF. IMHO Pytest is far better and everything you wrote above can be also done there. It will be a bit more code, that's true, but overall I find pytest better tool for automated testing.

[denzil]

Having had worked with both Pytest and Robot, I have to agree. And to expand on that, Robot framework had multiple disadvantages for us. (Note that this is about 2-3 years, so maybe things improved).

It was pretty slow, just rewriting the same test using Pytest sped it up by about 50% on average and that was despite our test environment being the slow part.

The Robot language was quite error prone as it uses spaces inside single keyword as well as to separate parameters. And the IDE support (VSCode in our case) wasn't particularly great. We ended up having to write custom static analysis tools, to catch some of the more common Robot problems.

And pretty much every time something more complicated was required, it was easier to switch to Python to do it. So in the end two languages were required instead of one, making maintenance more painful.

That is not to say that Robot doesn't have its good points. For example retrying on error was much easier than in pure Python. But in the end it was much more work than just using pure Python with Pytest as testing framework.

[rasjani]

> And in order to display test statistics in Jenkins, just install RF plugin in Jenkins and instruct your job to read the output XML and you get nice statistics, reporting etc.

Sadly, that same thing can be also a negative side.. There is really no proper way to expose the results if you are not using Jenkins as your main CI. Ofcourse one can use junit reports or whip up a own listener that will generate some sort of test reports for the given ci platform...

[hpaavola]

As RF already outputs jUnit style XML file, for sure you can instruct your CI to utilize that. And at worst `robot` will return relevant exit code so CI can at least get pass/fail information from the whole process. Also, https://pypi.org/project/allure-robotframework/ generates Allure reports from RF, you might be able to utilize that.

Not sure what else one might need. Surely any relevant CI can also store the HTML files RF generates.

EDIT: Just realized who you are. We've met few times at RF events. I'm not sure what other information one might want besides jUnit, console output and exit code, Allure reports, HTML files and RF output.xml.

[rasjani]

My "proud" moment - get recognized in HN discussions - thanks for that ;)

Yeah, point was that jenkins integration and how/what reports are shown is really good but utilizing junit does not provide the same experience. Way back in the past, Peke (the lead dev for those who dont know) was even griefing about RF's build in junit support "there is no standard!" - things have changed now but when i came to RF ecosystem, the buildin junit support was *shit* and it was using a format that wasn't really even close to what some other CI systems where expecting (except jenkins) ..