In the previous part we got our test suite set up and running. We even saw how to generate a test that is random, without having to write a lot of string handling in Robot Framework.

In this part we will start writing dynamic tests. (We will also ditch the taco example for something more relevant!)

04 – Dynamic Tests from Data

Let’s start with creating dynamic tests from data:

04_dynamic_tests.py:

from robot.api import TestSuite, ResultWriter

OUTPUT_PATH_PREFIX = "./output/04-dynamic-tests"

servers = [
    {"name": "DNS Servers", "ips": ["8.8.8.8", "1.1.1.1"]},
    {"name": "NTP Servers", "ips": ["129.6.15.28", "132.163.97.1"]},
]

suite = TestSuite(name="Testing Connectivity to Servers")

for server in servers:
    test = suite.tests.create(f"Testing {server['name']}")

    for ip in server["ips"]:
        test.body.create_keyword("Log", args=[f"Need to test connectivity to {ip}."])

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

In this example we have a provided list of servers we want to test to and a given set of IPs in the servers variable:

servers = [
    {"name": "DNS Servers", "ips": ["8.8.8.8", "1.1.1.1"]},
    {"name": "NTP Servers", "ips": ["129.6.15.28", "132.163.97.1"]},
]

This data can be sourced from, say, a nearby YAML or JSON file or via an API call. But for now we will keep the example code simple and say this is a solved problem.

Let’s say we want output test summary to be pass/fail based on the server category, “DNS Servers” and “NTP Servers”. We will then loop over the server dictionaries and create a test for each:

for server in servers:
    test = suite.tests.create(f"Testing {server['name']}")
    # ...

Then we can add the keywords for each IP to be the “assertions” of the test:

    for ip in server["ips"]:
        test.body.create_keyword("Log", args=[f"Need to test connectivity to {ip}."])

Again, we are using the Log keyword for this example. However, our tests are being dynamically generated. Our total tests are not fixed. If we want to add new server categories to tests, we only need to provide additional data.

If all tests pass, we should see 2 tests, 2 passed, 0 failed in our output.

What if you wanted each IP to be registered as its own test?

We could change our loops to the following:

for server in servers:
    for ip in server["ips"]:
        test = suite.tests.create(f"Testing {server['name']} - {ip}")
        test.body.create_keyword("Log", args=[f"Need to test connectivity to {ip}."])

This would create a unique test for each IP.

If all tests pass, we should see 4 tests, 4 passed, 0 failed in our output.

05 – Import Resources

Now that we are generating tests from data, let’s do actual testing. To do this in our current example, we’ll use the ping utility to try reaching these servers.

To do that we will need to import the OperatingSystem library in Robot Framework. We will also need to work with assignments to capture the output of the ping command and determine whether the run was successful.

05_import_resources.py:

from robot.api import TestSuite, ResultWriter

OUTPUT_PATH_PREFIX = "./output/05-import-resources"

servers = [
    {"name": "DNS Servers", "ips": ["8.8.8.8", "1.1.1.1"]},
    {"name": "NTP Servers", "ips": ["129.6.15.28", "132.163.97.1"]},
]

suite = TestSuite(name="Testing Connectivity to Servers")

suite.resource.imports.library("OperatingSystem")

for server in servers:
    test = suite.tests.create(f"Testing {server['name']}")

    for ip in server["ips"]:

        test.body.create_keyword(
            "Run and Return Rc", args=[f"ping {ip} -c 1 -W 5"], assign=["${rc}"]
        )
        test.body.create_keyword("Should Be Equal As Integers", args=["${rc}", "0"])

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

This looks a bit different from our previous example, but only a few lines have been modified or added.

Let’s first look at the import:

suite.resource.imports.library("OperatingSystem")

Robot Framework registers libraries and defined keywords as resources for the tests. If the library is imported from somewhere, it’s an imported resource.

This would normally be seen as:

*** Settings ***
Library           OperatingSystem

Now that we have OperatingSystem imported we can use the Run and Return Rc keyword to run the ping command. We’ll also learn how to use assignments.

test.body.create_keyword(
            "Run and Return Rc", args=[f"ping {ip} -c 1 -W 5"], assign=["${rc}"]
        )

Again we create keywords normally, along with specifying the command we want to run as an argument. We need to specify the assignment variable with the assign= argument. The value you supply should follow normal Robot Framework convention.

This would normally be seen as (if ip was specified as 8.8.8.8):

${rc}=   Run and Return Rc   "ping 8.8.8.8 -c 1 -W 5"

Once we have run the ping test and captured the return code, we can evaluate it to ensure it’s returned 0.

test.body.create_keyword("Should Be Equal As Integers", args=["${rc}", "0"])

This is equivalent to:

Should Be Equal As Integers  ${rc}  "0"

A pretty common equivalancy test.

Let’s run it and see the output:

$> python 05_import_resources.py

==============================================================================
Testing Connectivity to Servers                                               
==============================================================================
Testing DNS Servers                                                   | PASS |
------------------------------------------------------------------------------
Testing NTP Servers                                                   | FAIL |
1 != 0
------------------------------------------------------------------------------
Testing Connectivity to Servers                                       | FAIL |
2 tests, 1 passed, 1 failed
==============================================================================

Dratz! Our NTP servers don’t like to be pinged. But our DNS servers returned results, therefor we pass those tests! Again, because our tests are defined at the category level and not the IP level we are running two tests, each having two assertions.

06 – Create Keywords

Now, in the example above we likely don’t need to register our own keyword. Since we are creating the tests with Python, it’s easy to create factories to generate the necessary keywords.

But let’s say, for example, we want to collapse the Run and Return Rc and Should Be Equal As Integers keywords into a single one, called Test Connection To.

06_create_keyword.py:

from robot.api import TestSuite, ResultWriter

OUTPUT_PATH_PREFIX = "./output/06-create-keyword"

servers = [
    {"name": "DNS Servers", "ips": ["8.8.8.8", "1.1.1.1"]},
    {"name": "NTP Servers", "ips": ["129.6.15.28", "132.163.97.1"]},
]

suite = TestSuite(name="Testing Connectivity to Servers")

suite.resource.imports.library("OperatingSystem")

test_connection_to_kw = suite.resource.keywords.create("Test Connection To")
test_connection_to_kw.args = ["${ip}"]
test_connection_to_kw.body.create_keyword(
    "Run and Return Rc", args=["ping ${ip} -c 1 -W 5"], assign=["${rc}"]
)
test_connection_to_kw.body.create_keyword(
    "Should Be Equal As Integers", args=["${rc}", "0"]
)

for server in servers:
    test = suite.tests.create(f"Testing {server['name']}")

    for ip in server["ips"]:
        test.body.create_keyword("Test Connection To", args=[ip])

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

Again, user-defined keywords are resources for tests, like library imports. But instead of importing them, we will create them:

test_connection_to_kw = suite.resource.keywords.create("Test Connection To")

Because we need to add keywords and args to this new keyword, we are going to simplify the use by capturing the created object (a UserKeyword) and assigning it to test_connection_to_kw.

We will need to receive the IP to ping as an argument. Like args being a keyword argument for calling/using exisitng keywords, it’s an attribute of the UserKeyword object for us to define.

test_connection_to_kw.args = ["${ip}"]

Again, we use the syntax we are used to for defining variables in Robot Framework.

Next a similar method of adding the command run and assertion keywords to our test is used to add them to our new keyword:

test_connection_to_kw.body.create_keyword(
    "Run and Return Rc", args=["ping ${ip} -c 1 -W 5"], assign=["${rc}"]
)
test_connection_to_kw.body.create_keyword(
    "Should Be Equal As Integers", args=["${rc}", "0"]
)

Instead of calling .body.create_keyword(...) on our test, we are calling it on our UserKeyword object.

That’s all we need to do to define the keyword. Now it’s time to use it!

test.body.create_keyword("Test Connection To", args=[ip])

Our tests now call a single keyword instead of two. This will simplify matters if we decide to change how we generate the tests. And again, because we are using native Python we can create a factory that generates our keywords (maybe even making those dynamic too!) on the fly.

Our output in our HTML will be marginally different with this structure because it will make reference to the new keyword instead of as in example five, which called the commands directly. However, it’s functionally identical:

$> python 06_create_keyword.py 
==============================================================================
Testing Connectivity to Servers                                               
==============================================================================
Testing DNS Servers                                                   | PASS |
------------------------------------------------------------------------------
Testing NTP Servers                                                   | FAIL |
1 != 0
------------------------------------------------------------------------------
Testing Connectivity to Servers                                       | FAIL |
2 tests, 1 passed, 1 failed
==============================================================================

Bummer! Our NTP servers still don’t like to be pinged. Oh well!

I hope you enjoyed this two-part series on generating dynamic Robot Framework test cases by calling the API directly!

Again, feel free to drop messages in the comments if you have questions. As well, all source code from both parts can be found on GitHub.

Robot Framework is a useful testing tool that generates a great human-readable report of your tests. Additionally, their domain-specific language (DSL) is one that can be written by both engineers and non-engineers alike.

However, one complexity to using Robot Framework is that currently there is no way to generate your test cases dynamically. Frequently, people drop into Python code or scripts to do more custom tests of their code or infrastructure. Using templating tools like Jinja can also help create a more varied test suite between runs. For a lot of use cases, these can be sufficient; although both can quickly become tedious to update or can obscure exactly what is being tested.

Thankfully since Robot Framework’s DSL must be parsed, the package exposes a full API to directly trigger this parsing through the robot.api modules.

00 – Introduction

I will be covering these concepts in a layered approach, consisting of two major parts. Part 1 (this post) will cover getting our tests running without using any .robot files or robot CLI commands. I will explain what each line means so that you can know where to augment for your use case. By the end of Part 1, you should be able to generate dynamic test contents. In Part 2, I will build upon these foundations and generate a dynamic test suite, including generating our own keywords.

The source code for the examples in each section of these posts can be found on GitHub.

Before getting started, be sure to install the robotframework package via your package manager of choice. In the example repo above we use Poetry, however you can use pip if you’d like.

Additionally, we are using version 4.x of Robot Framework, which made some API changes. So if you’re using 3.x, some of the syntax below will be different. Most IDEs should provide some hinting for the version you are using, but the biggest change to note is the migration to creating keywords on .body of the test versus it being an attribute on the keyword added.

If you are using 3.x and are having significant challenges writing your own tests, feel free to open issues on the example repo above.

Let’s get started with setting up our test suite and getting it to run:

01 – Core Concept

Let’s review a trivial example of a Robot Framework test suite:

01_core_concept.robot:

*** Test Cases ***
Taco Time
    Log     "Fish tacos are the best tacos."

Yes, I hold strong opinions about my tacos. This isn’t even a test, since the Log keyword almost never causes tests to fail. Bear with me as we will build upon this foundation, and doing anything with imports requires other foundations to be in place when writing them in Python.

So how would we write this in Python?

01_core_concept.py:

from robot.api import TestSuite, ResultWriter

suite = TestSuite(name="My Very Simple Test Suite")

test = suite.tests.create("Taco Time")

test.body.create_keyword("Log", args=["Fish tacos are the best tacos."])

suite.run(output="./01-core-concept-run-output.xml")

ResultWriter("./01-core-concept-run-output.xml").write_results()

Hmmm… three lines to six? Why would we do this? For tests as simple as this, I would suggest you write .robot files. However, once we get to the dynamic parts, your templating engine or other Python code may add an order of magnitude of complexity than writing them natively.

Let’s review each line and discuss what each line does.

from robot.api import TestSuite, ResultWriter

If you’ve written Python before you will know what this line does—we will need the TestSuite and ResultWriter to define our tests and generate our output HTML files, respectively.

suite = TestSuite(name="My Very Simple Test Suite")

Here we establish the test suite that we are going to run. When writing native DSL, this is done automatically by creating the .robot file. The name of the suite is based on the filename or can be overridden via command line.

test = suite.tests.create("Taco Time")

This is equivalent to the Taco Time line under the *** Tests *** section in our native DSL example.

Here we define our first test. By assigning our .create calls to a variable, it provides an easy way to attach keywords and tests to our parent objects.

test.body.create_keyword("Log", args=["Fish tacos are the best tacos."])

Finally! Let’s create some keywords in our test! This can quickly feel more intuitive than the native DSL, which requires tabs or multiple spaces to separate arguments and keywords.

This is equivalent to:

   Log     "Fish tacos are the best tacos."

We will get to assignments in a further section.

Our test is written; let’s run it:

suite.run(output="./01-core-concept-run-output.xml")

Normally, when you call robot it’s running both the test and result generation in one. In Python, we must do these separately, but this allows us to skip the HTML generation should we want to run a tool like rebot on a large test suite without passing additional arguments.

Robot Framework outputs the results of the test first in an XML file (by default output.xml) then parses that XML file into the human-readable HTML report and log files.

Calling suite.run executes the tests, so it will have to be one of the last functions you call.

Also, we should specify the output path of the XML in this step, as we will need it in the next step, the ResultWriter:

ResultWriter("./01-core-concept-run-output.xml").write_results()

Here we point the ResultWriter at the previous suite run’s output file and trigger the write_results function. This will output the log and report files to the current directory as log.html and report.html respectively, as you would expect from calling robot.

If you saved this file as 01_core_concept.py like our example, you would call python 01_core_concept.py. You should see:

$> python 01_core_concept.py

==============================================================================
My Very Simple Test Suite                                                     
==============================================================================
Taco Time                                                             | PASS |
------------------------------------------------------------------------------
My Very Simple Test Suite                                             | PASS |
1 test, 1 passed, 0 failed
==============================================================================
Output:  /path/to/current/directory/01-core-concept-run-output.xml

As well as your desired output, HTML files should be present.

Awesome! Now that we have the setup out of the way, let’s continue with making more dynamic tests. But before we do that, we should avoid stepping on other tests’ output and results.

02 – Sidebar: Organized Output

Let’s review the above Python file with a slightly more organized output:

02_organized_output.py:

from robot.api import TestSuite, ResultWriter

OUTPUT_PATH_PREFIX = "./output/02-organized-output-suite-run"

suite = TestSuite(name="My Very Simple Test Suite")

test = suite.tests.create("Taco Time")

test.body.create_keyword("Log", args=["Fish tacos are the best tacos."])

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

There are a few lines that have been modified that you will see unchanged in the rest of the tutorial:

OUTPUT_PATH_PREFIX = "./output/02-organized-output-suite-run"

Here we just define a constant to prefix all outputs with a directory and filename prefix.

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

Here we generate the output XML filename using f-strings and our prefix.

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

Here we use the same templated XML filename and specify the report and log paths using the same prefix, instead of using the default report.html and log.html paths.

These changes will make it easier for you to run the example code and any future tests you write.

Let’s write a more dynamic test:

03 – Dynamic Test Arguments

Maybe you want to randomize your favorite tacos that you log?

We’ll begin with the full file and then dive into the changes.

03_dynamic_test_args.py:

from robot.api import TestSuite, ResultWriter
from random import choice

OUTPUT_PATH_PREFIX = "./output/03-dynamic-test-args"

taco_types = ["Fish", "Chicken", "Vegetarian"]
best_taco = choice(taco_types)

suite = TestSuite(name="My Very Simple Test Suite")

test = suite.tests.create("Taco Time")

test.body.create_keyword("Log", args=[f"{best_taco} tacos are the best tacos."])

result = suite.run(output=f"{OUTPUT_PATH_PREFIX}-output.xml")

ResultWriter(f"{OUTPUT_PATH_PREFIX}-output.xml").write_results(
    report=f"{OUTPUT_PATH_PREFIX}-report.html", log=f"{OUTPUT_PATH_PREFIX}-log.html"
)

We import the choice function from the random module so we can grab a random entry from our taco_types list.

Then, instead of our args being a static string, it’s an f-string. Since this string is rendered as the test is written, to the test runner it’s the same normal string it would expect. Except now it’s different with each test run.

Hopefully, this will get you to start thinking about how you can write more dynamic tests, which reduces a significant headache and burden from doing string handling inside the Robot Framework test itself.

In the next part, we will cover creating multiple tests derived from data, importing libraries, and creating our own keywords using native Python robot.api calls.