Unit Test Guidance

Why Write Unit Tests?

Unit Tests Become a Safety Net for Developers

A comprehensive suite of unit tests can act as a safety net for developers. By frequently running the tests, they can assure their recent modifications to the code haven’t broken anything. In other words, unit tests help prevent regressions.

Unit Tests Can Contribute to Higher Code Quality

Since unit tests act as a safety net, developers become more confident when changing the code. They can refactor the code without fear of breaking things, driving the general quality of the code base up.

Unit Tests Can Contribute to Better Application Architecture

If you can add unit tests easily to a code base, that’s usually a good sign regarding the quality of the app’s architecture. So, the drive to write testable code can be an incentive for better architecture.

Detects Code Smells in your Codebase

If ease of adding unit tests to a code base is a good sign, the opposite is also true. Having a hard time creating unit tests for a given piece of code might be a sign of code smells in the code—e.g. functions that are too complex.

Writing Good Unit Tests

Tests Should Be Fast

If they’re slow, developers won’t run them as often as they should. That defeats the whole purpose of having a suite of unit tests in the first place, which is to boost the developers’ confidence to make changes to the code. The tests can’t work as the safety net if they’re not run often.

Tests Should Be Simple

There are several techniques we can apply to have a high degree of confidence in the correctness of our tests. One of those is to keep your tests with low cyclomatic complexity. Cyclomatic complexity is a code metric that indicates the number of possible execution paths a given method can follow. A piece of code with lower complexity is easier to understand and maintain, which means developers are less likely to introduce bugs when working on it. We can measure the cyclomatic complexity of your tests (using, for instance, a linter tool) and do your best to keep it low.

Test Shouldn’t Duplicate Implementation Logic

If the same person wrote both the test and the implementation, it’s possible they made the same errors in both places. Since the tests mirror the implementation, they might still pass, and the implementation could be wrong, but the tests might fool you into thinking otherwise. Resist the urge to make your tests fancy, keep them simple, and your testing suite will be better for it.

Tests Should Be Readable

This best practice overlaps a little bit with the one about keeping your tests simple. If tests are hard to read, developers are more likely to misunderstand them and introduce bugs. Test cases could be used as a form of documentation, so they obviously need to be readable.

Running Unit Tests Part of the Build Process

Automate the whole process of running the unit tests and taking some action when they fail. Your build process should execute your unit tests and mark the build as broken when the tests fail.

Writing Unit Tests in {admiral}

Plan your Unit Tests

Start by considering the derivation rule you are testing and the possible arguments/flexibilities of your function code. Then plan which scenarios you will test. These can either involve generating different input test cases or feeding them into different calls of your function.

Test Coverage

Unit tests should cover the functionality of the function. If another function g() is called within a function f(), the unit tests of f() should not test the functionality of g(). This should be tested by the unit tests of g(), i.e. unit tests should be added at the lowest level.

Tests Should be Robust to Cover Realistic Data Scenarios

For generating input test cases, it can be helpful to consider regular cases (expected common data scenarios), boundary cases (where data points are close or equal), and special cases (uncommon but valid data scenarios, e.g. missing or special characters). Although you will never cover every single eventuality of possible input data (no reliability testing method ever gives 100% certainty), you do need to give confidence that the code is robust enough to work across most data scenarios.

Testing Should Cover Possible Arguments

For the different calls of your function, consider how the user might apply your function and test a variety of possible calls, whilst still remembering the tips above that tests should be fast and simple. This is only needed in cases where the complexity and level of flexibility of your function justifies it, e.g. see the test script: https://github.com/pharmaverse/admiral/blob/main/tests/testthat/test-derive_var_extreme_flag.R.

Exported Functions

Don’t forget to add a unit test for each exported function.

Snapshot Testing

Standard unit tests are not always convenient to record the expected behavior with code. Some challenges include:

Output that is large, making it painful to define the reference output, and bloating the size of the test file and making it hard to navigate.
Text output that includes many characters like quotes and newlines that require special handling in a string.
Binary formats like plots or images, which are very difficult to describe in code: i.e. the plot looks right, the error message is useful to a human, the print method uses color effectively.

For these situations, testthat provides an alternative mechanism: snapshot tests. Snapshot tests record results in a separate human readable file and records the results, including output, messages, warnings, and errors. Review the {testthat} snapshot vignette for details.

Testing Errors and Warnings

A unit test that checks a function throws an error or issues a warning must always verify that the expected condition is raised, not just that some error or warning happens. Otherwise the test will still pass even if the function starts throwing a completely different, unintended error, or if the specific warning it is supposed to test for is accidentally removed while an unrelated warning is still triggered elsewhere in the function.

Never call expect_error()/expect_warning() without also identifying the condition being tested. Depending on where the error/warning message is defined, use one the of the following two options instead:

If it is defined in the tested function: Use expect_snapshot(..., error = TRUE, cnd_class = TRUE) (drop error = TRUE when testing a warning), which records the condition class together with the full message text in a snapshot file.
Otherwise: Use the class argument of expect_error()/expect_warning(), when the underlying condition has a class (as is the case for all assert_*() helpers and for conditions raised with cli::cli_abort()/cli::cli_warn()).

This ensures that the message is checked but avoids that many tests need to be updated when the message changes.

Good Example

Consider a function myfun1() which errors if:

The input argument is not a character scalar
The length of the input argument is greater than 32 characters.

myfun1 <- function(arg) {
  assert_character_scalar(arg)
  if (nchar(arg) > 32) {
    cli_abort(c(
      "The value of {.arg arg} is expected to be at most 32 characters.",
      i = "Its length is {.val {length(arg)}}."
    ))
  }
}

We can unit test the argument validation as follows:

test_that("myfun1 validates its arguments", {
  expect_error(myfun1(123), class = "assert_character_scalar")
  expect_snapshot(
    myfun1(long_string),
    error = TRUE
  )
}

where the first validation is tested with expect_error() since the error is thrown by assert_character_scalar(), and the second validation with expect_snapshot() since the error is thrown directly by myfun1(). Note that with expect_snapshot() we are also locking in the exact message text of the error.

Bad Example

Consider a function myfun2() which errors if the input is not a logical scalar:

  myfun2 <- function(arg) {
    assert_logical_scalar(arg)
  }

Now consider the following test:

test_that("myfun2 errors", {
  expect_error(example_fun(NA))
  expect_error(example_fun("test"))
})

This test would still pass even if assert_logical_scalar() were broken and threw an unrelated error (e.g. could not find function "assert_logical_scalar"), because expect_error() without class, a message, or a regular expression accepts any error. The same applies to expect_warning(): expect_warning(my_fun()) on its own only confirms that some warning was raised, not that it is the one the test is meant to verify.

Set up the Test Script

Within the tests/testthat folder of the project, add a script with the naming convention test-<script_containing_function>.R., the unit test script can be created from the console also, as follows:

usethis::use_test("<script_containing_function>")

the testing framework used is testthat and has the following format :

## Test 1: <Explanation of the test> ----
test_that("<function_name> Test 1: <Explanation of the test>", {
  
  input <- dplyr::tribble(
    ~inputvar1, ~inputvar2, ...
    <Add Test Data Scenarios>
    ...
  )

  expected_output <- mutate(input, outputvar = c(<Add Expected Outputs>))

  expect_dfs_equal(<function name>(input), expected_output)
  
})

For example, if you are testing a function called my_new_func that is contained in script all_funcs.R then from console use:

usethis::use_test("all_funcs")

Open the newly created file test-all_funcs.R and use the following format:

# my_new_func ----
## Test 1: <Explanation of the test> ----
test_that("my_new_func Test 1: <Explanation of the test>", {
  
  input <- dplyr::tribble(
    ~inputvar1, ~inputvar2, ...
    <Add Test Data Scenarios>
    ...
  )

  expected_output <- mutate(input, outputvar = c(<Add Expected Outputs>))

  expect_dfs_equal(<function name>(input), expected_output)
})

Note: When comparing datasets in admiral we use function expect_dfs_equal().

The input and expected output for the unit tests must follow the following rules:

Input and output should be as simple as possible.
Values should be hard-coded whenever possible.
If values need to be derived, only unit tested functions can be used.

In contrast to the Programming Strategy documentation for function examples, test files should not include library(pkg_name) calls. If a dataset needs to be created for testing purposes, it should be done so using the function tribble() from the tibble package with the following command dplyr::tribble(<data here>). Furthermore, if other functions need to be called, it should also be done using pkg_name::fun() notation. Make sure to align columns as well. This ensures quick code readability.

Ensure you give a meaningful explanation of the test in the testthat call, as these will be compiled in the package validation report. Having the name of the function and test ID included in title will also help with traceability.

The comments ending with ---- create entries in the TOC in RStudio.

Addin `pharmaverse4devs::format_test_that_file()`

To ease the burden on developers for writing and adding tests we have developed an Addin for formatting test_that test files according to admiral programming standards. The Addin will add and update comments as well as number or re-numbers the tests. To access the Addin, be sure to install {pharmaverse4devs} from Github.

To install the latest development version of the package directly from GitHub use the following code:

    if (!requireNamespace("remotes", quietly = TRUE)) {
      install.packages("remotes")
    }

    remotes::install_github("pharmaverse/pharmaverse4devs")

Then use the Addin button and select the “Format test_that test file” as seen in the image. Be sure to have the test-file open and selected when calling the Addin.

The Addin will perform the following:

Updates or adds the number of the tests in the comments and in the test_that() call
Updates the comments based on the description provided in the test_that() call
Updates the function name in the test_that() call. The function name is extracted from the last # <function name> ---- comment before the test_that() call. If a test file tests more than one function, such comments should be added before the first test of each function. If a test files tests a single function only, the comments can be omitted. In this case the addin determines the function name from the file name by stripping of the “test-” prefix and the “.R” suffix.

When writing new unit tests, just provide a description in the test_that() call and if necessary the function name in a # <function name> ---- comment:

# derive_vars_merged ----
test_that( "works if it merges all variables", {
  actual <- derive_vars_merged(advs,
    dataset_add = adsl,
    by_vars = exprs(STUDYID, USUBJID)
  )

# convert_dtm_to_dtc ----
test_that("works if dtm is in correct format", {
  expect_equal(
    convert_dtm_to_dtc(as.POSIXct("2022-04-05 15:34:07 UTC")),
    "2022-04-05T15:34:07"
  )
})

test_that("Error is thrown if dtm is not in correct format", {
  expect_error(
    convert_dtm_to_dtc("2022-04-05T15:26:14"),
    "lubridate::is.instant(dtm) is not TRUE",
    fixed = TRUE
  )
})

Call the addin and get:

# derive_vars_merged ----
## Test 1: derive_vars_merged ----
test_that( "derive_vars_merged Test 1: it merges all variables", {
  actual <- derive_vars_merged(advs,
    dataset_add = adsl,
    by_vars = exprs(STUDYID, USUBJID)
  )

# convert_dtm_to_dtc ----
## Test 2: works if dtm is in correct format ----
test_that("convert_dtm_to_dtc Test 2: works if dtm is in correct format", {
  expect_equal(
    convert_dtm_to_dtc(as.POSIXct("2022-04-05 15:34:07 UTC")),
    "2022-04-05T15:34:07"
  )
})

## Test 3: Error is thrown if dtm is not in correct format ----
test_that("convert_dtm_to_dtc Test 3: Error is thrown if dtm is not in correct format", {
  expect_error(
    convert_dtm_to_dtc("2022-04-05T15:26:14"),
    "lubridate::is.instant(dtm) is not TRUE",
    fixed = TRUE
  )
})

Once you have tested your unit test program, you can run all unit tests from the console, as follows.

devtools::test()

For running just the tests of the current file call

devtools::test_file()

Automation of Unit Tests

When a user actions a pull request in the admiral GitHub repo, the unit tests are automatically run and pull request will be denied if any unit tests fail.