STF Testing Frameworks

The test.assessr package checks package structure and test drivers to detect and classify testing configurations.

If it detects a standard testing framework (STF) (i.e. testthat or testit), it runs a standard testing workflow involving a number of steps:

Step 1

• run callr::r_safe which calls covr::package_coverage.

• The r_safe function has the parameter error = "error".

• This means if package_coverage encounters an error, then r_safe returns an error to the main testing process.

• If Step 1 is successful, then a unit test data coverage object like this is produced:

Step 1 unit test data coverage object

R object	field	element 1	element 2	value
test_package	pkg_name			“test.package.0001”
	pkg_ver			“0.1.0”
	date_time			“2026-03-09 14:27:36.673915”
	executor			“xxxxxxx”
	sysname			“Linux”
	version			“#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025”
	release			“6.1.158-180.294.amzn2023.x86_64”
	machine			“x86_64”
	r_version			“4.4.1”
	test_framework_type			standard testing framework
	total_cov			1
	res_cov	name		“test.package.0001”
		coverage	filecoverage	R/myscript.R 100
			totalcoverage	100
	errors			NA
	notes			NA

Step 2

• run setup_covr_env which

  • loads helper files from tests/testthat,
  • loads package namespace objects into cov_env,
  • resolves the project root,
  • loads package datasets into cov_env,
  • loads Base R datasets and exports into cov_env,
  • optionally loads nycflights13 datasets if available,
  • loads dependencies into cov_env, and
  • unloads the package namespace afterward to reduce conflicts.

• run create fail reporter to find failing tests

• run create test reporter to create a reporter for all tests

• map exported functions to test files

• create a filter for test status i.e. tests that have "FAILED", "ERROR", "SKIPPED"

• identify tests to be skipped based on status

• comment out tests to be skipped

• run covr::environment_coverage on remaining test files.

• If Step 2 is successful, then a unit test data coverage object like this is produced:

Step 2 unit test data coverage object

R object	field	element 1	element 2	value
test_package	pkg_name			“test.package.0003”
	pkg_ver			“0.1.0”
	date_time			“2026-04-09 10:17:27.71564”
	executor			“xxxxxxx”
	sysname			“Linux”
	version			“#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025”
	release			“6.1.158-180.294.amzn2023.x86_64”
	machine			“x86_64”
	r_version			“4.4.1”
	test_framework_type			standard testing framework
	total_cov			0
	res_cov	name		“test.package.0003”
		coverage	filecoverage	num [1(1d)] 0
				attr(*, “dimnames”)=List of 1 chr “myscript.R”
			totalcoverage	0
	errors			NA
	notes			NA
long_summary	file			“test-myscript.R#L2_L2”
	code_script			“myscript”
	context			“this works”
	status			“FAILED”
	n			1
	time			0.291
test_skip	file			“test-myscript.R”
	code_script			“myscript”
	context			“this works”
	status			“FAILED”
	n			1
	time			0.291
	expectation:			“expect_equal”
	line1			2
	line2			2

This data object differs from Step 1 data object in that it gives information about which tests/test blocks were skipped and the status tells the user why they were skipped.

Step 3

• run withr::with_dir which calls covr::package_coverage.

• covr::package_coverage has the parameters type = "tests", quiet = TRUE

• If Step 3 is successful, then a unit test data coverage object like this is produced:

Step 3 unit test data coverage object

R object	field	element 1	element 2	value
test_package	pkg_name			“dplyr”
	pkg_ver			“1.1.4”
	date_time			“2026-04-09 10:17:27.71564”
	executor			“xxxxxxx”
	sysname			“Linux”
	version			“#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025”
	release			“6.1.158-180.294.amzn2023.x86_64”
	machine			“x86_64”
	r_version			“4.4.1”
	test_framework_type			standard testing framework
	total_cov			0
	res_cov	name		“dplyr”
		coverage	filecoverage	num [1:109(1d)] 98.3 76.7 97.5 100 100 …
				attr(*, “dimnames”)=List of 1 [1:109] “across.R” “all-equal.R” “arrange.R” “bind-cols.R” …
			totalcoverage	0.907
	errors			NA
	notes			NA
long_summary	file			[1:3432] “test-across.R#L7_L7” “test-across.R#L14_L14” “test-across.R#L20_L20” “test-across.R#L24_L24” …
	code_script			[1:3432] “across” “across” “across” “across” …
	context			[1:3432] “across() works on one column data.frame” “across() does not select grouping variables” “across() corre”
	status			[1:3432] “PASS” “PASS” “PASS” “PASS” …
	n			[1:3432] 1 1 1 1 1 1 1 1 1 1 …
	time			[1:3432] 0.041 0.014 0.077 0.077 0.077 …
test_skip	file			[1:448] “test-across.R” “test-across.R” “test-across.R” “test-across.R” …
	code_script			[1:448] “across” “across” “across” “across” …
	context			“across() gives meaningful messages” “across() gives meaningful messages” “across() gives meaningful mes”
	status			[1:448] “FAILED” “FAILED” “FAILED” “FAILED” …
	n			[1:448] 1 1 1 1 1 1 1 1 1 1 …
	time			[1:448] 2.03 2.03 2.03 2.03 2.03 …
	expectation			[1:448] “expect_snapshot” “expect_error” “expect_error” “expect_error” …
	line1			[1:448] 209 212 216 222 226 230 234 238 243 244 …
	line2			[1:448] 280 215 219 225 229 233 237 241 243 244 …

Both Step 2 unit test data coverage object and Step 3 unit test data coverage object give the user details about the status of the tests FAILED, PASSED and SKIPPED.

They also contain the test block start line (i.e. line1 and line2) and the expectation type in the expectation column (e.g. expect_snapshot expect_error).