CI/CD in the {pharmaverse}

R/Pharma

November 10th, 2022

Ben Straub (GSK) & Craig Gower-Page (Roche)

What is CI/CD?

• Continuous Integration (CI): Frequent merging of several small changes into a main branch

• Continuous Delivery (CD): Repeatable deployment process when deciding to deploy

CI/CD bridges the gaps between development and operation activities and teams by enforcing automation in building, testing and deployment of applications. CI/CD services compile the incremental code changes made by developers, then link and package them into software deliverables

Does it help?

…Yes! Yes, it does!!

How does CI/CD help R packages?

• Catch issues (bugs) early on
• User base on multiple OSes and multiple R versions
• Faster turnaround on Code Review
• Multiple Contributors on your R Package
• Enforce style conventions and preferences
• Measure test coverage for new code
• Keep docs up-to-date
• And we can just keep going!

We just did all these in the R/Pharma Workshop: Intro to CI/CD for R Packages

Case Study - admiral

About admiral

• Provide an open source, modularized toolbox that enables the pharmaceutical programming community to develop ADaM datasets in R.
  
• ADaM is one of the required standards for data submission to FDA (U.S.) and PMDA (Japan) for clinical trials
• Links
  • CDISC
  • https://github.com/pharmaverse/admiral
• Issue 1: Checking ADaM Template code
• Issue 2: Common CI/CD workflows for the admiral family of packages

Issue 1 - How to Check our Template Code

• Create a reference files to build common ADaM datasets that shows users how to implement our functions
• Way less text than a Vignette - Code is ready to go and build a dataset
• Where we store this code is not checked by R-CMD
• How to ensure code stays up to date with deprecated functions or unforeseen bugs get in from functions working together?
• CI/CD for the win!

Solution 1 - CI/CD for Templates

• Dedicated CI/CD workflow that executes the Template code
• Once a Code Review is completed the Check Template Workflow is executed
• If any errors or warnings are detected the CI/CD check fails and the contributor must fix the error or warning.

.github/workflows/check-templates.yml

Issue 2 - admiral upstream and downstream dependencies

• As you can imagine there can be a lot of different types of ADaMs!
• Extension packages focus on specific disease areas like oncology
• The admiral family has a package for developers, template R package repo and dummy data
• Eek!! How to keep this all in line!

Solution 2 - Common CI/CD workflows for admiral upstream and downstream dependencies

• Using admiralci, we have a common set of CI/CD workflows
• Developers moving between packages are familiar with these workflows
• Common documentation between packages for CI/CD workflows - easy to maintain and provide to new contributors

Case Study - NEST

About NEST

• A collection of R packages for creating TLGs/TFLs and exploratory clinical trials data visualization
  
• tern for creating TLGs
  
• teal for creating exploratory web applications for analyzing clinical trial data
  
• Links
  • rtables
  • NEST GitHub Organization

Use Case 1 - Testing Packages as a Cohort

• An in-development package must be tested against the latest versions of upstream dependencies
• Monorepo emulation via a git branch naming strategy is achieved by using
  • the staged.dependencies R package
  • and the staged.dependencies GitHub Action
• Testing as a cohort can be done at any stage (eg. development, pre-release, release)

Use Case 2 - Shiny App Testing & Deployment

• Analysts create Shiny web apps via the teal framework for analyzing data
• Apps are tested via a CI pipeline that uses the shinytest2 R package
• Apps deployed to an RSConnect Server instance via a CD pipeline
  • With the help of the rsconnect and connectapi R packages

Use Case 3 - Validating R Packages

• R packages are validated by an internal validation team that uses CI/CD pipelines to automatically
  • accept new package submissions via a form
  • running tests against the new package to ensure package integrity
  • enforcing criteria to ensure that the package meets regulatory requirements
• Also validated externally via an open source project called thevalidatoR

Case Study - rbmi

About rbmi

• Reference Based Multiple Imputation
  
• Implements imputation for longitudinal data in accordance with the ICH E9(R1) Addendum on Estimands
  
• Acknowledgements to Alessandro Noci, Marcel Wolbers & Daniel Sabanes Bove
  
• Links
  • https://arxiv.org/abs/2109.11162
  • https://github.com/insightsengineering/rbmi

Key component of the package is that many of the methods rely on resampling techniques to generate confidence intervals. This requires fitting potentially 1000’s of MMRM models which means run times tend to be in the order of minutes to hours for a full run of the package depending on the size of the data

Package Development Problems

• Installation takes >3 minutes to compile STAN code
• Full test suite takes >50 minutes to run
• Vignettes took >5 minutes to run
• Need to release on in-house servers running legacy versions of R
• Many PR’s forgot to run devtools::document()
• Many PR’s forgot to rebuild pkgdown site

2 key issues that we are about to cover: - Need to get run times down to <10 minutes - Need to ensure package works both on internal server and CRAN

Solution 1 - Reducing the Test Suite Runtime

tests/testthat/test-bigtest.R

test_that("Some long running section", {
  skip_if_not(Sys.getenv("R_TEST_FULL") == "TRUE")

  # <rest of the test code>
})

.github/workflows/on_biweekly.yaml

on:
  schedule:
    - cron: '0 4 1,15 * *'

jobs:
  rcmdcheck-main:
    env:
      R_TEST_FULL: 'TRUE' 
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Check
        uses: ./.github/actions/rcmdcheck

• Use of skip-if-not + environment variable to conditionally control which tests get executed
• Use of CICD to ensure that the skipped tests do still get run on a period basis

Solution 2 - Reducing the Vignette Runtime

.
├── DESCRIPTION
├── R/
├── tests/
└── vignettes/
    ├── quickstart.Rmd
    ├── quickstart.html
    └── quickstart.html.asis

DESCRIPTION

Suggests: R.rsp
VignetteBuilder: R.rsp

vignettes/quickstart.html.asis

%\VignetteIndexEntry{rbmi: Quickstart}
%\VignetteEngine{R.rsp::asis}

• Use of R.rsp package to use “asis” vignette builder
• This is where we make the vignettes locally in advance and directly add them to the repo
• When R “builds” the vignette it simply just uses the pre-built file asis without re-running it
• This opens up a problem though of how do we know if the vignette is up-to-date or even runs properly anymore?

Solution 2 - Reducing the Vignette Runtime

.github/workflow/on_pr.yaml

  vignettes:
    runs-on: ubuntu-latest
    container:
      image: "ghcr.io/insightsengineering/rbmi:latest"
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Build Vignettes
        run: |
          Rscript ./vignettes/build.R

• We thus have a dedicated CICD job specifically to re-build the vignette.
• This way Vignette execution is doing in parallel to the running of the test suite
• Trade off here though is that it means our vignettes are never executed on CRANs servers
• This trade-off can be partially mitigated by setting up our own CICD process to use a variety of different OS’s / machine builds

Solution 3 - Testing Against In-House Servers

• Use of docker files to emulate our in-house servers
• Had double benefit of drastically reducing package install times on standard push / pr builds as all required dependencies are already installed and ready to go

Solution 3 - Testing Against In-House Servers

.github/workflows/on_pr_main.yaml

jobs:
  rcmdcheck:
    strategy:
      matrix:
       config:
        - { image: "ghcr.io/insightsengineering/rbmi:r404"}
        - { image: "ghcr.io/insightsengineering/rbmi:r410"}
        - { image: "ghcr.io/insightsengineering/rbmi:latest"}
    runs-on: ubuntu-latest
    container:
      image: ${{ matrix.config.image }}
    steps:
      - name: Checkout
        uses: actions/checkout@v3
        
      - name: Build src
        uses: ./.github/actions/build-src
        
      - name: Check
        uses: ./.github/actions/rcmdcheck

• Final bit on this is just to highlight that we can use githubs matrix feature to ensure that our job is run against all the different combination of images and r versions that we want to ensure we can support

Additional Materials

• Further Reading

  • GitHub Actions
  • GitLab CI

• Advanced Examples

  • r-lib/actions
  • {admiralci}
  • Docker

• Presentation built with Quarto

• R/Pharma 2022 CI/CD Workshop

• This Presentation