Integration Tests

Introduction

The tests/integration directory contains an automated test harness for Slurm-Mail. It makes use of the docker images created by the Docker files in the docker-slurm sub directory.

These tests can be run locally via Docker (see below).

The test_and_release GitHub workflow executes the integration tests when:

new commits are pushed
pull requests are created
merges to the main branch are performed
new releases are created with a v tag

Adding new Slurm version support

When a new version of Slurm is released the Docker files in the docker-slurm sub directory need to be updated to the new version and the images created and pushed to the ghcr.io/neilmunday/slurm-mail image repository. This is partially automated via the slurm_version_check and build_docker_slurm workflows.

Each day slurm_version_check checks for new versions of Slurm. If a new version is found the workflow updates all of the files in the docker-slurm directory with the new version string and creates a pull request.

When the pull request is merged the build_docker_slurm workflow automatically creates the new Slurm-Mail docker images so they are ready to use.

Finally, the ../.github/workflows/testing.yml workflow must be updated to include the new Slurm version.

Adding tests

Slurm tests should be added to tests.yml.

A test is defined like so:

  testX:
    # commands to run in batch script
    commands: |
      srun hostname  
    # test description
    description: one node test
    # is the job expected to fail?
    job_fail: false
    # sbatch options
    options:
      mail-type: ALL
      time: "01:00"
    # optionally specify any commands to run after job submission
    post_submit: |
      echo "I was run"
    # are errors expected in the slurm-send-mail log?
    send_errors: false
    # are errors expected in the slurm-spool-mail log?
    spool_errors: false
    # how many spool files are expected?
    spool_file_total: 2

Running tests

The run.sh script is used to run the test suite against a particular version of Slurm.

The script performs the following tasks:

Builds the Slurm-Mail RPM for the target OS
Creates a temporary docker compose configuration to run a two node cluster
Builds the head node image to include the Slurm-Mail RPM
Starts the containers
Initiates the test suite from the head node
Stops the containers and removes the images

Within the image that is created for the head node, the mail-server.py script is used to create a simple mail server to process e-mails generated by Slurm-Mail.

The run-tests.py together with tests.yml are copied inside the head node image.

Once the head node container is up and running the run.sh script executes run-tests.py which submits the jobs defined in tests.yml. As each job runs the run-tests.py script checks if each job was processed as expected by Slurm-Mail.

A summary of how many tests passed and failed is printed at the end of the tests.

Excute all tests against Slurm 24.11.4 on Rocky Linux 9:

./run.sh -s 24.11.4 -o el9

Execute a particular test:

./run.sh -s 24.11.4  -o el9 -t test2

Enable verbose logging:

./run.sh -s 24.11.4 -o el9 -v

Tip: As run.sh will build a Slurm-Mail RPM you can use the -r flag on subsequent invocations to skip building the RPM and thus save time.

Testing e-mails

If you want to see how the e-mails will look in an e-mail client you can use the provided demo.sh script to start-up Slurm-Mail containers together with a MailHog container.

Usage:

./demo.sh -s SLURM_VERSION -o OS_VERSION [-p]

If the Slurm-Mail RPM has already been built and exists in this directory you can use -p option to skip building the RPM.

./demo.sh -s 24.05.0 -o el9

Once the containers are up and running you can access the MailHog web GUI at http://localhost:8025

To submit jobs you can run the following command to launch an interactive bash shell:

docker exec -it slurm-mail-el9-24.05.0-head /usr/bin/bash -i

Reverting to /usr/bin/mail

If you want to see how e-mails from Slurm look like instead (i.e. not using Slurm-Mail), perform the following steps:

Disable the cron job at /etc/cron.d/slurm-mail
Install sendmail:

dnf install -y sendmail

Add to the end of /etc/mail.rc:

set smtp=mailhog:1025

Set MailProg in /etc/slurm/slurm/conf to /usr/bin/mail.
Restart slurmctld:

supervisorctl stop slurmctld
supervisorctl start slurmctld

GitHub Actions Set-Up

In order to execute Slurm-Mail’s GitHub actions with your repository you will need to create a personal access token and a classic token.

Fine Grained Personal Access Token

A personal accesst token (PAT) is required to allow the slurm_version_check workflow to create pull requests when a new Slurm version is released. This workflow runs daily.

Replace $GH_USER in the instructions below with your GitHub username.

Create a new PAT with the following values:

Setting	Value
Name	slurm-mail
Description	Token to allow PRs from workflows to trigger other workflows.
Repository Access	`$GH_USER`/slurm-mail
User Permissions	None
Repository Permissions	Read access to actions, code, and metadata
Repository Permissions	Read and Write access to pull requests
Expiration	Select a time of your choosing, e.g. 90 days

Copy the generated secret and go to https://github.com/$GH_USER/slurm-mail/settings/secrets/actions and create a new repository secret called PRPAT and paste your generated secret. Now save the secret.

Tokens (classic)

A classic token is required to allow the purge_docker_slurm_images to automatically delete old Slurm Docker images from the GHCR repository.

Create a new token called Image Repository Access with the following sopes enabled:

repo: all
write:packages
read:packages
delete:packages

Copy the generated secret and go to https://github.com/$GH_USER/slurm-mail/settings/secrets/actions and create a new repository secret called PACKAGE_REPO and paste your generated secret. Now save the secret.

Image repository

The test_and_release relies upon being able to use the Slurm images in the Slurm-Mail image repository.

These images are public and can therefore be consumed by anyone. If you want to build your own images then you will need to build all of the images and push them to your GitHub image repository.

Note: If you decide to make your image repositories private you will need to give access to your Slurm-Mail repository in order for the test_and_release to work.

To build the images proceed as follows from within the root of your local copy of the Slurm-Mail repository:

cd slurm-mail/tests/integration/docker-slurm

Now edit build-all.sh and change ghcr.io/neilmunday/slurm-mail to include your GitHub username instead of mine.

Then run:

for v in `jq -r .[] https://github.com/neilmunday/slurm-mail/blob/main/supported_slurm_versions.json`; do ./build-all.sh $v; done

This will build Slurm Docker images for all versions of Slurm and all operating systems supported by Slurm-Mail - it will take some time!

Now push the images to your GitHub image repository:

docker images | grep slurm | awk '{ print $1":"$2 }' | while read i; do docker push $i; done

Update integration tests Docker files

Under the tests/integration directory you will need to change the FROM line in each Dockerfile to use your image repository.

You will also need to edit tests/integration/run.sh to use your image repository.

You are now ready to run Slurm-Mail’s GitHub workflows with your repository.