Testing Package Compatibility with Matrix Strategies in GitHub Actions

I recently published a Django package, django-anchor, and wanted to ensure it was compatible with environments people might have in their own setups. The package mainly depends on three other components: Python itself, Pillow (the PIL image manipulation library) and Django itself, of course!

The package isn't very picky about what it needs from the dependencies (we're not using the latest Django features and the API surface we target in PIL is very limited) so I think it's reasonable to support anything that was released in the past 2 years or so. Moreover, Django 4.2. is a long-term support release (LTS), which means it'll receive security updates for the next three years. Many teams also choose to upgrade only to LTS releases which is also a very reasonable strategy.

With that in mind I tested the package manually by installing older versions of dependencies locally but quickly found out that GitHub Actions supports something called Matrix Strategies which allow you to generate multiple jobs with just one YAML definition by specifying which versions should be tested.

A simple example would be to test the package against the last 3 python versions which can be done like this:

jobs:
  python_compatibility_test:
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version: ["3.10", "3.11", "3.12"]

    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v3
        with:
          python-version: ${{ matrix.python-version }}
      - name: Run tests
        run: |
          python runtests.py

The above definition will create three jobs when it's run, one with each Python version which gets set up in the second step.

⚠️ Make sure to quote your versions since YAML will interpret 3.10 as a float and thus be passed as 3.1 to the runner.

You can of course combine several dimensions to test all 6 possible combinations of a Python + Django version:

jobs:
  python_compatibility_test:
    runs-on: ubuntu-latest
    strategy:
      max-parallel: 4
      matrix:
        python-version: ["3.10", "3.11", "3.12"]
        django-version: ["4.2", "5.0"]

    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v3
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install Django ${{ matrix.django-version }}
        run: |
          pip install Django==${{ matrix.django-version }}
      - name: Run tests
        run: |
          python runtests.py

In practice, I chose not to test all Django versions against all Python versions since those are already well tested by the Django community (Django itself maintains a list of the Python versions supported by each release). Also, for the django-anchor package, I don't think it's necessary to test the combinations of Django and Pillow versions, since those two packages don't interact at all. This leaves us with three separate compatibility tests which still make use of a matrix strategy but the matrix is really a 1-dimensional array. The Actions job definition for all compatibility tests can be found in the package repository.