Leveraging pytest-xdist To Have 1000s of Tests

A few years ago I was contracting for Telia, where our microservice would orchestrate calls to a bunch of others:

• tie a SIM to an IMEI
• set up some base connection and 3G/4G/5G
• apply rate limits
• set up extras like smartwatch pairing

It consumed JSON payloads from RabbitMQ (fast) and if we put a bug into production and not notice it for an hour, there’d be hell to pay in clean-up costs. Getting away with doing the ‘anti-action’ for failed payloads would have been considered a lucky break. Determining which 13 of the 27 API calls were ignored/partially completed/successful — not so lucky.

To avoid this as much as possible, we made integration tests for all kinds of likely & unlikely payloads that ran on the exact same version of Postgres we have live. During my tenure there, at one point we added test # 900 and if not for pytest-xdist, the CI pipeline would have finished once pre-lunch and once post, and then we’d go home : )

I’ve created a dead simple Django app here to prove my point.

def do_stuff(request, stuff_id: int):
    thing = get_object_or_404(Stuff, pk=stuff_id)
    thing.save()

    return JsonResponse({
        'id': thing.id,
        'name': thing.name,
        'updatedAt': thing.updated_at,
    })

I’d recommend anyone maintaining a pytest suite out there to install pytest-random-order as early on as possible. Taking a…