As discussed here https://github.com/pytest-dev/pytest-runner/issues/7 and sort of https://github.com/pytest-dev/pytest/issues/567