pytest does not reset the state of logging if we install a file handler, which will cause FileNotFoundError after the temporary folder is removed. Semi-related: https://github.com/pytest-dev/pytest/issues/5502