v11.4.0 release notes

Fixes #692

.pre-commit-config.yaml

@@ -19,5 +19,5 @@ repos:
     rev: 19.10b0
     hooks:
       - id: black
-        language_version: python3.8
+        language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py

README.md

@@ -92,6 +92,9 @@ apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified lan
 
 # Arch Linux users
 pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
+
+# brew macOS users
+brew install tesseract-lang
 ```
 
 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.

azure-pipelines.yml

@@ -22,6 +22,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -59,6 +61,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -156,9 +160,8 @@ stages:
           #     versionSpec: "$(python.version)"
           - bash: |
               brew update
-              brew unlink python@2
               brew upgrade python
-              echo "Using Python `python3 --version`"
+              echo "Using `python3 --version`"
             displayName: "Update brew and Python"
           - bash: |
               brew install \

docs/advanced.rst

@@ -125,8 +125,7 @@ include:
 .. envvar:: OMP_THREAD_LIMIT
 
    Controls the number of threads Tesseract will use. OCRmyPDF will
-   manage this environment if it is not already set. (Currently, it will
+   manage this environment variable if it is not already set.
-   set it to 1 because this gives the best results in testing.)
 
 For example, if you have a development build of Tesseract don't wish to
 use the system installation, you can launch OCRmyPDF as follows:

docs/api.rst

@@ -20,7 +20,8 @@ and largely have the same functions.
 
     import ocrmypdf
 
-    ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
+    if __name__ == '__main__':  # To ensure correct behavior on Windows and macOS
+        ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
 With a few exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
@@ -35,21 +36,21 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
 execution. To do this, it will:
 
 - create a monitoring thread
-- create worker processes (forking itself)
+- create worker processes (on Linux, forking itself; on Windows and macOS, by
-- manage the signal flags of worker processes
+  spawning)
+- manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
 The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
-privileged to perform these actions. If it is not, ``ocrmypdf()`` will
+privileged to perform these actions.
-fail.
 
 There is no currently no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
-OCRmyPDF does not.
+OCRmyPDF fails for any reason.
 
 Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
@@ -57,11 +58,10 @@ mapped file fails. OCRmyPDF may use memory mapping.
 
 .. warning::
 
-    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
-    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
-    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    not take at least one of these steps, process semantics will prevent
-    of these steps, Windows process semantics will prevent OCRmyPDF from working
+    OCRmyPDF from working correctly.
-    correctly.
 
 Logging
 -------

docs/batch.rst

@@ -127,7 +127,7 @@ Users may need to customize the script to meet their requirements.
     "OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
     "OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
     "OCR_DESKEW", "Apply deskew to crooked input PDFs"
-    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={"rotate_pages": true}'``.
+    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={""rotate_pages"": true}'``."
     "OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
     "OCR_LOGLEVEL", "Level of log messages to report"
 

docs/docker.rst

@@ -1,3 +1,5 @@
+.. _docker:
+
 =====================
 OCRmyPDF Docker image
 =====================

docs/index.rst

@@ -1,7 +1,7 @@
 OCRmyPDF documentation
 ======================
 
-OCRmyPDF adds an optical charcter recognition (OCR) text layer to scanned PDF
+OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
 files, allowing them to be searched.
 
 PDF is the best format for storing and exchanging scanned documents.

docs/installation.rst

@@ -495,10 +495,6 @@ Installing on Windows
 Native Windows
 --------------
 
-.. note::
-
-    It is easier to install OCRmyPDF on Windows Subsystem for Linux.
-
 .. note::
 
     Administrator privileges will be required for some of these steps.
@@ -509,30 +505,33 @@ You must install the following for Windows:
 * Tesseract 4.0 or later
 * Ghostscript 9.50 or later
 
-You can install these with the Chocolatey package manager:
+Using the `Chocolatey <https://chocolatey.org/>`_ package manager, install the
+following when running in an Administrator command prompt:
 
 * ``choco install python3``
 * ``choco install --pre tesseract``
 * ``choco install ghostscript``
+* ``choco install pngquant`` (optional)
 
-Also consider adding:
+The commands above will install Python 3.x (latest version), Tesseract, Ghostscript
+and pngquant. Chocolatey may also need to install the Windows Visual C++ Runtime
+DLLs or other Windows patches, and may require a reboot.
 
-* ``choco install pngquant``
+You may then use ``pip`` to install ocrmypdf. (This can performed by a user or
+Administrator.):
 
-Windows 10 64-bit and 64-bit versions of applications are recommended. Earlier
+* ``pip install ocrmypdf
-versions of Windows and 32-bit versions of these programs are not tested, and not
-supported at this time.
 
-OCRmyPDF will check for Tesseract-OCR and Ghostscript in your Program Files folder.
+Chocolatey automatically selects appropriate versions of these applications. If you
-If they are in some other location, you may need to modify the ``PATH``
+are installing them manually, please install 64-bit versions of all applications for
-environment variable so Tesseract, Ghostscript, and other any optional executables can
+64-bit Windows, or 32-bit versions of all applications for 32-bit Windows. Mixing
-be found. You can enter it in the command line or
+the "bitness" of these programs will lead to errors.
-`follow these directions <https://www.computerhope.com/issues/ch000549.htm#dospath>`_
-to make the change persistent and system-wide.
 
-You may then use pip to install ocrmypdf:
+OCRmyPDF will check the Windows Registry and standard locations in your Program Files
-
+for third party software it needs (specifically, Tesseract and Ghostscript). To
-* ``pip install ocrmypdf``
+override the versions OCRmyPDF selects, you can modify the ``PATH`` environment
+variable. `Follow these directions <https://www.computerhope.com/issues/ch000549.htm#dospath>`_
+to change the PATH.
 
 Windows Subsystem for Linux
 ---------------------------
@@ -604,7 +603,7 @@ However, the OCR-to-text-layer functionality is available.
 Docker
 ------
 
-You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
+You can also :ref:`Install the Docker <docker>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
 Installing on FreeBSD
@@ -630,7 +629,7 @@ Installing the Docker image
 For some users, installing the Docker image will be easier than
 installing all of OCRmyPDF's dependencies.
 
-See `OCRmyPDF Docker Image <docker>`__ for more information.
+See :ref:`docker` for more information.
 
 Installing with Python pip
 ==========================

docs/release_notes.rst

@@ -12,6 +12,89 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+v11.4.0
+=======
+
+-  When looking for Tesseract and Ghostscript, we now check the Windows Registry to
+   see if their installers registered the location of their executables. This should
+   help Windows users who have installed these programs to non-standard
+   locations.
+-  We now report on the progress of PDF/A conversion, since this operation is
+   sometimes slow.
+-  Improved command line completions.
+-  The prefix of the temporary folder OCRmyPDF creates has been changed from
+   ``com.github.ocrmypdf`` to ``ocrmypdf.io``. Scripts that chose to depend on this
+   prefix may need to be adjusted. (This has always been an implementation detail so is
+   not considered part of the semantic versioning "contract".)
+-  Fixed issue #692, where a particular file with malformed fonts would flood an
+   internal message cue by generating so many debug messages.
+-  Fixed an exception on processing hOCR files with no page record. Tesseract
+   is not known to generate such files.
+
+v11.3.4
+=======
+
+-  Fixed an error message 'called readLinearizationData for file that is not
+   linearized' that may occur when pikepdf 2.1.0 is used. (Upgrading to pikepdf
+   2.1.1 also fixes the issue.)
+-  File watcher now automatically includes ``.PDF`` in addition to ``.pdf`` to
+   better support case sensitive file systems.
+-  Some documentation and comment improvements.
+
+v11.3.3
+=======
+
+-  If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
+   conversion. (Possibly addresses #671.)
+
+v11.3.2
+=======
+
+-  Explicitly require pikepdf 2.0.0 or newer when running on Python 3.9. (There are
+   concerns about the stability of pybind11 2.5.x with Python 3.9, which is used in
+   pikepdf 1.x.)
+-  Fixed another issue related to page rotation.
+-  Fixed an issue where image marked as image masks were not properly considered
+   as optimization candidates.
+-  On some systems, unpaper seems to be unable to process the PNGs we offer it
+   as input. We now convert the input to PNM format, which unpaper always accepts.
+   Fixes #665 and #667.
+-  DPI sent to unpaper is now rounded to a more reasonable number of decimal digits.
+-  Debug and error messages from unpaper were being suppressed.
+-  Some documentation tweaks.
+
+v11.3.1
+=======
+
+-  Declare support for new versions: pdfminer.six 20201018 and pikepdf 2.x
+-  Fix warning related to ``--pdfa-image-compression`` that appears at the wrong
+   time.
+
+v11.3.0
+=======
+
+-  The "OCR" step is describing as "Image processing" in the output messages when
+   OCR is disabled, to better explain the application's behavior.
+-  Debug logs are now only created when run as a command line, and not when OCR
+   is performed for an API call. It is the calling application's responsibility
+   to set up logging.
+-  For PDFs with a low number of pages, we gathered information about the input PDF
+   in a thread rather than process (when there are more pages). When run as a
+   thread, we did not close the file handle to the working PDF, leaking one file
+   handle per call of ``ocrmypdf.ocr``.
+-  Fixed an issue where debug messages send by child worker processes did not match
+   the log settings of parent process, causing messages to be dropped. This affected
+   macOS and Windows only where the parent process is not forked.
+-  Fixed the hookspec of rasterize_pdf_page to remove default parameters that
+   were not handled in an expected way by pluggy.
+-  Fixed another issue with automatic page rotation (#658) due to the issue above.
+
+v11.2.1
+=======
+
+-  Fixed an issue where optimization of a 1-bit image with a color palette or
+   associated ICC that was optimized to JBIG2 could have its colors inverted.
+
 v11.2.0
 =======
 

misc/completion/ocrmypdf.fish

@@ -59,7 +59,8 @@ complete -c ocrmypdf -x -l output-type -a '(__fish_ocrmypdf_output_type)' -d "se
 
 function __fish_ocrmypdf_pdf_renderer
     echo -e "auto\t"(_ "auto select PDF renderer")
-    echo -e "hocr\t"(_ "use hocr renderer")
+    echo -e "hocr\t"(_ "use hOCR renderer")
+    echo -e "hocrdebug\t"(_ "uses hOCR renderer in debug mode, showing recognized text")
     echo -e "sandwich\t"(_ "use sandwich renderer")
 end
 complete -c ocrmypdf -x -l pdf-renderer -a '(__fish_ocrmypdf_pdf_renderer)' -d "select PDF renderer options"
@@ -135,4 +136,4 @@ complete -c ocrmypdf -r -l user-words -d "specify location of user words file"
 complete -c ocrmypdf -r -l user-patterns -d "specify location of user patterns file"
 complete -c ocrmypdf -x -l fast-web-view -d "if file size if above this amount in MB, linearize PDF"
 
-complete -c ocrmypdf -x -a "(__fish_complete_suffix .pdf)"
+complete -c ocrmypdf -x -a "(__fish_complete_suffix .pdf; __fish_complete_suffix .PDF; __fish_complete_suffix .jpg; __fish_complete_suffix .png)"

misc/watcher.py

@@ -45,7 +45,7 @@ OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = int(os.getenv('OCR_POLL_NEW_FILE_SECONDS', '1'))
 USE_POLLING = bool(os.getenv('OCR_USE_POLLING', ''))
 LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
-PATTERNS = ['*.pdf']
+PATTERNS = ['*.pdf', '*.PDF']
 
 log = logging.getLogger('ocrmypdf-watcher')
 

requirements/main.txt

@@ -1,12 +1,12 @@
 # requirements.txt can be used to replicate the developer's build environment
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
-cffi == 1.14.0
+cffi == 1.14.3
 coloredlogs == 14.0  # technically optional
-img2pdf == 0.3.6
+img2pdf == 0.4.0
-pdfminer.six == 20200517
+pdfminer.six == 20201018
-pikepdf == 1.16.1
+pikepdf == 2.0.0
 pluggy == 0.13.1
-Pillow == 7.1.2
+Pillow == 8.0.1
-reportlab == 3.5.42
+reportlab == 3.5.55
-tqdm == 4.46.1
+tqdm == 4.51.0

setup.py

@@ -63,7 +63,6 @@ setup(
     python_requires=' >= 3.6',
     setup_requires=[  # can be removed whenever we can drop pip 9 support
         'cffi >= 1.9.1',  # to build the leptonica module
-        'pytest-runner',  # to enable python setup.py test
         'setuptools_scm',  # so that version will work
         'setuptools_scm_git_archive',  # enable version from github tarballs
     ],
@@ -73,10 +72,11 @@ setup(
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'coloredlogs >= 14.0',  # strictly optional
         'img2pdf >= 0.3.0, < 0.5',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20191110, != 20200720, <= 20200726',
+        'pdfminer.six >= 20191110, != 20200720, <= 20201018',
-        'pikepdf >= 1.14.0, < 2',
+        "pikepdf >= 1.14.0, < 3 ; python_version < '3.9'",
+        "pikepdf >= 2.0.0 ; python_version >= '3.9'",
         'Pillow >= 7.0.0',
-        'pluggy >= 0.13.0',
+        'pluggy >= 0.13.0, < 1.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling
         'tqdm >= 4',
     ],

src/ocrmypdf/_concurrent.py

@@ -49,7 +49,7 @@ def process_sigbus(*args):
     raise InputFileError("A worker process lost access to an input file")
 
 
-def process_init(queue, user_init):
+def process_init(queue, user_init, loglevel):
     """Initialize a process pool worker"""
 
     # Ignore SIGINT (our parent process will kill us gracefully)
@@ -62,6 +62,7 @@ def process_init(queue, user_init):
     # Reconfigure the root logger for this process to send all messages to a queue
     h = logging.handlers.QueueHandler(queue)
     root = logging.getLogger()
+    root.setLevel(loglevel)
     root.handlers = []
     root.addHandler(h)
 
@@ -69,7 +70,7 @@ def process_init(queue, user_init):
         user_init()
 
 
-def thread_init(_queue, user_init):
+def thread_init(_queue, user_init, _loglevel):
     # As a thread, block SIGBUS so the main thread deals with it...
     if hasattr(signal, 'SIGBUS'):
         signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGBUS})
@@ -102,7 +103,7 @@ def exec_progress_pool(
         pool = pool_class(
             processes=max_workers,
             initializer=initializer,
-            initargs=(log_queue, task_initializer),
+            initargs=(log_queue, task_initializer, logging.getLogger("").level),
         )
         try:
             results = pool.imap_unordered(task, task_arguments)

src/ocrmypdf/_exec/ghostscript.py

@@ -21,28 +21,28 @@ from PIL import Image
 
 from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
 from ocrmypdf.helpers import Resolution
-from ocrmypdf.subprocess import get_version, run
+from ocrmypdf.subprocess import get_version, run, run_polling_stderr
 
 log = logging.getLogger(__name__)
 
+missing_gs_error = """
+---------------------------------------------------------------------
+This error normally occurs when ocrmypdf find can't Ghostscript.
+Please ensure Ghostscript is installed and its location is added to
+the system PATH environment variable.
+
+For details see:
+    https://ocrmypdf.readthedocs.io/en/latest/installation.html
+---------------------------------------------------------------------
+"""
+
 _gswin = None
 if os.name == 'nt':
     _gswin = which('gswin64c')
     if not _gswin:
         _gswin = which('gswin32c')
         if not _gswin:
-            raise MissingDependencyError(
+            raise MissingDependencyError(missing_gs_error)
-                """
-                ---------------------------------------------------------------------
-                This error normally occurs when ocrmypdf can't Ghostscript.  Please
-                ensure Ghostscript is installed and its location is added to the
-                system PATH environment variable.
-
-                For details see:
-                    https://ocrmypdf.readthedocs.io/en/latest/installation.html
-                ---------------------------------------------------------------------
-                """
-            )
     _gswin = Path(_gswin).stem
 
 GS = _gswin if _gswin else 'gs'
@@ -81,8 +81,8 @@ def rasterize_pdf(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int = 1,
-    page_dpi: Resolution = None,
+    page_dpi: Optional[Resolution] = None,
-    rotation: int = None,
+    rotation: Optional[int] = None,
     filter_vector: bool = False,
 ):
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units."""
@@ -122,8 +122,6 @@ def rasterize_pdf(
         stderr = p.stderr.decode(errors='replace')
         if _gs_error_reported(stderr):
             log.error(stderr)
-        elif stderr:
-            log.debug(stderr)
 
     with Image.open(BytesIO(p.stdout)) as im:
         if rotation is not None:
@@ -141,13 +139,43 @@ def rasterize_pdf(
         im.save(fspath(output_file), dpi=page_dpi)
 
 
+class GhostscriptFollower:
+    re_process = re.compile(r"Processing pages \d+ through (\d+).")
+    re_page = re.compile(r"Page (\d+)")
+
+    def __init__(self, progressbar_class):
+        self.count = 0
+        self.progressbar_class = progressbar_class
+        self.progressbar = None
+
+    def __call__(self, line):
+        if not self.progressbar_class:
+            return
+        if not self.progressbar:
+            m = self.re_process.match(line.strip())
+            if m:
+                self.count = int(m.group(1))
+                self.progressbar = self.progressbar_class(
+                    total=self.count, desc="PDF/A conversion", unit='page'
+                )
+                return
+        else:
+            m = self.re_page.match(line.strip())
+            if m:
+                self.progressbar.update()
+
+
 def generate_pdfa(
     pdf_pages,
     output_file: os.PathLike,
     compression: str,
     pdf_version: str = '1.5',
     pdfa_part: str = '2',
+    progressbar_class=None,
 ):
+    # Ghostscript's compression is all or nothing. We can either force all images
+    # to JPEG, force all to Flate/PNG, or let it decide how to encode the images.
+    # In most case it's best to let it decide.
     compression_args = []
     if compression == 'jpeg':
         compression_args = [
@@ -175,8 +203,9 @@ def generate_pdfa(
     strategy = 'RGB' if version() >= '9.19' else '/RGB'
 
     if version() == '9.23':
-        # 9.23: new feature JPEG passthrough is broken in some cases, best to
+        # 9.23: added JPEG passthrough as a new feature, but with a bug that
-        # disable it always
+        # incorrectly formats some images. Fixed as of 9.24. So we disable this
+        # feature for 9.23.
         # https://bugs.ghostscript.com/show_bug.cgi?id=699216
         compression_args.append('-dPassThroughJPEGImages=false')
 
@@ -186,7 +215,6 @@ def generate_pdfa(
     args_gs = (
         [
             GS,
-            "-dQUIET",
             "-dBATCH",
             "-dNOPAUSE",
             "-dSAFER",
@@ -206,16 +234,26 @@ def generate_pdfa(
         ]
     )
     args_gs.extend(fspath(s) for s in pdf_pages)  # Stringify Path objs
+
     try:
         with Path(output_file).open('wb') as output:
-            p = run(args_gs, stdout=output, stderr=PIPE, check=True)
+            p = run_polling_stderr(
+                args_gs,
+                stdout=output,
+                stderr=PIPE,
+                check=True,
+                text=True,
+                encoding='utf-8',
+                errors='replace',
+                callback=GhostscriptFollower(progressbar_class),
+            )
     except CalledProcessError as e:
         # Ghostscript does not change return code when it fails to create
         # PDF/A - check PDF/A status elsewhere
-        log.error(e.stderr.decode(errors='replace'))
+        log.error(e.stderr)
-        raise SubprocessOutputError('Ghostscript PDF/A rendering failed')
+        raise SubprocessOutputError('Ghostscript PDF/A rendering failed') from e
     else:
-        stderr = p.stderr.decode('utf-8', errors='replace')
+        stderr = p.stderr
         if _gs_error_reported(stderr):
             last_part = None
             repcount = 0

src/ocrmypdf/_exec/tesseract.py

@@ -100,7 +100,12 @@ def get_languages():
     args_tess = ['tesseract', '--list-langs']
     try:
         proc = run(
-            args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
+            args_tess,
+            text=True,
+            stdout=PIPE,
+            stderr=STDOUT,
+            logs_errors_to_stdout=True,
+            check=True,
         )
         output = proc.stdout
     except CalledProcessError as e:

src/ocrmypdf/_exec/unpaper.py

@@ -13,10 +13,11 @@
 import logging
 import os
 import shlex
+from decimal import Decimal
 from pathlib import Path
-from subprocess import PIPE, STDOUT, CalledProcessError
+from subprocess import PIPE, STDOUT
 from tempfile import TemporaryDirectory
-from typing import Tuple
+from typing import List, Optional, Tuple, Union
 
 from PIL import Image
 
@@ -24,10 +25,12 @@ from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
 from ocrmypdf.subprocess import get_version
 from ocrmypdf.subprocess import run as external_run
 
+DecFloat = Union[Decimal, float]
+
 log = logging.getLogger(__name__)
 
 
-def version():
+def version() -> str:
     return get_version('unpaper')
 
 
@@ -53,23 +56,25 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
         except KeyError:
             raise MissingDependencyError(
                 "Failed to convert image to a supported format."
-            ) from e
+            ) from None
 
-        if im_modified or input_file.suffix != '.png':
+        if im_modified or input_file.suffix != '.pnm':
-            input_png = tmpdir / 'input.png'
+            input_pnm = tmpdir / 'input.pnm'
-            im.save(input_png, format='PNG', compress_level=1)
+            im.save(input_pnm, format='PPM')
         else:
             # No changes, PNG input, just use the file we already have
-            input_png = input_file
+            input_pnm = input_file
         output_pnm = tmpdir / f'output{suffix}'
-    return input_png, output_pnm
+    return input_pnm, output_pnm
 
 
-def run(input_file, output_file, dpi, mode_args):
+def run(
-    args_unpaper = ['unpaper', '-v', '--dpi', str(dpi)] + mode_args
+    input_file: Path, output_file: Path, dpi: DecFloat, mode_args: List[str]
+) -> None:
+    args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
 
     with TemporaryDirectory() as tmpdir:
-        input_png, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
+        input_pnm, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
 
         # To prevent any shenanigans from accepting arbitrary parameters in
         # --unpaper-args, we:
@@ -78,41 +83,40 @@ def run(input_file, output_file, dpi, mode_args):
         # 3) append absolute paths for the input and output file
         # This should ensure that a user cannot clobber some other file with
         # their unpaper arguments (whether intentionally or otherwise)
-        args_unpaper.extend([os.fspath(input_png), os.fspath(output_pnm)])
+        args_unpaper.extend([os.fspath(input_pnm), os.fspath(output_pnm)])
+        external_run(
+            args_unpaper,
+            close_fds=True,
+            check=True,
+            stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+            stdout=PIPE,  # and cannot send file output to stdout
+            cwd=tmpdir,
+            logs_errors_to_stdout=True,
+        )
         try:
-            proc = external_run(
+            with Image.open(output_pnm) as imout:
-                args_unpaper,
+                imout.save(output_file, dpi=(dpi, dpi))
-                check=True,
+        except (FileNotFoundError, OSError):
-                close_fds=True,
+            raise SubprocessOutputError(
-                universal_newlines=True,
+                "unpaper: failed to produce the expected output file. "
-                stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+                + " Called with: "
-                cwd=tmpdir,  # and cannot send file output to stdout
+                + str(args_unpaper)
-                stdout=PIPE,
+            ) from None
-            )
-        except CalledProcessError as e:
-            log.debug(e.stderr)
-            raise e from e
-        else:
-            log.debug(proc.stderr)
-            try:
-                with Image.open(output_pnm) as imout:
-                    imout.save(output_file, dpi=(dpi, dpi))
-            except (FileNotFoundError, OSError):
-                raise SubprocessOutputError(
-                    "unpaper: failed to produce the expected output file. "
-                    + " Called with: "
-                    + str(args_unpaper)
-                ) from None
 
 
-def validate_custom_args(args: str):
+def validate_custom_args(args: str) -> List[str]:
     unpaper_args = shlex.split(args)
     if any(('/' in arg or arg == '.' or arg == '..') for arg in unpaper_args):
         raise ValueError('No filenames allowed in --unpaper-args')
     return unpaper_args
 
 
-def clean(input_file, output_file, dpi, unpaper_args=None):
+def clean(
+    input_file: Path,
+    output_file: Path,
+    dpi: DecFloat,
+    unpaper_args: Optional[List[str]] = None,
+):
     default_args = [
         '--layout',
         'none',

src/ocrmypdf/_graft.py

@@ -109,6 +109,7 @@ class OcrGrafter:
         if textpdf and not self.font:
             self.font, self.font_key = self._find_font(textpdf)
 
+        emplaced_page = False
         content_rotation = self.pdfinfo[pageno].rotation
         path_image = Path(image).resolve() if image else None
         if path_image is not None and path_image != self.path_base:
@@ -122,24 +123,21 @@ class OcrGrafter:
                 local_image_page = self.pdf_base.pages[-1]
                 self.pdf_base.pages[pageno].emplace(local_image_page)
                 del self.pdf_base.pages[-1]
-            # The pdf_image_page will always be created with any /Rotate applied
+            emplaced_page = True
-            # applied already
-            content_rotation = 0
 
-        if content_rotation != 0:
+        # Calculate if the text is misaligned compared to the content
-            # Text can be misaligned on a /Rotate'd page.
+        if emplaced_page:
-            # That is because we rasterize pages with /Rotate applied,
+            content_rotation = autorotate_correction
-            # so that the OCR image text is upright and comes back upright.
+        text_rotation = autorotate_correction
-            text_misaligned = (autorotate_correction - content_rotation) % 360
+        text_misaligned = (text_rotation - content_rotation) % 360
-            log.debug(
+        log.debug(
-                f"Text rotation: (autorotate, content) -> text misalignment = "
+            f"Text rotation: (text, autorotate, content) -> text misalignment = "
-                f"({autorotate_correction}, {content_rotation}) -> {text_misaligned}"
+            f"({text_rotation}, {autorotate_correction}, {content_rotation}) -> {text_misaligned}"
-            )
+        )
-        else:
-            text_misaligned = 0
 
         if textpdf and self.font:
-            # Graft the text layer onto this page, whether new or old
+            # Graft the text layer onto this page, whether new or old, possibly
+            # rotating the text layer by the amount is misaligned.
             strip_old = self.context.options.redo_ocr
             self._graft_text_layer(
                 page_num=pageno + 1,
@@ -151,14 +149,14 @@ class OcrGrafter:
                 strip_old_text=strip_old,
             )
 
-        # Correct the page rotation
+        # Correct the overall page rotation if needed, now that the text and content
+        # are aligned
         page_rotation = (content_rotation - autorotate_correction) % 360
         self.pdf_base.pages[pageno].Rotate = page_rotation
         log.debug(
             f"Page rotation: (content, auto) -> page = "
             f"({content_rotation}, {autorotate_correction}) -> {page_rotation}"
         )
-
         if self.emplacements % MAX_REPLACE_PAGES == 0:
             self.save_and_reload()
 

src/ocrmypdf/_pipeline.py

@@ -19,6 +19,7 @@ import img2pdf
 import pikepdf
 from pikepdf.models.metadata import encode_pdf_date
 from PIL import Image, ImageColor, ImageDraw
+from tqdm import tqdm
 
 from ocrmypdf import leptonica
 from ocrmypdf._exec import unpaper
@@ -332,8 +333,10 @@ def rasterize_preview(input_file: Path, page_context: PageContext):
         output_file=output_file,
         raster_device='jpeggray',
         raster_dpi=canvas_dpi,
-        page_dpi=page_dpi,
         pageno=page_context.pageinfo.pageno + 1,
+        page_dpi=page_dpi,
+        rotation=0,
+        filter_vector=False,
     )
     return output_file
 
@@ -433,7 +436,7 @@ def rasterize(
 
     device = colorspaces[device_idx]
 
-    log.debug(f"Rasterize with {device}")
+    log.debug(f"Rasterize with {device}, rotation {correction}")
 
     # Produce the page image with square resolution or else deskew and OCR
     # will not work properly.
@@ -599,14 +602,17 @@ def create_pdf_page_from_image(image: Path, page_context: PageContext):
 
 
 def render_hocr_page(hocr: Path, page_context: PageContext):
+    options = page_context.options
     output_file = page_context.get_path('ocr_hocr.pdf')
-    dpi = get_page_square_dpi(page_context.pageinfo, page_context.options)
+    dpi = get_page_square_dpi(page_context.pageinfo, options)
+    debug_mode = options.pdf_renderer == 'hocrdebug'
+
     hocrtransform = HocrTransform(hocr, dpi.x)  # square
     hocrtransform.to_pdf(
         output_file,
         image_filename=None,
-        show_bounding_boxes=False,
+        show_bounding_boxes=False if not debug_mode else True,
-        invisible_text=True,
+        invisible_text=True if not debug_mode else False,
         interword_spaces=True,
     )
     return output_file
@@ -707,6 +713,7 @@ def convert_to_pdfa(input_pdf: Path, input_ps_stub: Path, context: PdfContext):
         output_file=output_file,
         compression=options.pdfa_image_compression,
         pdfa_part=options.output_type[-1],  # is pdfa-1, pdfa-2, or pdfa-3
+        progressbar_class=tqdm if options.progress_bar else None,
     )
 
     return output_file

src/ocrmypdf/_plugin_manager.py

@@ -82,7 +82,7 @@ def _setup_plugins(
         pm.register(module)
 
 
-def get_plugin_manager(plugins: List[str], builtins=True):
+def get_plugin_manager(plugins: List[Union[str, Path]], builtins=True):
     pm = OcrmypdfPluginManager(
         project_name='ocrmypdf',
         setup_func=partial(_setup_plugins, plugins=plugins, builtins=builtins),

src/ocrmypdf/_sync.py

@@ -55,6 +55,7 @@ from ocrmypdf._validation import (
 )
 from ocrmypdf.exceptions import ExitCode, ExitCodeException
 from ocrmypdf.helpers import (
+    NeverRaise,
     available_cpu_count,
     check_pdf,
     pikepdf_enable_mmap,
@@ -206,12 +207,13 @@ def exec_page_sync(page_context: PageContext):
             visible_image_out, page_context
         )
 
-    if options.pdf_renderer == 'hocr':
+    if options.pdf_renderer.startswith('hocr'):
         (hocr_out, text_out) = ocr_engine_hocr(ocr_image_out, page_context)
         ocr_out = render_hocr_page(hocr_out, page_context)
-
+    elif options.pdf_renderer == 'sandwich':
-    if options.pdf_renderer == 'sandwich':
         (ocr_out, text_out) = ocr_engine_textonly_pdf(ocr_image_out, page_context)
+    else:
+        raise NotImplementedError(f"pdf_renderer {options.pdf_renderer}")
 
     return PageResult(
         pageno=page_context.pageno,
@@ -244,7 +246,8 @@ def exec_concurrent(context: PdfContext):
     """Execute the pipeline concurrently"""
 
     # Run exec_page_sync on every page context
-    max_workers = min(len(context.pdfinfo), context.options.jobs)
+    options = context.options
+    max_workers = min(len(context.pdfinfo), options.jobs)
     if max_workers > 1:
         log.info("Start processing %d pages concurrently", max_workers)
 
@@ -267,14 +270,14 @@ def exec_concurrent(context: PdfContext):
             tls.pageno = None
 
     exec_progress_pool(
-        use_threads=context.options.use_threads,
+        use_threads=options.use_threads,
         max_workers=max_workers,
         tqdm_kwargs=dict(
             total=(2 * len(context.pdfinfo)),
-            desc='OCR',
+            desc='OCR' if options.tesseract_timeout > 0 else 'Image processing',
             unit='page',
             unit_scale=0.5,
-            disable=not context.options.progress_bar,
+            disable=not options.progress_bar,
         ),
         task_initializer=partial(worker_init, PIL.Image.MAX_IMAGE_PIXELS),
         task=exec_page_sync,
@@ -283,10 +286,10 @@ def exec_concurrent(context: PdfContext):
     )
 
     # Output sidecar text
-    if context.options.sidecar:
+    if options.sidecar:
         text = merge_sidecars(sidecars, context)
         # Copy text file to destination
-        copy_final(text, context.options.sidecar, context)
+        copy_final(text, options.sidecar, context)
 
     # Merge layers to one single pdf
     pdf = ocrgraft.finalize()
@@ -296,16 +299,17 @@ def exec_concurrent(context: PdfContext):
     pdf = post_process(pdf, context)
 
     # Copy PDF file to destination
-    copy_final(pdf, context.options.output_file, context)
+    copy_final(pdf, options.output_file, context)
 
 
-class NeverRaise(Exception):
+def configure_debug_logging(log_filename, prefix: str = ''):
-    """An exception that is never raised"""
+    """
+    Create a debug log file at a specified location.
 
-    pass  # pylint: disable=unnecessary-pass
+    Arguments:
-
+        log_filename: Where to the put the log file.
-
+        prefix: The logging domain prefix that should be sent to the log.
-def configure_debug_logging(log_filename, prefix=''):
+    """
     log_file_handler = logging.FileHandler(log_filename, delay=True)
     log_file_handler.setLevel(logging.DEBUG)
     formatter = logging.Formatter(
@@ -326,11 +330,14 @@ def run_pipeline(options, *, plugin_manager, api=False):
     if not plugin_manager:
         plugin_manager = get_plugin_manager(options.plugins)
 
-    work_folder = Path(mkdtemp(prefix="com.github.ocrmypdf."))
+    work_folder = Path(mkdtemp(prefix="ocrmypdf.io."))
     debug_log_handler = None
-    if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
+    if (
-        'PYTEST_CURRENT_TEST', ''
+        (options.keep_temporary_files or options.verbose >= 1)
+        and not os.environ.get('PYTEST_CURRENT_TEST', '')
+        and not api
     ):
+        # Debug log for command line interface only with verbose output
         debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
 
     pikepdf_enable_mmap()

src/ocrmypdf/_validation.py

@@ -78,7 +78,7 @@ def check_options_languages(options, ocr_engine_languages):
 def check_options_output(options):
     is_latin = options.languages.issubset(HOCR_OK_LANGS)
 
-    if options.pdf_renderer == 'hocr' and not is_latin:
+    if options.pdf_renderer.startswith('hocr') and not is_latin:
         msg = (
             "The 'hocr' PDF renderer is known to cause problems with one "
             "or more of the languages in your document.  Use "
@@ -213,12 +213,12 @@ def check_options_optimizing(options):
 
 
 def check_options_advanced(options):
-    if options.pdfa_image_compression != 'auto' and options.output_type.startswith(
+    if options.pdfa_image_compression != 'auto' and not options.output_type.startswith(
         'pdfa'
     ):
         log.warning(
-            "--pdfa-image-compression argument has no effect when "
+            "--pdfa-image-compression argument only applies when "
-            "--output-type is not 'pdfa', 'pdfa-1', or 'pdfa-2'"
+            "--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
         )
 
 

src/ocrmypdf/api.py

@@ -9,8 +9,10 @@ import logging
 import os
 import sys
 from enum import IntEnum
+from io import IOBase
 from pathlib import Path
-from typing import BinaryIO, Iterable, Union
+from typing import AnyStr, BinaryIO, Iterable, Optional, Union
+from warnings import warn
 
 from ocrmypdf._logging import PageNumberFilter, TqdmConsole
 from ocrmypdf._plugin_manager import get_plugin_manager
@@ -25,7 +27,8 @@ except ModuleNotFoundError:
     coloredlogs = None
 
 
-PathOrIO = Union[BinaryIO, os.PathLike, str, bytes]
+StrPath = Union[os.PathLike, AnyStr]
+PathOrIO = Union[BinaryIO, StrPath]
 
 
 class Verbosity(IntEnum):
@@ -44,16 +47,28 @@ def configure_logging(
 ):
     """Set up logging.
 
-    Library users may wish to use this function if they want their log output to be
+    Before calling :func:`ocrmypdf.ocr()`, you can use this function to
-    similar to ocrmypdf command line interface. If not used, the external application
+    configure logging, if you want ocrmypdf's output to look like the ocrmypdf
-    should configure logging on its own.
+    command line interface. It will register log handlers, log filters, and
+    formatters, configure color logging to standard error, and adjust the log
+    levels of third party libraries. Details of this are fine-tuned and subject
+    to change. The ``verbosity`` argument is equivalent to the argument
+    ``--verbose`` and applies those settings.
 
-    ocrmypdf will perform all of its logging under the ``"ocrmypdf"`` logging namespace.
+    If this function is not called, ocrmypdf will not configure logging, and it
-    In addition, ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library
+    is up to the caller of ``ocrmypdf.ocr()`` to set up logging as it wishes using
-    user may wish to configure both; note that pdfminer is extremely chatty at the log
+    the Python standard library's logging module. If this function is called,
-    level ``logging.INFO``.
+    the caller may of course make further adjustments to logging.
 
-    Library users may perform additional configuration afterwards.
+    Regardless of whether this function is called, ocrmypdf will perform all of
+    its logging under the ``"ocrmypdf"`` logging namespace. In addition,
+    ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library user
+    may wish to configure both; note that pdfminer is extremely chatty at the
+    log level ``logging.INFO``.
+
+    This function does not set up the ``debug.log`` log file that the command
+    line interface does at certain verbosity levels. Applications should configure
+    their own debug logging.
 
     Args:
         verbosity (Verbosity): Verbosity level.
@@ -160,14 +175,14 @@ def create_options(
         else:
             raise TypeError(f"{arg}: {val} ({type(val)})")
 
-    try:
+    if isinstance(input_file, (BinaryIO, IOBase)):
-        cmdline.append(os.fspath(input_file))
-    except TypeError:
         cmdline.append('stream://input_file')
-    try:
+    else:
-        cmdline.append(os.fspath(output_file))
+        cmdline.append(os.fspath(input_file))
-    except TypeError:
+    if isinstance(output_file, (BinaryIO, IOBase)):
         cmdline.append('stream://output_file')
+    else:
+        cmdline.append(os.fspath(output_file))
 
     parser._api_mode = True
     options = parser.parse_args(cmdline)
@@ -189,7 +204,7 @@ def ocr(  # pylint: disable=unused-argument
     language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
-    sidecar: os.PathLike = None,
+    sidecar: Optional[StrPath] = None,
     jobs: int = None,
     use_threads: bool = None,
     title: str = None,
@@ -226,7 +241,7 @@ def ocr(  # pylint: disable=unused-argument
     user_words: os.PathLike = None,
     user_patterns: os.PathLike = None,
     fast_web_view: float = None,
-    plugins: Iterable[Union[str, Path]] = None,
+    plugins: Iterable[StrPath] = None,
     keep_temporary_files: bool = None,
     progress_bar: bool = None,
     **kwargs,
@@ -248,7 +263,7 @@ def ocr(  # pylint: disable=unused-argument
             read.
         output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
             interpreted as file system path to the output file. If the object
-            appears to be a writable stream (with methods such as ``.read()`` and
+            appears to be a writable stream (with methods such as ``.write()`` and
             ``.seek()``), the output will be written to this stream. If
             ``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
             (provided that standard output does not seem to be a terminal device).
@@ -285,15 +300,18 @@ def ocr(  # pylint: disable=unused-argument
     else:
         plugins = list(plugins)
 
-    parser = get_parser()
+    # No new variable names should be assigned until these two steps are run
-    _plugin_manager = get_plugin_manager(plugins)
+    create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
-    _plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
-
-    create_options_kwargs = {
-        k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
-    }
     create_options_kwargs.update(kwargs)
 
+    parser = get_parser()
+    create_options_kwargs['parser'] = parser
+    plugin_manager = get_plugin_manager(plugins)
+    plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
+
+    if 'verbose' in kwargs:
+        warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
+
     options = create_options(**create_options_kwargs)
-    check_options(options, _plugin_manager)
+    check_options(options, plugin_manager)
-    return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
+    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)

src/ocrmypdf/builtin_plugins/ghostscript.py

@@ -61,9 +61,9 @@ def rasterize_pdf_page(
     raster_device,
     raster_dpi,
     pageno,
-    page_dpi=None,
+    page_dpi,
-    rotation=None,
+    rotation,
-    filter_vector=False,
+    filter_vector,
 ):
     ghostscript.rasterize_pdf(
         input_file,
@@ -79,12 +79,21 @@ def rasterize_pdf_page(
 
 
 @hookimpl
-def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
+def generate_pdfa(
+    pdf_pages,
+    pdfmark,
+    output_file,
+    compression,
+    pdf_version,
+    pdfa_part,
+    progressbar_class,
+):
     ghostscript.generate_pdfa(
         pdf_pages=[*pdf_pages, pdfmark],
         output_file=output_file,
         compression=compression,
         pdf_version=pdf_version,
         pdfa_part=pdfa_part,
+        progressbar_class=progressbar_class,
     )
     return output_file

src/ocrmypdf/builtin_plugins/tesseract_ocr.py

@@ -121,9 +121,7 @@ def validate(pdfinfo, options):
         os.environ['OMP_THREAD_LIMIT'] = str(tess_threads)
     else:
         tess_threads = int(os.environ['OMP_THREAD_LIMIT'])
-
+    log.debug("Using Tesseract OpenMP thread limit %d", tess_threads)
-    if tess_threads > 1:
-        log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
 
 
 class TesseractOcrEngine(OcrEngine):

src/ocrmypdf/cli.py

@@ -6,12 +6,15 @@
 
 
 import argparse
+from typing import Optional, Type, TypeVar
 
 from ocrmypdf._version import PROGRAM_NAME as _PROGRAM_NAME
 from ocrmypdf._version import __version__ as _VERSION
 
+T = TypeVar('T')
 
-def numeric(basetype, min_=None, max_=None):
+
+def numeric(basetype: Type[T], min_: Optional[T] = None, max_: Optional[T] = None):
     """Validator for numeric params"""
     min_ = basetype(min_) if min_ is not None else None
     max_ = basetype(max_) if max_ is not None else None
@@ -407,7 +410,7 @@ Online documentation is located at:
     )
     advanced.add_argument(
         '--pdf-renderer',
-        choices=['auto', 'hocr', 'sandwich'],
+        choices=['auto', 'hocr', 'sandwich', 'hocrdebug'],
         default='auto',
         help="Choose OCR PDF renderer - the default option is to let OCRmyPDF "
         "choose.  See documentation for discussion.",

src/ocrmypdf/helpers.py

@@ -58,6 +58,10 @@ class Resolution(namedtuple('Resolution', ('x', 'y'))):
         return f"Resolution({self.x}x{self.y} dpi)"
 
 
+class NeverRaise(Exception):
+    """An exception that is never raised"""
+
+
 def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike):
     """
     Helper function: relinks soft symbolic link if necessary
@@ -186,17 +190,25 @@ def check_pdf(input_file: Path) -> bool:
                 log.warning(msg)
 
         sio = StringIO()
-        linearize = None
+        linearize_msgs = ''
         try:
+            # If linearization is missing entirely, we do not complain. We do
+            # complain if linearization is present but incorrect.
             pdf.check_linearization(sio)
         except RuntimeError:
             pass
+        except (
+            getattr(pikepdf, 'ForeignObjectError')
+            if pikepdf.__version__ == '2.1.0'  # This version may throw wrong exception
+            else NeverRaise
+        ):
+            pass
         else:
-            linearize = sio.getvalue()
+            linearize_msgs = sio.getvalue()
-            if linearize:
+            if linearize_msgs:
-                log.warning(linearize)
+                log.warning(linearize_msgs)
 
-        if not messages and not linearize:
+        if not messages and not linearize_msgs:
             return True
         return False
     finally:
@@ -204,10 +216,7 @@ def check_pdf(input_file: Path) -> bool:
             pdf.close()
 
 
-T = TypeVar('T')
+def clamp(n, smallest, largest):  # mypy doesn't understand types for this
-
-
-def clamp(n: T, smallest: T, largest: T) -> T:
     """Clamps the value of n to between smallest and largest."""
     return max(smallest, min(n, largest))
 
@@ -220,6 +229,7 @@ def pikepdf_enable_mmap():
     #     log.debug("pikepdf mmap not available")
     # We found a race condition probably related to pybind issue #2252 that can
     # cause a crash. For now, disable pikepdf mmap to be on the safe side.
+    # Fix is not in pybind11 2.6.0
     log.debug("pikepdf mmap disabled")
     return
 

src/ocrmypdf/hocrtransform.py

@@ -42,6 +42,8 @@ from reportlab.lib.colors import black, cyan, magenta, red
 from reportlab.lib.units import inch
 from reportlab.pdfgen.canvas import Canvas
 
+Element = ElementTree.Element
+
 Rect = namedtuple('Rect', ['x1', 'y1', 'x2', 'y2'])
 
 
@@ -105,7 +107,7 @@ class HocrTransform:
         else:
             return ''
 
-    def _get_element_text(self, element):
+    def _get_element_text(self, element: Element):
         """
         Return the textual content of the element and its children
         """
@@ -119,7 +121,7 @@ class HocrTransform:
         return text
 
     @classmethod
-    def element_coordinates(cls, element) -> Rect:
+    def element_coordinates(cls, element: Element) -> Rect:
         """
         Returns a tuple containing the coordinates of the bounding box around
         an element
@@ -133,7 +135,7 @@ class HocrTransform:
         return out
 
     @classmethod
-    def baseline(cls, element) -> Tuple[float, float]:
+    def baseline(cls, element: Element) -> Tuple[float, float]:
         """
         Returns a tuple containing the baseline slope and intercept.
         """
@@ -149,7 +151,7 @@ class HocrTransform:
         """
         return Rect._make((c / self.dpi * inch) for c in pxl)
 
-    def _child_xpath(self, html_tag, html_class=None):
+    def _child_xpath(self, html_tag: str, html_class: Optional[str] = None) -> str:
         xpath = f".//{self.xmlns}{html_tag}"
         if html_class:
             xpath += f"[@class='{html_class}']"
@@ -280,13 +282,15 @@ class HocrTransform:
     def _do_line(
         self,
         pdf: Canvas,
-        line,
+        line: Optional[Element],
         elemclass: str,
         fontname: str,
         invisible_text: bool,
         interword_spaces: bool,
         show_bounding_boxes: bool,
     ):
+        if not line:
+            return
         pxl_line_coords = self.element_coordinates(line)
         line_box = self.pt_from_pixel(pxl_line_coords)
         line_height = line_box.y2 - line_box.y1

src/ocrmypdf/leptonica.py

@@ -27,15 +27,16 @@ from tempfile import TemporaryFile
 
 from ocrmypdf.exceptions import MissingDependencyError
 from ocrmypdf.lib._leptonica import ffi
-from ocrmypdf.subprocess import shim_paths_with_program_files
 
 # pylint: disable=protected-access
 
 logger = logging.getLogger(__name__)
 
 if os.name == 'nt':
+    from ocrmypdf.subprocess._windows import shim_env_path
+
     libname = 'liblept-5'
-    os.environ['PATH'] = shim_paths_with_program_files()
+    os.environ['PATH'] = shim_env_path()
 else:
     libname = 'lept'
 _libpath = find_library(libname)
@@ -58,9 +59,9 @@ if not _libpath:
         ---------------------------------------------------------------------
         """
     )
-if os.name == 'nt':  
+if os.name == 'nt':
-    # On Windows, recent versions of libpng require zlib. We have to make sure 
+    # On Windows, recent versions of libpng require zlib. We have to make sure
-    # the zlib version being loaded is the same one that libpng was built with. 
+    # the zlib version being loaded is the same one that libpng was built with.
     # This tries to import zlib from Tesseract's installation folder, falling back
     # to find_library() if liblept is being loaded from somewhere else.
     # Loading zlib from other places could cause a version mismatch

src/ocrmypdf/optimize.py

@@ -25,7 +25,6 @@ from typing import (
     Optional,
     Sequence,
     Tuple,
-    Union,
 )
 
 import img2pdf
@@ -115,14 +114,29 @@ def extract_image_jbig2(
         and filtdp[0] != Name.JBIG2Decode
         and jbig2enc.available()
     ):
-        try:
+        # Save any colorspace associated with the image, so that we
-            imgname = root / f'{xref:08d}'
+        # will export a pure 1-bit PNG with no palette or ICC profile.
-            with imgname.open('wb') as f:
+        # Showing the palette or ICC to jbig2enc will cause it to perform
-                ext = pim.extract_to(stream=f)
+        # colorspace transform to 1bpp, which will conflict the palette or
-            imgname.rename(imgname.with_suffix(ext))
+        # ICC if it exists.
-        except pikepdf.UnsupportedImageTypeError:
+        colorspace = pim.obj.get(pikepdf.Name.ColorSpace, None)
-            return None
+        if colorspace is not None or pim.image_mask:
-        return XrefExt(xref, ext)
+            try:
+                # Set to DeviceGray temporarily; we already in 1 bpc.
+                pim.obj.ColorSpace = pikepdf.Name.DeviceGray
+                imgname = root / f'{xref:08d}'
+                with imgname.open('wb') as f:
+                    ext = pim.extract_to(stream=f)
+                imgname.rename(imgname.with_suffix(ext))
+            except pikepdf.UnsupportedImageTypeError:
+                return None
+            finally:
+                # Restore image colorspace after temporarily setting it to DeviceGray
+                if colorspace is not None:
+                    pim.obj.ColorSpace = colorspace
+                else:
+                    del pim.obj.ColorSpace
+            return XrefExt(xref, ext)
     return None
 
 
@@ -279,10 +293,6 @@ def extract_images_jbig2(pike: Pdf, root: Path, options) -> Dict[int, List[XrefE
         group = pageno // options.jbig2_page_group_size
         jbig2_groups[group].append(xref_ext)
 
-    # Elide empty groups
-    jbig2_groups = {
-        group: xrefs for group, xrefs in jbig2_groups.items() if len(xrefs) > 0
-    }
     log.debug("Optimizable images: JBIG2 groups: %s", (len(jbig2_groups),))
     return jbig2_groups
 
@@ -602,7 +612,7 @@ def optimize(input_file: Path, output_file: Path, context, save_settings) -> Non
         )
     ratio = input_size / output_size
     savings = 1 - output_size / input_size
-    log.info(f"Optimize ratio: {ratio:.2f} savings: {(100 * savings):.1f}%")
+    log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
 
     if savings < 0:
         log.info("Image optimization did not improve the file - discarded")

src/ocrmypdf/pdfa.py

@@ -33,6 +33,7 @@ def _postscript_objdef(
     objtype = '/stream' if stream_name else '/dict'
 
     if stream_name:
+        assert stream_data is not None
         a85_data = base64.a85encode(stream_data, adobe=True).decode('ascii')
         yield f'{stream_name} ' + a85_data
         yield 'def'

src/ocrmypdf/pdfinfo/info.py

@@ -627,9 +627,15 @@ def _pdf_get_pageinfo(
 worker_pdf = None
 
 
-def _pdf_pageinfo_sync_init(infile):
+def _pdf_pageinfo_sync_init(infile: Path, pdfminer_loglevel):
     global worker_pdf  # pylint: disable=global-statement
     pikepdf_enable_mmap()
+
+    logging.getLogger('pdfminer').setLevel(pdfminer_loglevel)
+
+    # If this function is called as a thread initializer, we need a messy hack
+    # to close worker_pdf. If called as a process, it will be released when the
+    # process is terminated.
     worker_pdf = pikepdf.open(infile)
 
 
@@ -643,6 +649,7 @@ def _pdf_pageinfo_sync(args):
 def _pdf_pageinfo_concurrent(
     pdf, infile, progbar, max_workers, check_pages, detailed_analysis=False
 ):
+    global worker_pdf  # pylint: disable=global-statement
     pages = [None] * len(pdf.pages)
 
     def update_pageinfo(result, pbar):
@@ -663,17 +670,25 @@ def _pdf_pageinfo_concurrent(
         # a separate process.
         use_threads = True
 
-    exec_progress_pool(
+    try:
-        use_threads=use_threads,
+        exec_progress_pool(
-        max_workers=n_workers,
+            use_threads=use_threads,
-        tqdm_kwargs=dict(
+            max_workers=n_workers,
-            total=total, desc="Scanning contents", unit='page', disable=not progbar
+            tqdm_kwargs=dict(
-        ),
+                total=total, desc="Scanning contents", unit='page', disable=not progbar
-        task_initializer=partial(_pdf_pageinfo_sync_init, infile),
+            ),
-        task=_pdf_pageinfo_sync,
+            task_initializer=partial(
-        task_arguments=contexts,
+                _pdf_pageinfo_sync_init, infile, logging.getLogger('pdfminer').level
-        task_finished=update_pageinfo,
+            ),
-    )
+            task=_pdf_pageinfo_sync,
+            task_arguments=contexts,
+            task_finished=update_pageinfo,
+        )
+    finally:
+        if worker_pdf and use_threads:
+            assert n_workers == 1, "Should have only one worker when threaded"
+            # This is messy, but if we ran in thread, close worker_pdf
+            worker_pdf.close()
     return pages
 
 
@@ -808,12 +823,12 @@ class PdfInfo:
                 check_pages=check_pages,
                 detailed_analysis=detailed_analysis,
             )
-            self._needs_rendering = pdf.root.get('/NeedsRendering', False)
+            self._needs_rendering = pdf.Root.get('/NeedsRendering', False)
             self._has_acroform = False
-            if '/AcroForm' in pdf.root:
+            if '/AcroForm' in pdf.Root:
-                if len(pdf.root.AcroForm.get('/Fields', [])) > 0:
+                if len(pdf.Root.AcroForm.get('/Fields', [])) > 0:
                     self._has_acroform = True
-                elif '/XFA' in pdf.root.AcroForm:
+                elif '/XFA' in pdf.Root.AcroForm:
                     self._has_acroform = True
 
     @property

src/ocrmypdf/pluginspec.py

@@ -59,7 +59,7 @@ def check_options(options: Namespace) -> None:
     Note:
         This hook will be called from the main process, and may modify global state
         before child worker processes are forked.
-	"""
+    """
 
 
 @hookspec
@@ -89,9 +89,9 @@ def rasterize_pdf_page(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int,
-    page_dpi: Optional[Resolution] = None,
+    page_dpi: Optional[Resolution],
-    rotation: Optional[int] = None,
+    rotation: Optional[int],
-    filter_vector: bool = False,
+    filter_vector: bool,
 ) -> Path:
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units.
 
@@ -280,6 +280,7 @@ def generate_pdfa(
     compression: str,
     pdf_version: str,
     pdfa_part: str,
+    progressbar_class,
 ) -> Path:
     """Generate a PDF/A.
 
@@ -302,10 +303,21 @@ def generate_pdfa(
             At its own discretion, the PDF/A generator may raise the version,
             but should not lower it.
         pdfa_part: The desired PDF/A compliance level, such as ``'2B'``.
+        progressbar_class: The class of a progress bar with a tqdm-like API. An
+            instance of this class will be initialized when PDF/A conversion
+            begins, using
+            ``instance = progressbar_class(total: int, desc: str, unit:str)``,
+            defining the number of work units, a user-visible description,
+            and the name of the work units ("page"). Then ``instance.update()``
+            will be called when a work unit is completed. If ``None``, no
+            progress information is reported.
 
     Returns:
         Path: If successful, the hook should return ``output_file``.
 
     Note:
         This is a :ref:`firstresult hook<firstresult>`.
+
+    See also:
+        https://github.com/tqdm/tqdm
     """

src/ocrmypdf/py.typed

@@ -0,0 +1 @@
+# ocrmypdf is typed

src/ocrmypdf/subprocess/__init__.py

@@ -10,14 +10,13 @@
 import logging
 import os
 import re
-import shutil
 import sys
 from collections.abc import Mapping
 from contextlib import suppress
 from distutils.version import LooseVersion
 from functools import lru_cache
 from pathlib import Path
-from subprocess import PIPE, STDOUT, CalledProcessError
+from subprocess import PIPE, STDOUT, CalledProcessError, CompletedProcess, Popen
 from subprocess import run as subprocess_run
 
 from ocrmypdf.exceptions import MissingDependencyError
@@ -25,14 +24,78 @@ from ocrmypdf.exceptions import MissingDependencyError
 log = logging.getLogger(__name__)
 
 
-def run(args, *, env=None, **kwargs):
+def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
     """Wrapper around :py:func:`subprocess.run`
 
     The main purpose of this wrapper is to log subprocess output in an orderly
     fashion that indentifies the responsible subprocess. An additional
     task is that this function goes to greater lengths to find possible Windows
     locations of our dependencies when they are not on the system PATH.
+
+    Arguments should be identical to ``subprocess.run``, except for following:
+
+    Arguments:
+        logs_errors_to_stdout: If True, indicates that the process writes its error
+            messages to stdout rather than stderr, so stdout should be logged
+            if there is an error. If False, stderr is logged. Could be used with
+            stderr=STDOUT, stdout=PIPE for example.
     """
+    args, env, process_log, _text = _fix_process_args(args, env, kwargs)
+
+    stderr = None
+    stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
+    try:
+        proc = subprocess_run(args, env=env, **kwargs)
+    except CalledProcessError as e:
+        stderr = getattr(e, stderr_name, None)
+        raise
+    else:
+        stderr = getattr(proc, stderr_name, None)
+    finally:
+        if process_log.isEnabledFor(logging.DEBUG) and stderr:
+            with suppress(AttributeError, UnicodeDecodeError):
+                stderr = stderr.decode('utf-8', 'replace')
+            if logs_errors_to_stdout:
+                process_log.debug("stdout/stderr = %s", stderr)
+            else:
+                process_log.debug("stderr = %s", stderr)
+    return proc
+
+
+def run_polling_stderr(args, *, callback, check=False, env=None, **kwargs):
+    """Run a process like ``ocrmypdf.subprocess.run``, and poll stderr.
+
+    Every line of produced by stderr will be forwarded to the callback function.
+    The intended use is monitoring progress of subprocesses that output their
+    own progress indicators. In addition, each line will be logged if debug
+    logging is enabled.
+
+    Requires stderr to be opened in text mode for ease of handling errors. In
+    addition the expected encoding= and errors= arguments should be set. Note
+    that if stdout is already set up, it need not be binary.
+    """
+    args, env, process_log, text = _fix_process_args(args, env, kwargs)
+    assert text, "Must use text=True"
+
+    proc = Popen(args, env=env, **kwargs)
+
+    lines = []
+    while proc.poll() is None:
+        for msg in iter(proc.stderr.readline, ''):
+            if process_log.isEnabledFor(logging.DEBUG):
+                process_log.debug(msg.strip())
+            callback(msg)
+            lines.append(msg)
+    stderr = ''.join(lines)
+
+    if check and proc.returncode != 0:
+        raise CalledProcessError(proc.returncode, args, output=None, stderr=stderr)
+    return CompletedProcess(args, proc.returncode, None, stderr=stderr)
+
+
+def _fix_process_args(args, env, kwargs):
+    assert 'universal_newlines' not in kwargs, "Use text= instead of universal_newlines"
+
     if not env:
         env = os.environ
 
@@ -40,53 +103,23 @@ def run(args, *, env=None, **kwargs):
     program = args[0]
 
     if os.name == 'nt':
-        args = _fix_windows_args(program, args, env)
+        from ocrmypdf.subprocess._windows import fix_windows_args
+
+        args = fix_windows_args(program, args, env)
 
     log.debug("Running: %s", args)
-    process_log = log.getChild('subprocess.' + os.path.basename(program))
+    process_log = log.getChild(os.path.basename(program))
-    if sys.version_info < (3, 7) and os.name == 'nt':
+    text = kwargs.get('text', False)
-        # Can't use close_fds=True on Windows with Python 3.6 or older
+    if sys.version_info < (3, 7):
-        # https://bugs.python.org/issue19575, etc.
+        if os.name == 'nt':
-        kwargs['close_fds'] = False
+            # Can't use close_fds=True on Windows with Python 3.6 or older
-
+            # https://bugs.python.org/issue19575, etc.
-    stderr = None
+            kwargs['close_fds'] = False
-    try:
+        if 'text' in kwargs:
-        proc = subprocess_run(args, env=env, **kwargs)
+            # Convert run(...text=) to run(...universal_newlines=) for Python 3.6
-    except CalledProcessError as e:
+            kwargs['universal_newlines'] = kwargs['text']
-        stderr = getattr(e, 'stderr', None)
+            del kwargs['text']
-        raise
+    return args, env, process_log, text
-    else:
-        stderr = getattr(proc, 'stderr', None)
-    finally:
-        if process_log.isEnabledFor(logging.DEBUG) and stderr:
-            with suppress(AttributeError, UnicodeDecodeError):
-                stderr = stderr.decode('utf-8', 'replace')
-            process_log.debug("stderr = %s", stderr)
-    return proc
-
-
-def _fix_windows_args(program, args, env):
-    """Adjust our desired program and command line arguments for use on Windows"""
-
-    if sys.version_info < (3, 8):
-        # bpo-33617 - Windows needs manual Path -> str conversion
-        args = [os.fspath(arg) for arg in args]
-        program = os.fspath(program)
-
-    # If we are running a .py on Windows, ensure we call it with this Python
-    # (to support test suite shims)
-    if program.lower().endswith('.py'):
-        args = [sys.executable] + args
-
-    paths = os.pathsep.join(os.get_exec_path(env))
-    if not shutil.which(args[0], path=paths):
-        # If the program we want is not on the PATH, add some interesting
-        # locations in %PROGRAMFILES% to the PATH and try again
-        shimmed_path = shim_paths_with_program_files(env)
-        new_args0 = shutil.which(args[0], path=shimmed_path)
-        if new_args0:
-            args[0] = new_args0
-    return args
 
 
 @lru_cache(maxsize=None)
@@ -107,7 +140,7 @@ def get_version(
         proc = run(
             args_prog,
             close_fds=True,
-            universal_newlines=True,
+            text=True,
             stdout=PIPE,
             stderr=STDOUT,
             check=True,
@@ -126,44 +159,18 @@ def get_version(
         raise MissingDependencyError(
             f"Could not find program '{program}' on the PATH"
         ) from e
-    try:
+
-        version = re.match(regex, output.strip()).group(1)
+    match = re.match(regex, output.strip())
-    except AttributeError as e:
+    if not match:
         raise MissingDependencyError(
             f"The program '{program}' did not report its version. "
             f"Message was:\n{output}"
         )
+    version = match.group(1)
 
     return version
 
 
-def shim_paths_with_program_files(env=None):
-    if not env:
-        env = os.environ
-    program_files = env.get('PROGRAMFILES', '')
-    if not program_files:
-        return env.get('PATH', '')
-
-    def path_walker():
-        for path in Path(program_files).iterdir():
-            if not path.is_dir():
-                continue
-            if path.name.lower() == 'tesseract-ocr':
-                yield path
-            elif path.name.lower() == 'gs':
-                yield from (p for p in path.glob('**/bin') if p.is_dir())
-
-    paths = sorted(
-        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
-    )
-    paths.extend(
-        Path(str_path)
-        for str_path in os.get_exec_path(env)
-        if Path(str_path) not in set(paths)
-    )
-    return os.pathsep.join(str(p) for p in paths)
-
-
 missing_program = '''
 The program '{program}' could not be executed or was not found on your
 system PATH.

src/ocrmypdf/subprocess/_windows.py

@@ -0,0 +1,162 @@
+# © 2020 James R. Barlow: github.com/jbarlow83
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import logging
+import os
+import shutil
+import sys
+from distutils.version import LooseVersion
+from itertools import chain, filterfalse
+from pathlib import Path
+from typing import Any, Callable, Iterator, Optional, Tuple, TypeVar, cast
+
+try:
+    import winreg
+except ModuleNotFoundError as e:
+    raise ModuleNotFoundError("This module is for Windows only") from e
+
+log = logging.getLogger(__name__)
+
+T = TypeVar('T')
+
+
+def registry_enum(
+    key: winreg.HKEYType, enum_fn: Callable[[winreg.HKEYType, int], T]
+) -> Iterator[T]:
+    LIMIT = 999
+    n = 0
+    while n < LIMIT:
+        try:
+            yield enum_fn(key, n)
+            n += 1
+        except OSError:
+            break
+    if n == LIMIT:
+        raise ValueError(f"Too many registry keys under {key}")
+
+
+def registry_subkeys(key: winreg.HKEYType) -> Iterator[str]:
+    return registry_enum(key, winreg.EnumKey)
+
+
+def registry_values(key: winreg.HKEYType) -> Iterator[Tuple[str, Any, int]]:
+    return registry_enum(key, winreg.EnumValue)
+
+
+def registry_path_ghostscript(env=None) -> Iterator[Path]:
+    try:
+        with winreg.OpenKey(
+            winreg.HKEY_LOCAL_MACHINE, r"SOFTWARE\Artifex\GPL Ghostscript"
+        ) as k:
+            latest_gs = max(registry_subkeys(k), key=LooseVersion)
+        with winreg.OpenKey(
+            winreg.HKEY_LOCAL_MACHINE, fr"SOFTWARE\Artifex\GPL Ghostscript\{latest_gs}"
+        ) as k:
+            _, gs_path, _ = next(registry_values(k))
+            yield Path(gs_path) / 'bin'
+    except OSError as e:
+        log.warning(e)
+
+
+def registry_path_tesseract(env=None) -> Iterator[Path]:
+    try:
+        with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r"SOFTWARE\Tesseract-OCR") as k:
+            for subkey, val, _valtype in registry_values(k):
+                if subkey == 'InstallDir':
+                    tesseract_path = Path(val)
+                    yield tesseract_path
+    except OSError as e:
+        log.warning(e)
+
+
+def program_files_paths(env=None) -> Iterator[Path]:
+    if not env:
+        env = os.environ
+    program_files = env.get('PROGRAMFILES', '')
+
+    def path_walker() -> Iterator[Path]:
+        for path in Path(program_files).iterdir():
+            if not path.is_dir():
+                continue
+            if path.name.lower() == 'tesseract-ocr':
+                yield path
+            elif path.name.lower() == 'gs':
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
+
+    return iter(
+        sorted(
+            (p for p in path_walker()),
+            key=lambda p: (p.name, p.parent.name),
+            reverse=True,
+        )
+    )
+
+
+def paths_from_env(env=None) -> Iterator[Path]:
+    return (Path(p) for p in os.get_exec_path(env) if p)
+
+
+def shim_path(new_paths: Callable[[Any], Iterator[Path]], env=None) -> str:
+    if not env:
+        env = os.environ
+    return os.pathsep.join(str(p) for p in new_paths(env) if p)
+
+
+SHIMS = [
+    paths_from_env,
+    registry_path_ghostscript,
+    registry_path_tesseract,
+    program_files_paths,
+]
+
+
+def fix_windows_args(program, args, env):
+    """Adjust our desired program and command line arguments for use on Windows"""
+
+    if sys.version_info < (3, 8):
+        # bpo-33617 - Windows needs manual Path -> str conversion
+        args = [os.fspath(arg) for arg in args]
+        program = os.fspath(program)
+
+    # If we are running a .py on Windows, ensure we call it with this Python
+    # (to support test suite shims)
+    if program.lower().endswith('.py'):
+        args = [sys.executable] + args
+
+    # If the program we want is not on the PATH, check elsewhere
+    for shim in SHIMS:
+        shimmed_path = shim_path(shim, env)
+        new_args0 = shutil.which(args[0], path=shimmed_path)
+        if new_args0:
+            args[0] = new_args0
+            break
+
+    return args
+
+
+def unique_everseen(iterable, key=None):
+    "List unique elements, preserving order. Remember all elements ever seen."
+    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
+    # unique_everseen('ABBCcAD', str.lower) --> A B C D
+    seen = set()
+    seen_add = seen.add
+    if key is None:
+        key = lambda x: x
+    for element in iterable:
+        k = key(element)
+        if k not in seen:
+            seen_add(k)
+            yield element
+
+
+def shim_env_path(env=None):
+    if env is None:
+        env = os.environ
+
+    shim_paths = chain.from_iterable(shim(env) for shim in SHIMS)
+    return os.pathsep.join(
+        str(p) for p in unique_everseen(shim_paths, key=lambda p: str.casefold(str(p)))
+    )

tests/conftest.py

@@ -128,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
 
 
 @pytest.helpers.register
-def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
+def run_ocrmypdf(input_file, output_file, *args, text=True):
     "Run ocrmypdf and let caller deal with results"
 
     p_args = (
@@ -151,7 +151,7 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
         p_args,
         stdout=PIPE,
         stderr=PIPE,
-        universal_newlines=universal_newlines,
+        universal_newlines=text,  # When dropping support for Python 3.6 change to text=
         env=env,
         check=False,
     )

tests/plugins/gs_feature_elision.py

@@ -45,5 +45,6 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
         return output_file

tests/plugins/gs_pdfa_failure.py

@@ -23,7 +23,7 @@ from unittest.mock import patch
 
 from ocrmypdf import hookimpl
 from ocrmypdf.builtin_plugins import ghostscript
-from ocrmypdf.subprocess import run
+from ocrmypdf.subprocess import run_polling_stderr
 
 
 def run_rig_args(args, **kwargs):
@@ -33,13 +33,13 @@ def run_rig_args(args, **kwargs):
     new_args = [
         arg for arg in args if not arg.startswith('-dPDFA') and not arg.endswith('.ps')
     ]
-    proc = run(new_args, **kwargs)
+    proc = run_polling_stderr(new_args, **kwargs)
     return proc
 
 
 @hookimpl
 def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
-    with patch('ocrmypdf._exec.ghostscript.run', new=run_rig_args):
+    with patch('ocrmypdf._exec.ghostscript.run_polling_stderr', new=run_rig_args):
         ghostscript.generate_pdfa(
             pdf_pages=pdf_pages,
             pdfmark=pdfmark,
@@ -47,5 +47,6 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
         return output_file

tests/plugins/gs_render_failure.py

@@ -34,7 +34,7 @@ def raise_gs_fail(*args, **kwargs):
 
 @hookimpl
 def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
-    with patch('ocrmypdf._exec.ghostscript.run', new=raise_gs_fail):
+    with patch('ocrmypdf._exec.ghostscript.run_polling_stderr', new=raise_gs_fail):
         ghostscript.generate_pdfa(
             pdf_pages=pdf_pages,
             pdfmark=pdfmark,
@@ -42,5 +42,6 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
         return output_file

tests/plugins/tesseract_cache.py

@@ -165,7 +165,7 @@ def cached_run(options, run_args, **run_kwargs):
 
     def clean_sys_argv():
         for arg in run_args[1:]:
-            yield re.sub(r'.*/com.github.ocrmypdf[^/]+[/](.*)', r'$TMPDIR/\1', arg)
+            yield re.sub(r'.*/ocrmypdf[.]io[.][^/]+[/](.*)', r'$TMPDIR/\1', arg)
 
     manifest['args'] = list(clean_sys_argv())
     with (Path(CACHE_ROOT) / 'manifest.jsonl').open('a') as f:

tests/test_helpers.py

@@ -13,7 +13,6 @@ from unittest.mock import MagicMock
 import pytest
 
 from ocrmypdf import helpers as helpers
-from ocrmypdf.subprocess import shim_paths_with_program_files
 
 
 class TestSafeSymlink:
@@ -94,7 +93,10 @@ class TestFileIsWritable:
         assert not helpers.is_file_writable(pathmock)
 
 
+@pytest.mark.skipif(os.name != 'nt', reason="Windows test")
 def test_shim_paths(tmp_path):
+    from ocrmypdf.subprocess._windows import shim_env_path
+
     progfiles = tmp_path / 'Program Files'
     progfiles.mkdir()
     (progfiles / 'tesseract-ocr').mkdir()
@@ -103,9 +105,9 @@ def test_shim_paths(tmp_path):
     syspath = tmp_path / 'bin'
     env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
 
-    result_str = shim_paths_with_program_files(env=env)
+    result_str = shim_env_path(env=env)
     results = result_str.split(os.pathsep)
-    assert results[0].endswith('tesseract-ocr')
+    assert results[0] == str(syspath), results
-    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
+    assert results[-3].endswith('tesseract-ocr'), results
-    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
+    assert results[-2].endswith(os.path.join('gs', '9.52', 'bin')), results
-    assert results[3] == str(syspath)
+    assert results[-1].endswith(os.path.join('gs', '9.51', 'bin')), results

tests/test_main.py

@@ -598,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
 
@@ -659,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
         assert p.returncode == ExitCode.ok, p.stderr

tests/test_metadata.py

@@ -302,8 +302,8 @@ def test_kodak_toc(resources, outpdf):
 
     p = pikepdf.open(outpdf)
 
-    if pikepdf.Name.First in p.root.Outlines:
+    if pikepdf.Name.First in p.Root.Outlines:
-        assert isinstance(p.root.Outlines.First, pikepdf.Dictionary)
+        assert isinstance(p.Root.Outlines.First, pikepdf.Dictionary)
 
 
 def test_metadata_fixup_warning(resources, outdir, caplog):

tests/test_rotation.py

@@ -15,6 +15,7 @@ from PIL import Image
 
 from ocrmypdf import leptonica
 from ocrmypdf._exec import ghostscript, tesseract
+from ocrmypdf._plugin_manager import get_plugin_manager
 from ocrmypdf.helpers import Resolution
 from ocrmypdf.pdfinfo import PdfInfo
 
@@ -240,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
         '--rotate-pages',
         '--rotate-pages-threshold',
         '0.001',
-        universal_newlines=False,
+        text=False,
     )
     err = err.decode('utf-8', errors='replace')
     assert p.returncode == 0, err
@@ -256,3 +257,33 @@ def test_tesseract_orientation(resources, tmp_path):
     tesseract.get_orientation(  # Test results of this are unreliable
         tmp_path / '000001.png', engine_mode='3', timeout=10
     )
+
+
+def test_rasterize_rotates(resources, tmp_path):
+    pm = get_plugin_manager([])
+
+    img = tmp_path / 'img90.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=90,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (123, 151), "Image not rotated"
+
+    img = tmp_path / 'img180.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=180,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (151, 123), "Image not rotated"

tests/test_validation.py

@@ -137,9 +137,9 @@ def test_report_file_size(tmp_path, caplog):
     caplog.clear()
 
     waste_of_space = b'Dummy' * 5000
-    pdf.root.Dummy = waste_of_space
+    pdf.Root.Dummy = waste_of_space
     pdf.save(in_)
-    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.Root.Dummy2 = waste_of_space + waste_of_space
     pdf.save(out)
 
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(