v11.3.3 release notes

The parameters are equivalent but the latter is better named. Since
Python 3.6 doesn't support text= we use our wrapper to add it in that
place.

This is for subprocess.run.

.pre-commit-config.yaml

@@ -19,5 +19,5 @@ repos:
     rev: 19.10b0
     hooks:
       - id: black
-        language_version: python3.8
+        language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py

README.md

@@ -92,6 +92,9 @@ apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified lan
 
 # Arch Linux users
 pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
+
+# brew macOS users
+brew install tesseract-lang
 ```
 
 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.

azure-pipelines.yml

@@ -22,6 +22,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -59,6 +61,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -149,12 +153,6 @@ stages:
       - job: "macOS_Mojave"
         pool:
           vmImage: "macos-10.14"
-        strategy:
-          matrix:
-            Python37:
-              python.version: ""
-            Python38:
-              python.version: "python@3.8"
         steps:
           # https://github.com/actions/virtual-environments/issues/664
           # - task: UsePythonVersion@0
@@ -162,12 +160,8 @@ stages:
           #     versionSpec: "$(python.version)"
           - bash: |
               brew update
-              brew unlink python@2
+              brew upgrade python
-              if [ "$(python.version)" != "" ]; then
+              echo "Using `python3 --version`"
-                brew upgrade $(python.version)
-              else
-                echo "Using Python `python3 --version`"
-              fi
             displayName: "Update brew and Python"
           - bash: |
               brew install \
@@ -177,8 +171,7 @@ stages:
                 leptonica \
                 openjpeg \
                 pngquant \
-                tesseract \
+                tesseract
-                unpaper
             displayName: "Install system packages"
           - bash: |
               pip3 install --upgrade pip

docs/api.rst

@@ -20,7 +20,8 @@ and largely have the same functions.
 
     import ocrmypdf
 
-    ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
+    if __name__ == '__main__':  # To ensure correct behavior on Windows
+        ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
 With a few exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
@@ -35,8 +36,9 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
 execution. To do this, it will:
 
 - create a monitoring thread
-- create worker processes (forking itself)
+- create worker processes (on Linux, forking itself; on Windows and macOS, by
-- manage the signal flags of worker processes
+  spawning)
+- manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
 The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
@@ -47,9 +49,9 @@ There is no currently no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
-OCRmyPDF does not.
+OCRmyPDF fails for any reason.
 
 Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
@@ -57,11 +59,10 @@ mapped file fails. OCRmyPDF may use memory mapping.
 
 .. warning::
 
-    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
-    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
-    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    not take at least one of these steps, process semantics will prevent
-    of these steps, Windows process semantics will prevent OCRmyPDF from working
+    OCRmyPDF from working correctly.
-    correctly.
 
 Logging
 -------

docs/installation.rst

@@ -190,7 +190,8 @@ of ocrmypdf, and install the following dependencies:
         python3-reportlab \
         qpdf \
         tesseract-ocr \
-        zlib1g
+        zlib1g \
+        unpaper
 
 We will need a newer version of ``pip`` then was available for Ubuntu 18.04:
 

docs/release_notes.rst

@@ -12,6 +12,99 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+v11.3.3
+=======
+
+-  If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
+   conversion. (Possibly addresses #671.)
+
+v11.3.2
+=======
+
+-  Explicitly require pikepdf 2.0.0 or newer when running on Python 3.9. (There are
+   concerns about the stability of pybind11 2.5.x with Python 3.9, which is used in
+   pikepdf 1.x.)
+-  Fixed another issue related to page rotation.
+-  Fixed an issue where image marked as image masks were not properly considered
+   as optimization candidates.
+-  On some systems, unpaper seems to be unable to process the PNGs we offer it
+   as input. We now convert the input to PNM format, which unpaper always accepts.
+   Fixes #665 and #667.
+-  DPI sent to unpaper is now rounded to a more reasonable number of decimal digits.
+-  Debug and error messages from unpaper were being suppressed.
+-  Some documentation tweaks.
+
+v11.3.1
+=======
+
+-  Declare support for new versions: pdfminer.six 20201018 and pikepdf 2.x
+-  Fix warning related to ``--pdfa-image-compression`` that appears at the wrong
+   time.
+
+v11.3.0
+=======
+
+-  The "OCR" step is describing as "Image processing" in the output messages when
+   OCR is disabled, to better explain the application's behavior.
+-  Debug logs are now only created when run as a command line, and not when OCR
+   is performed for an API call. It is the calling application's responsibility
+   to set up logging.
+-  For PDFs with a low number of pages, we gathered information about the input PDF
+   in a thread rather than process (when there are more pages). When run as a
+   thread, we did not close the file handle to the working PDF, leaking one file
+   handle per call of ``ocrmypdf.ocr``.
+-  Fixed an issue where debug messages send by child worker processes did not match
+   the log settings of parent process, causing messages to be dropped. This affected
+   macOS and Windows only where the parent process is not forked.
+-  Fixed the hookspec of rasterize_pdf_page to remove default parameters that
+   were not handled in an expected way by pluggy.
+-  Fixed another issue with automatic page rotation (#658) due to the issue above.
+
+v11.2.1
+=======
+
+-  Fixed an issue where optimization of a 1-bit image with a color palette or
+   associated ICC that was optimized to JBIG2 could have its colors inverted.
+
+v11.2.0
+=======
+
+-  Fixed an issue with optimizing PNG-type images that had soft masks or image masks.
+   This is a regression introduced in (or about) v11.1.0.
+-  Improved type checking of the ``plugins`` parameter for the ``ocrmypdf.ocr``
+   API call.
+
+v11.1.2
+=======
+
+-  Fixed hOCR renderer writing the text in roughly reverse order. This should not
+   affect reasonably smart PDF readers that properly locate the position of all
+   text, but may confuse those that rely on the order of objects in the content
+   stream. (#642)
+
+v11.1.1
+=======
+
+-  We now avoid using named temporary files when using pngquant allowing containerized
+   pngquant installs to be used.
+-  Clarified an error message.
+-  Highest number of 1's in a release ever!
+
+v11.1.0
+=======
+
+-  Fixed page rotation issues: #634, #589.
+-  Fixed some cases where optimization created an invalid image such as a
+   1-bit "RGB" image: #629, #620.
+-  Page numbers are now displayed in debug logs when pages are being grafted.
+-  ocrmypdf.optimize.rewrite_png and ocrmypdf.optimize.rewrite_png_as_g4 were
+   marked deprecated. Strictly speaking these should have been internal APIs,
+   but they were never hidden.
+-  As a precaution, pikepdf mmap-based file access has been disabled due to a
+   rare race condition that causes a crash when certain objects are deallocated.
+   The problem is likely in pikepdf's dependency pybind11.
+-  Extended the example plugin to demonstrate conversion to mono.
+
 v11.0.2
 =======
 

misc/example_plugin.py

@@ -18,6 +18,25 @@
 # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 # SOFTWARE.
 
+"""
+An example of an OCRmyPDF plugin.
+
+This plugin adds two new command line arguments
+    --grayscale-ocr: converts the image to grayscale before performing OCR on it
+        (This is occasionally useful for images whose color confounds OCR. It only
+        affects the image shown to OCR. The image is not saved.)
+    --mono-page: converts pages all pages in the output file to black and white
+
+To use this from the command line:
+    ocrmypdf --plugin path/to/example_plugin.py --mono-page input.pdf output.pdf
+
+To use this as an API:
+    import ocrmypdf
+    ocrmypdf.ocr('input.pdf', 'output.pdf',
+        plugins=['path/to/example_plugin.py'], mono_page=True
+    )
+"""
+
 import logging
 
 from PIL import Image
@@ -30,6 +49,7 @@ log = logging.getLogger(__name__)
 @hookimpl
 def add_options(parser):
     parser.add_argument('--grayscale-ocr', action='store_true')
+    parser.add_argument('--mono-page', action='store_true')
 
 
 @hookimpl
@@ -52,7 +72,13 @@ def filter_ocr_image(page, image):
 
 @hookimpl
 def filter_page_image(page, image_filename):
-    output = image_filename.with_suffix('.jpg')
+    if page.options.mono_page:
-    with Image.open(image_filename) as im:
+        with Image.open(image_filename) as im:
-        im.save(output)
+            im = im.convert('1')
-    return output
+            im.save(image_filename)
+        return image_filename
+    else:
+        output = image_filename.with_suffix('.jpg')
+        with Image.open(image_filename) as im:
+            im.save(output)
+        return output

requirements/main.txt

@@ -1,12 +1,12 @@
 # requirements.txt can be used to replicate the developer's build environment
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
-cffi == 1.14.0
+cffi == 1.14.3
 coloredlogs == 14.0  # technically optional
-img2pdf == 0.3.6
+img2pdf == 0.4.0
-pdfminer.six == 20200517
+pdfminer.six == 20201018
-pikepdf == 1.16.1
+pikepdf == 2.0.0
 pluggy == 0.13.1
-Pillow == 7.1.2
+Pillow == 8.0.1
-reportlab == 3.5.42
+reportlab == 3.5.55
-tqdm == 4.46.1
+tqdm == 4.51.0

setup.py

@@ -63,7 +63,6 @@ setup(
     python_requires=' >= 3.6',
     setup_requires=[  # can be removed whenever we can drop pip 9 support
         'cffi >= 1.9.1',  # to build the leptonica module
-        'pytest-runner',  # to enable python setup.py test
         'setuptools_scm',  # so that version will work
         'setuptools_scm_git_archive',  # enable version from github tarballs
     ],
@@ -73,10 +72,11 @@ setup(
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'coloredlogs >= 14.0',  # strictly optional
         'img2pdf >= 0.3.0, < 0.5',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20191110, != 20200720, <= 20200726',
+        'pdfminer.six >= 20191110, != 20200720, <= 20201018',
-        'pikepdf >= 1.14.0, < 2',
+        "pikepdf >= 1.14.0, < 3 ; python_version < '3.9'",
+        "pikepdf >= 2.0.0 ; python_version >= '3.9'",
         'Pillow >= 7.0.0',
-        'pluggy >= 0.13.0',
+        'pluggy >= 0.13.0, < 1.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling
         'tqdm >= 4',
     ],

src/ocrmypdf/_concurrent.py

@@ -49,7 +49,7 @@ def process_sigbus(*args):
     raise InputFileError("A worker process lost access to an input file")
 
 
-def process_init(queue, user_init):
+def process_init(queue, user_init, loglevel):
     """Initialize a process pool worker"""
 
     # Ignore SIGINT (our parent process will kill us gracefully)
@@ -62,6 +62,7 @@ def process_init(queue, user_init):
     # Reconfigure the root logger for this process to send all messages to a queue
     h = logging.handlers.QueueHandler(queue)
     root = logging.getLogger()
+    root.setLevel(loglevel)
     root.handlers = []
     root.addHandler(h)
 
@@ -69,7 +70,7 @@ def process_init(queue, user_init):
         user_init()
 
 
-def thread_init(_queue, user_init):
+def thread_init(_queue, user_init, _loglevel):
     # As a thread, block SIGBUS so the main thread deals with it...
     if hasattr(signal, 'SIGBUS'):
         signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGBUS})
@@ -102,7 +103,7 @@ def exec_progress_pool(
         pool = pool_class(
             processes=max_workers,
             initializer=initializer,
-            initargs=(log_queue, task_initializer),
+            initargs=(log_queue, task_initializer, logging.getLogger("").level),
         )
         try:
             results = pool.imap_unordered(task, task_arguments)

src/ocrmypdf/_exec/ghostscript.py

@@ -81,8 +81,8 @@ def rasterize_pdf(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int = 1,
-    page_dpi: Resolution = None,
+    page_dpi: Optional[Resolution] = None,
-    rotation: int = None,
+    rotation: Optional[int] = None,
     filter_vector: bool = False,
 ):
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units."""
@@ -122,8 +122,6 @@ def rasterize_pdf(
         stderr = p.stderr.decode(errors='replace')
         if _gs_error_reported(stderr):
             log.error(stderr)
-        elif stderr:
-            log.debug(stderr)
 
     with Image.open(BytesIO(p.stdout)) as im:
         if rotation is not None:

src/ocrmypdf/_exec/pngquant.py

@@ -7,7 +7,11 @@
 
 """Interface to pngquant executable"""
 
+from contextlib import contextmanager
+from io import BytesIO
 from os import fspath
+from pathlib import Path
+from subprocess import PIPE
 from tempfile import NamedTemporaryFile
 
 from PIL import Image
@@ -28,34 +32,32 @@ def available():
     return True
 
 
-def quantize(input_file, output_file, quality_min, quality_max):
+@contextmanager
-    input_file = fspath(input_file)
+def input_as_png(input_file: Path):
-    output_file = fspath(output_file)
+    if not input_file.name.endswith('.png'):
-    if input_file.endswith('.jpg'):
+        with Image.open(input_file) as im:
-        with Image.open(input_file) as im, NamedTemporaryFile(suffix='.png') as tmp:
+            bio = BytesIO()
-            im.save(tmp)
+            im.save(bio, format='png')
-            args = [
+            bio.seek(0)
-                'pngquant',
+            yield bio
-                '--force',
-                '--skip-if-larger',
-                '--output',
-                output_file,
-                '--quality',
-                f'{quality_min}-{quality_max}',
-                '--',
-                tmp.name,
-            ]
-            run(args)
     else:
+        with open(input_file, 'rb') as f:
+            yield f
+
+
+def quantize(input_file: Path, output_file: Path, quality_min: int, quality_max: int):
+    with input_as_png(input_file) as input_stream:
         args = [
             'pngquant',
             '--force',
             '--skip-if-larger',
-            '--output',
-            output_file,
             '--quality',
             f'{quality_min}-{quality_max}',
-            '--',
+            '--',  # pngquant: stop processing arguments
-            input_file,
+            '-',  # pngquant: stream input and output
         ]
-        run(args)
+        result = run(args, stdin=input_stream, stdout=PIPE, stderr=PIPE, check=False)
+
+    if result.returncode == 0:
+        # input_file could be the same as output_file, so we defer the write
+        output_file.write_bytes(result.stdout)

src/ocrmypdf/_exec/tesseract.py

@@ -99,9 +99,7 @@ def get_languages():
 
     args_tess = ['tesseract', '--list-langs']
     try:
-        proc = run(
+        proc = run(args_tess, text=True, stdout=PIPE, stderr=STDOUT, check=True)
-            args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
-        )
         output = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(lang_error(e.output)) from e

src/ocrmypdf/_exec/unpaper.py

@@ -55,21 +55,21 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
                 "Failed to convert image to a supported format."
             ) from e
 
-        if im_modified or input_file.suffix != '.png':
+        if im_modified or input_file.suffix != '.pnm':
-            input_png = tmpdir / 'input.png'
+            input_pnm = tmpdir / 'input.pnm'
-            im.save(input_png, format='PNG', compress_level=1)
+            im.save(input_pnm, format='PPM')
         else:
             # No changes, PNG input, just use the file we already have
-            input_png = input_file
+            input_pnm = input_file
         output_pnm = tmpdir / f'output{suffix}'
-    return input_png, output_pnm
+    return input_pnm, output_pnm
 
 
 def run(input_file, output_file, dpi, mode_args):
-    args_unpaper = ['unpaper', '-v', '--dpi', str(dpi)] + mode_args
+    args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
 
     with TemporaryDirectory() as tmpdir:
-        input_png, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
+        input_pnm, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
 
         # To prevent any shenanigans from accepting arbitrary parameters in
         # --unpaper-args, we:
@@ -78,36 +78,30 @@ def run(input_file, output_file, dpi, mode_args):
         # 3) append absolute paths for the input and output file
         # This should ensure that a user cannot clobber some other file with
         # their unpaper arguments (whether intentionally or otherwise)
-        args_unpaper.extend([os.fspath(input_png), os.fspath(output_pnm)])
+        args_unpaper.extend([os.fspath(input_pnm), os.fspath(output_pnm)])
+        external_run(
+            args_unpaper,
+            close_fds=True,
+            check=True,
+            stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+            stdout=PIPE,  # and cannot send file output to stdout
+            cwd=tmpdir,
+            logs_errors_to_stdout=True,
+        )
         try:
-            proc = external_run(
+            with Image.open(output_pnm) as imout:
-                args_unpaper,
+                imout.save(output_file, dpi=(dpi, dpi))
-                check=True,
+        except (FileNotFoundError, OSError):
-                close_fds=True,
+            raise SubprocessOutputError(
-                universal_newlines=True,
+                "unpaper: failed to produce the expected output file. "
-                stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+                + " Called with: "
-                cwd=tmpdir,  # and cannot send file output to stdout
+                + str(args_unpaper)
-                stdout=PIPE,
+            ) from None
-            )
-        except CalledProcessError as e:
-            log.debug(e.stderr)
-            raise e from e
-        else:
-            log.debug(proc.stderr)
-            try:
-                with Image.open(output_pnm) as imout:
-                    imout.save(output_file, dpi=(dpi, dpi))
-            except (FileNotFoundError, OSError):
-                raise SubprocessOutputError(
-                    "unpaper: failed to produce the expected output file. "
-                    + " Called with: "
-                    + str(args_unpaper)
-                ) from None
 
 
 def validate_custom_args(args: str):
     unpaper_args = shlex.split(args)
-    if any('/' in arg for arg in unpaper_args):
+    if any(('/' in arg or arg == '.' or arg == '..') for arg in unpaper_args):
         raise ValueError('No filenames allowed in --unpaper-args')
     return unpaper_args
 

src/ocrmypdf/_graft.py

@@ -125,34 +125,38 @@ class OcrGrafter:
                 del self.pdf_base.pages[-1]
             emplaced_page = True
 
+        # Calculate if the text is misaligned compared to the content
         if emplaced_page:
             content_rotation = autorotate_correction
         text_rotation = autorotate_correction
         text_misaligned = (text_rotation - content_rotation) % 360
         log.debug(
-            f"Rotations for page {pageno}: [text, auto, misalign, content] = "
+            f"Text rotation: (text, autorotate, content) -> text misalignment = "
-            f"{text_rotation}, {autorotate_correction}, "
+            f"({text_rotation}, {autorotate_correction}, {content_rotation}) -> {text_misaligned}"
-            f"{text_misaligned}, {content_rotation}"
         )
 
         if textpdf and self.font:
-            # Graft the text layer onto this page, whether new or old
+            # Graft the text layer onto this page, whether new or old, possibly
+            # rotating the text layer by the amount is misaligned.
             strip_old = self.context.options.redo_ocr
             self._graft_text_layer(
                 page_num=pageno + 1,
                 textpdf=textpdf,
                 font=self.font,
                 font_key=self.font_key,
-                rotation=text_misaligned,
+                text_rotation=text_misaligned,
                 procset=self.procset,
                 strip_old_text=strip_old,
             )
 
-        # Correct the rotation if applicable
+        # Correct the overall page rotation if needed, now that the text and content
-        self.pdf_base.pages[pageno].Rotate = (
+        # are aligned
-            content_rotation - autorotate_correction
+        page_rotation = (content_rotation - autorotate_correction) % 360
-        ) % 360
+        self.pdf_base.pages[pageno].Rotate = page_rotation
-
+        log.debug(
+            f"Page rotation: (content, auto) -> page = "
+            f"({content_rotation}, {autorotate_correction}) -> {page_rotation}"
+        )
         if self.emplacements % MAX_REPLACE_PAGES == 0:
             self.save_and_reload()
 
@@ -226,7 +230,7 @@ class OcrGrafter:
         font: pikepdf.Object,
         font_key: pikepdf.Object,
         procset: pikepdf.Object,
-        rotation: int,
+        text_rotation: int,
         strip_old_text: bool,
     ):
         """Insert the text layer from text page 0 on to pdf_base at page_num"""
@@ -256,13 +260,13 @@ class OcrGrafter:
             corner = pikepdf.PdfMatrix().translated(mediabox[0], mediabox[1])
             # -rotation because the input is a clockwise angle and this formula
             # uses CCW
-            rotation = -rotation % 360
+            text_rotation = -text_rotation % 360
-            rotate = pikepdf.PdfMatrix().rotated(rotation)
+            rotate = pikepdf.PdfMatrix().rotated(text_rotation)
 
             # Because of rounding of DPI, we might get a text layer that is not
             # identically sized to the target page. Scale to adjust. Normally this
             # is within 0.998.
-            if rotation in (90, 270):
+            if text_rotation in (90, 270):
                 wt, ht = ht, wt
             scale_x = wp / wt
             scale_y = hp / ht

src/ocrmypdf/_pipeline.py

@@ -332,8 +332,10 @@ def rasterize_preview(input_file: Path, page_context: PageContext):
         output_file=output_file,
         raster_device='jpeggray',
         raster_dpi=canvas_dpi,
-        page_dpi=page_dpi,
         pageno=page_context.pageinfo.pageno + 1,
+        page_dpi=page_dpi,
+        rotation=0,
+        filter_vector=False,
     )
     return output_file
 
@@ -433,7 +435,7 @@ def rasterize(
 
     device = colorspaces[device_idx]
 
-    log.debug(f"Rasterize with {device}")
+    log.debug(f"Rasterize with {device}, rotation {correction}")
 
     # Produce the page image with square resolution or else deskew and OCR
     # will not work properly.

src/ocrmypdf/_sync.py

@@ -209,9 +209,10 @@ def exec_page_sync(page_context: PageContext):
     if options.pdf_renderer == 'hocr':
         (hocr_out, text_out) = ocr_engine_hocr(ocr_image_out, page_context)
         ocr_out = render_hocr_page(hocr_out, page_context)
-
+    elif options.pdf_renderer == 'sandwich':
-    if options.pdf_renderer == 'sandwich':
         (ocr_out, text_out) = ocr_engine_textonly_pdf(ocr_image_out, page_context)
+    else:
+        raise NotImplementedError(f"pdf_renderer {options.pdf_renderer}")
 
     return PageResult(
         pageno=page_context.pageno,
@@ -244,7 +245,8 @@ def exec_concurrent(context: PdfContext):
     """Execute the pipeline concurrently"""
 
     # Run exec_page_sync on every page context
-    max_workers = min(len(context.pdfinfo), context.options.jobs)
+    options = context.options
+    max_workers = min(len(context.pdfinfo), options.jobs)
     if max_workers > 1:
         log.info("Start processing %d pages concurrently", max_workers)
 
@@ -252,25 +254,29 @@ def exec_concurrent(context: PdfContext):
     ocrgraft = OcrGrafter(context)
 
     def update_page(result: PageResult, pbar):
-        sidecars[result.pageno] = result.text
+        try:
-        pbar.update()
+            tls.pageno = result.pageno + 1
-        ocrgraft.graft_page(
+            sidecars[result.pageno] = result.text
-            pageno=result.pageno,
+            pbar.update()
-            image=result.pdf_page_from_image,
+            ocrgraft.graft_page(
-            textpdf=result.ocr,
+                pageno=result.pageno,
-            autorotate_correction=result.orientation_correction,
+                image=result.pdf_page_from_image,
-        )
+                textpdf=result.ocr,
-        pbar.update()
+                autorotate_correction=result.orientation_correction,
+            )
+            pbar.update()
+        finally:
+            tls.pageno = None
 
     exec_progress_pool(
-        use_threads=context.options.use_threads,
+        use_threads=options.use_threads,
         max_workers=max_workers,
         tqdm_kwargs=dict(
             total=(2 * len(context.pdfinfo)),
-            desc='OCR',
+            desc='OCR' if options.tesseract_timeout > 0 else 'Image processing',
             unit='page',
             unit_scale=0.5,
-            disable=not context.options.progress_bar,
+            disable=not options.progress_bar,
         ),
         task_initializer=partial(worker_init, PIL.Image.MAX_IMAGE_PIXELS),
         task=exec_page_sync,
@@ -279,10 +285,10 @@ def exec_concurrent(context: PdfContext):
     )
 
     # Output sidecar text
-    if context.options.sidecar:
+    if options.sidecar:
         text = merge_sidecars(sidecars, context)
         # Copy text file to destination
-        copy_final(text, context.options.sidecar, context)
+        copy_final(text, options.sidecar, context)
 
     # Merge layers to one single pdf
     pdf = ocrgraft.finalize()
@@ -292,7 +298,7 @@ def exec_concurrent(context: PdfContext):
     pdf = post_process(pdf, context)
 
     # Copy PDF file to destination
-    copy_final(pdf, context.options.output_file, context)
+    copy_final(pdf, options.output_file, context)
 
 
 class NeverRaise(Exception):
@@ -301,7 +307,14 @@ class NeverRaise(Exception):
     pass  # pylint: disable=unnecessary-pass
 
 
-def configure_debug_logging(log_filename, prefix=''):
+def configure_debug_logging(log_filename, prefix: str = ''):
+    """
+    Create a debug log file at a specified location.
+
+    Arguments:
+        log_filename: Where to the put the log file.
+        prefix: The logging domain prefix that should be sent to the log.
+    """
     log_file_handler = logging.FileHandler(log_filename, delay=True)
     log_file_handler.setLevel(logging.DEBUG)
     formatter = logging.Formatter(
@@ -324,9 +337,12 @@ def run_pipeline(options, *, plugin_manager, api=False):
 
     work_folder = Path(mkdtemp(prefix="com.github.ocrmypdf."))
     debug_log_handler = None
-    if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
+    if (
-        'PYTEST_CURRENT_TEST', ''
+        (options.keep_temporary_files or options.verbose >= 1)
+        and not os.environ.get('PYTEST_CURRENT_TEST', '')
+        and not api
     ):
+        # Debug log for command line interface only with verbose output
         debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
 
     pikepdf_enable_mmap()

src/ocrmypdf/_validation.py

@@ -213,12 +213,12 @@ def check_options_optimizing(options):
 
 
 def check_options_advanced(options):
-    if options.pdfa_image_compression != 'auto' and options.output_type.startswith(
+    if options.pdfa_image_compression != 'auto' and not options.output_type.startswith(
         'pdfa'
     ):
         log.warning(
-            "--pdfa-image-compression argument has no effect when "
+            "--pdfa-image-compression argument only applies when "
-            "--output-type is not 'pdfa', 'pdfa-1', or 'pdfa-2'"
+            "--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
         )
 
 
@@ -308,17 +308,6 @@ def check_closed_streams(options):  # pragma: no cover
     return True
 
 
-def log_page_orientations(pdfinfo):
-    direction = {0: 'n', 90: 'e', 180: 's', 270: 'w'}
-    orientations = []
-    for n, page in enumerate(pdfinfo):
-        angle = page.rotation or 0
-        if angle != 0:
-            orientations.append('{0}{1}'.format(n + 1, direction.get(angle, '')))
-    if orientations:
-        log.info('Page orientations detected: %s', ' '.join(orientations))
-
-
 def create_input_file(options, work_folder: Path) -> Tuple[Path, str]:
     if options.input_file == '-':
         # stdin

src/ocrmypdf/api.py

@@ -10,7 +10,8 @@ import os
 import sys
 from enum import IntEnum
 from pathlib import Path
-from typing import BinaryIO, Iterable, Union
+from typing import AnyStr, BinaryIO, Iterable, Optional, Union
+from warnings import warn
 
 from ocrmypdf._logging import PageNumberFilter, TqdmConsole
 from ocrmypdf._plugin_manager import get_plugin_manager
@@ -25,7 +26,8 @@ except ModuleNotFoundError:
     coloredlogs = None
 
 
-PathOrIO = Union[BinaryIO, os.PathLike, str, bytes]
+StrPath = Union[os.PathLike, AnyStr]
+PathOrIO = Union[BinaryIO, StrPath]
 
 
 class Verbosity(IntEnum):
@@ -44,16 +46,28 @@ def configure_logging(
 ):
     """Set up logging.
 
-    Library users may wish to use this function if they want their log output to be
+    Before calling :func:`ocrmypdf.ocr()`, you can use this function to
-    similar to ocrmypdf command line interface. If not used, the external application
+    configure logging, if you want ocrmypdf's output to look like the ocrmypdf
-    should configure logging on its own.
+    command line interface. It will register log handlers, log filters, and
+    formatters, configure color logging to standard error, and adjust the log
+    levels of third party libraries. Details of this are fine-tuned and subject
+    to change. The ``verbosity`` argument is equivalent to the argument
+    ``--verbose`` and applies those settings.
 
-    ocrmypdf will perform all of its logging under the ``"ocrmypdf"`` logging namespace.
+    If this function is not called, ocrmypdf will not configure logging, and it
-    In addition, ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library
+    is up to the caller of ``ocrmypdf.ocr()`` to set up logging as it wishes using
-    user may wish to configure both; note that pdfminer is extremely chatty at the log
+    the Python standard library's logging module. If this function is called,
-    level ``logging.INFO``.
+    the caller may of course make further adjustments to logging.
 
-    Library users may perform additional configuration afterwards.
+    Regardless of whether this function is called, ocrmypdf will perform all of
+    its logging under the ``"ocrmypdf"`` logging namespace. In addition,
+    ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library user
+    may wish to configure both; note that pdfminer is extremely chatty at the
+    log level ``logging.INFO``.
+
+    This function does not set up the ``debug.log`` log file that the command
+    line interface does at certain verbosity levels. Applications should configure
+    their own debug logging.
 
     Args:
         verbosity (Verbosity): Verbosity level.
@@ -189,7 +203,7 @@ def ocr(  # pylint: disable=unused-argument
     language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
-    sidecar: os.PathLike = None,
+    sidecar: Optional[StrPath] = None,
     jobs: int = None,
     use_threads: bool = None,
     title: str = None,
@@ -226,7 +240,7 @@ def ocr(  # pylint: disable=unused-argument
     user_words: os.PathLike = None,
     user_patterns: os.PathLike = None,
     fast_web_view: float = None,
-    plugins: Iterable[str] = None,
+    plugins: Iterable[StrPath] = None,
     keep_temporary_files: bool = None,
     progress_bar: bool = None,
     **kwargs,
@@ -248,7 +262,7 @@ def ocr(  # pylint: disable=unused-argument
             read.
         output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
             interpreted as file system path to the output file. If the object
-            appears to be a writable stream (with methods such as ``.read()`` and
+            appears to be a writable stream (with methods such as ``.write()`` and
             ``.seek()``), the output will be written to this stream. If
             ``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
             (provided that standard output does not seem to be a terminal device).
@@ -280,18 +294,23 @@ def ocr(  # pylint: disable=unused-argument
     """
     if not plugins:
         plugins = []
+    elif isinstance(plugins, (str, Path)):
+        plugins = [plugins]
     else:
         plugins = list(plugins)
 
-    parser = get_parser()
+    # No new variable names should be assigned until these two steps are run
-    _plugin_manager = get_plugin_manager(plugins)
+    create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
-    _plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
-
-    create_options_kwargs = {
-        k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
-    }
     create_options_kwargs.update(kwargs)
 
+    parser = get_parser()
+    create_options_kwargs['parser'] = parser
+    plugin_manager = get_plugin_manager(plugins)
+    plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
+
+    if 'verbose' in kwargs:
+        warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
+
     options = create_options(**create_options_kwargs)
-    check_options(options, _plugin_manager)
+    check_options(options, plugin_manager)
-    return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
+    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)

src/ocrmypdf/builtin_plugins/ghostscript.py

@@ -39,13 +39,11 @@ def check_options(options):
     if gs_version < '9.20' and options.output_type != 'pdf' and not is_latin:
         # https://bugs.ghostscript.com/show_bug.cgi?id=696874
         # Ghostscript < 9.20 fails to encode multibyte characters properly
-        msg = (
+        log.warning(
-            "The installed version of Ghostscript does not work correctly "
+            f"The installed version of Ghostscript ({gs_version}) does not work "
-            "with the OCR languages you specified. Use --output-type pdf or "
+            "correctly with the OCR languages you specified. Use --output-type pdf or "
             "upgrade to Ghostscript 9.20 or later to avoid this issue."
         )
-        msg += f"Found Ghostscript {gs_version}"
-        log.warning(msg)
 
     if options.output_type == 'pdfa':
         options.output_type = 'pdfa-2'
@@ -63,9 +61,9 @@ def rasterize_pdf_page(
     raster_device,
     raster_dpi,
     pageno,
-    page_dpi=None,
+    page_dpi,
-    rotation=None,
+    rotation,
-    filter_vector=False,
+    filter_vector,
 ):
     ghostscript.rasterize_pdf(
         input_file,

src/ocrmypdf/builtin_plugins/tesseract_ocr.py

@@ -121,9 +121,7 @@ def validate(pdfinfo, options):
         os.environ['OMP_THREAD_LIMIT'] = str(tess_threads)
     else:
         tess_threads = int(os.environ['OMP_THREAD_LIMIT'])
-
+    log.debug("Using Tesseract OpenMP thread limit %d", tess_threads)
-    if tess_threads > 1:
-        log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
 
 
 class TesseractOcrEngine(OcrEngine):

src/ocrmypdf/helpers.py

@@ -213,11 +213,15 @@ def clamp(n: T, smallest: T, largest: T) -> T:
 
 
 def pikepdf_enable_mmap():
-    try:
+    # try:
-        if pikepdf._qpdf.set_access_default_mmap(True):
+    #     if pikepdf._qpdf.set_access_default_mmap(True):
-            log.debug("pikepdf mmap enabled")
+    #         log.debug("pikepdf mmap enabled")
-    except AttributeError:
+    # except AttributeError:
-        log.debug("pikepdf mmap not available")
+    #     log.debug("pikepdf mmap not available")
+    # We found a race condition probably related to pybind issue #2252 that can
+    # cause a crash. For now, disable pikepdf mmap to be on the safe side.
+    log.debug("pikepdf mmap disabled")
+    return
 
 
 def deprecated(func):

src/ocrmypdf/hocrtransform.py

@@ -167,7 +167,10 @@ class HocrTransform:
     def topdown_position(self, element):
         pxl_line_coords = self.element_coordinates(element)
         line_box = self.pt_from_pixel(pxl_line_coords)
-        return -line_box.y2
+        # Coordinates here are still in the hocr coordinate system, so 0 on the y axis
+        # is the top of the page and increasing values of y will move towards the
+        # bottom of the page.
+        return line_box.y2
 
     def to_pdf(
         self,

src/ocrmypdf/leptonica.py

@@ -58,6 +58,24 @@ if not _libpath:
         ---------------------------------------------------------------------
         """
     )
+if os.name == 'nt':  
+    # On Windows, recent versions of libpng require zlib. We have to make sure 
+    # the zlib version being loaded is the same one that libpng was built with. 
+    # This tries to import zlib from Tesseract's installation folder, falling back
+    # to find_library() if liblept is being loaded from somewhere else.
+    # Loading zlib from other places could cause a version mismatch
+    _zlib_path = os.path.join(os.path.dirname(_libpath), 'zlib1.dll')
+    if not os.path.exists(_zlib_path):
+        _zlib_path = find_library('zlib')
+    try:
+        zlib = ffi.dlopen(_zlib_path)
+    except ffi.error as e:
+        raise MissingDependencyError(
+            """
+            Could not load the zlib library. It could be that Tesseract is not installed properly,
+            we can't find the installation on your system PATH environment variable.
+            """
+        ) from e
 try:
     lept = ffi.dlopen(_libpath)
     lept.setMsgSeverity(lept.L_SEVERITY_WARNING)

src/ocrmypdf/optimize.py

@@ -10,6 +10,7 @@ import sys
 import tempfile
 from collections import defaultdict
 from functools import partial
+from io import BytesIO
 from os import fspath
 from pathlib import Path
 from typing import (
@@ -27,6 +28,7 @@ from typing import (
     Union,
 )
 
+import img2pdf
 import pikepdf
 from pikepdf import Dictionary, Name, Object, Pdf, PdfImage
 from PIL import Image
@@ -37,7 +39,7 @@ from ocrmypdf._concurrent import exec_progress_pool
 from ocrmypdf._exec import jbig2enc, pngquant
 from ocrmypdf._jobcontext import PdfContext
 from ocrmypdf.exceptions import OutputFileAccessError
-from ocrmypdf.helpers import safe_symlink
+from ocrmypdf.helpers import deprecated, safe_symlink
 
 log = logging.getLogger(__name__)
 
@@ -75,23 +77,26 @@ def extract_image_filter(
     if image.Subtype != Name.Image:
         return None
     if image.Length < 100:
-        log.debug("Skipping small image, xref %s", xref)
+        log.debug(f"Skipping small image, xref {xref}")
         return None
 
     pim = PdfImage(image)
 
     if len(pim.filter_decodeparms) > 1:
-        log.debug("Skipping multiply filtered, xref %s", xref)
+        log.debug(f"Skipping multiply filtered image, xref {xref}")
         return None
     filtdp = pim.filter_decodeparms[0]
 
     if pim.bits_per_component > 8:
+        log.debug(f"Skipping wide gamut image, xref {xref}")
         return None  # Don't mess with wide gamut images
 
     if filtdp[0] == Name.JPXDecode:
+        log.debug(f"Skipping JPEG2000 iamge, xref {xref}")
         return None  # Don't do JPEG2000
 
     if Name.Decode in image:
+        log.debug(f"Skipping image with Decode table, xref {xref}")
         return None  # Don't mess with custom Decode tables
 
     return pim, filtdp
@@ -110,14 +115,29 @@ def extract_image_jbig2(
         and filtdp[0] != Name.JBIG2Decode
         and jbig2enc.available()
     ):
-        try:
+        # Save any colorspace associated with the image, so that we
-            imgname = root / f'{xref:08d}'
+        # will export a pure 1-bit PNG with no palette or ICC profile.
-            with imgname.open('wb') as f:
+        # Showing the palette or ICC to jbig2enc will cause it to perform
-                ext = pim.extract_to(stream=f)
+        # colorspace transform to 1bpp, which will conflict the palette or
-            imgname.rename(imgname.with_suffix(ext))
+        # ICC if it exists.
-        except pikepdf.UnsupportedImageTypeError:
+        colorspace = pim.obj.get(pikepdf.Name.ColorSpace, None)
-            return None
+        if colorspace is not None or pim.image_mask:
-        return XrefExt(xref, ext)
+            try:
+                # Set to DeviceGray temporarily; we already in 1 bpc.
+                pim.obj.ColorSpace = pikepdf.Name.DeviceGray
+                imgname = root / f'{xref:08d}'
+                with imgname.open('wb') as f:
+                    ext = pim.extract_to(stream=f)
+                imgname.rename(imgname.with_suffix(ext))
+            except pikepdf.UnsupportedImageTypeError:
+                return None
+            finally:
+                # Restore image colorspace after temporarily setting it to DeviceGray
+                if colorspace is not None:
+                    pim.obj.ColorSpace = colorspace
+                else:
+                    del pim.obj.ColorSpace
+            return XrefExt(xref, ext)
     return None
 
 
@@ -227,7 +247,9 @@ def extract_images(
                 # Ignore soft masks
                 smask_xref = Xref(image.SMask.objgen[0])
                 exclude_xrefs.add(smask_xref)
+                log.debug(f"Skipping image {smask_xref} because it is an SMask")
             include_xrefs.add(xref)
+            log.debug(f"Treating {xref} as an optimization candidate")
             if xref not in pageno_for_xref:
                 pageno_for_xref[xref] = pageno
 
@@ -393,6 +415,47 @@ def transcode_jpegs(pike: Pdf, jpegs: Sequence[Xref], root: Path, options) -> No
         im_obj.write(compdata.read(), filter=Name.DCTDecode)
 
 
+def _transcode_png(pike: Pdf, filename: Path, xref: Xref) -> bool:
+    output = filename.with_suffix('.png.pdf')
+    with output.open('wb') as f:
+        img2pdf.convert(fspath(filename), outputstream=f)
+
+    with pikepdf.open(output) as pdf_image:
+        foreign_image = next(pdf_image.pages[0].images.values())
+        local_image = pike.copy_foreign(foreign_image)
+
+        im_obj = pike.get_object(xref, 0)
+        im_obj.write(
+            local_image.read_raw_bytes(),
+            filter=local_image.Filter,
+            decode_parms=local_image.DecodeParms,
+        )
+
+        # Don't copy keys from the new image...
+        del_keys = set(im_obj.keys()) - set(local_image.keys())
+        # ...except for the keep_fields, which are essential to displaying
+        # the image correctly and preserving its metadata. (/Decode arrays
+        # and /SMaskInData are implicitly discarded prior to this point.)
+        keep_fields = {
+            '/ID',
+            '/Intent',
+            '/Interpolate',
+            '/Mask',
+            '/Metadata',
+            '/OC',
+            '/OPI',
+            '/SMask',
+            '/StructParent',
+        }
+        del_keys -= keep_fields
+        for key in local_image.keys():
+            if key != Name.Length and str(key) not in keep_fields:
+                im_obj[key] = local_image[key]
+        for key in del_keys:
+            del im_obj[key]
+    return True
+
+
 def transcode_pngs(
     pike: Pdf,
     images: Sequence[Xref],
@@ -435,34 +498,11 @@ def transcode_pngs(
         )
 
     for xref in modified:
-        im_obj = pike.get_object(xref, 0)
+        filename = png_name(root, xref)
-        try:
+        _transcode_png(pike, filename, xref)
-            pix = leptonica.Pix.open(png_name(root, xref))
-            if pix.mode == '1':
-                compdata = pix.generate_pdf_ci_data(leptonica.lept.L_G4_ENCODE, 0)
-            else:
-                compdata = leptonica.CompressedData.open(png_name(root, xref))
-        except leptonica.LeptonicaError as e:
-            # Most likely this means file not found, i.e. quantize did not
-            # produce an improved version
-            log.error(e)
-            continue
-
-        # If re-coded image is larger don't use it - we test here because
-        # pngquant knows the size of the temporary output file but not the actual
-        # object in the PDF
-        if len(compdata) > int(im_obj.stream_dict.Length):
-            log.debug(
-                f"pngquant: pngquant did not improve over original image "
-                f"{len(compdata)} > {int(im_obj.stream_dict.Length)}"
-            )
-            continue
-        if compdata.type == leptonica.lept.L_FLATE_ENCODE:
-            rewrite_png(pike, im_obj, compdata)
-        elif compdata.type == leptonica.lept.L_G4_ENCODE:
-            rewrite_png_as_g4(pike, im_obj, compdata)
 
 
+@deprecated
 def rewrite_png_as_g4(pike: Pdf, im_obj: Object, compdata) -> None:
     im_obj.BitsPerComponent = 1
     im_obj.Width = compdata.w
@@ -483,6 +523,7 @@ def rewrite_png_as_g4(pike: Pdf, im_obj: Object, compdata) -> None:
     return
 
 
+@deprecated
 def rewrite_png(pike: Pdf, im_obj: Object, compdata) -> None:
     # When a PNG is inserted into a PDF, we more or less copy the IDAT section from
     # the PDF and transfer the rest of the PNG headers to PDF image metadata.
@@ -576,7 +617,7 @@ def optimize(input_file: Path, output_file: Path, context, save_settings) -> Non
         )
     ratio = input_size / output_size
     savings = 1 - output_size / input_size
-    log.info(f"Optimize ratio: {ratio:.2f} savings: {(100 * savings):.1f}%")
+    log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
 
     if savings < 0:
         log.info("Image optimization did not improve the file - discarded")

src/ocrmypdf/pdfinfo/info.py

@@ -630,6 +630,9 @@ worker_pdf = None
 def _pdf_pageinfo_sync_init(infile):
     global worker_pdf  # pylint: disable=global-statement
     pikepdf_enable_mmap()
+    # If this function is called as a thread initializer, we need a messy hack
+    # to close worker_pdf. If called as a process, it will be released when the
+    # process is terminated.
     worker_pdf = pikepdf.open(infile)
 
 
@@ -643,6 +646,7 @@ def _pdf_pageinfo_sync(args):
 def _pdf_pageinfo_concurrent(
     pdf, infile, progbar, max_workers, check_pages, detailed_analysis=False
 ):
+    global worker_pdf  # pylint: disable=global-statement
     pages = [None] * len(pdf.pages)
 
     def update_pageinfo(result, pbar):
@@ -663,17 +667,23 @@ def _pdf_pageinfo_concurrent(
         # a separate process.
         use_threads = True
 
-    exec_progress_pool(
+    try:
-        use_threads=use_threads,
+        exec_progress_pool(
-        max_workers=n_workers,
+            use_threads=use_threads,
-        tqdm_kwargs=dict(
+            max_workers=n_workers,
-            total=total, desc="Scanning contents", unit='page', disable=not progbar
+            tqdm_kwargs=dict(
-        ),
+                total=total, desc="Scanning contents", unit='page', disable=not progbar
-        task_initializer=partial(_pdf_pageinfo_sync_init, infile),
+            ),
-        task=_pdf_pageinfo_sync,
+            task_initializer=partial(_pdf_pageinfo_sync_init, infile),
-        task_arguments=contexts,
+            task=_pdf_pageinfo_sync,
-        task_finished=update_pageinfo,
+            task_arguments=contexts,
-    )
+            task_finished=update_pageinfo,
+        )
+    finally:
+        if worker_pdf and use_threads:
+            assert n_workers == 1, "Should have only one worker when threaded"
+            # This is messy, but if we ran in thread, close worker_pdf
+            worker_pdf.close()
     return pages
 
 
@@ -808,12 +818,12 @@ class PdfInfo:
                 check_pages=check_pages,
                 detailed_analysis=detailed_analysis,
             )
-            self._needs_rendering = pdf.root.get('/NeedsRendering', False)
+            self._needs_rendering = pdf.Root.get('/NeedsRendering', False)
             self._has_acroform = False
-            if '/AcroForm' in pdf.root:
+            if '/AcroForm' in pdf.Root:
-                if len(pdf.root.AcroForm.get('/Fields', [])) > 0:
+                if len(pdf.Root.AcroForm.get('/Fields', [])) > 0:
                     self._has_acroform = True
-                elif '/XFA' in pdf.root.AcroForm:
+                elif '/XFA' in pdf.Root.AcroForm:
                     self._has_acroform = True
 
     @property

src/ocrmypdf/pluginspec.py

@@ -89,9 +89,9 @@ def rasterize_pdf_page(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int,
-    page_dpi: Optional[Resolution] = None,
+    page_dpi: Optional[Resolution],
-    rotation: Optional[int] = None,
+    rotation: Optional[int],
-    filter_vector: bool = False,
+    filter_vector: bool,
 ) -> Path:
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units.
 
@@ -154,11 +154,29 @@ def filter_page_image(page: 'PageContext', image_filename: Path) -> Path:
     produced for a given page, this function will not be called. This is not
     the image that will be shown to OCR.
 
-    ocrmypdf will create the PDF page based on the image format used. If you
+    If the function does not want to modify the image, it should return
+    ``image_filename``. The hook may overwrite ``image_filename`` with a new file.
+
+    The output image should preserve the same physical unit dimensions, that is
+    (width * dpi_x, height * dpi_y). That is, if the image is resized, the DPI
+    must be adjusted by the reciprocal. If this is not preserved, the PDF page
+    will be resized and the OCR layer misaligned. OCRmyPDF does not nothing
+    to enforce these constraints; it is up to the plugin to do sensible things.
+
+    OCRmyPDF will create the PDF page based on the image format used. If you
     convert the image to a JPEG, the output page will be created as a JPEG, etc.
-    Note that the ocrmypdf image optimization stage may ultimately chose a
+    If you change the colorspace, that change will be kept. Note that the
+    OCRmyPDF image optimization stage, if enabled, may ultimately chose a
     different format.
 
+    If the return value is a file that does not exist, ``FileNotFoundError``
+    will occur. The return value should be a path to a file in the same folder
+    as ``image_filename``.
+
+    Implementation detail: If the value returned is falsy, OCRmyPDF will ignore
+    the return value and assume the input file was unmodified. This is deprecated.
+    To leave the image unmodified, ``image_filename`` should be returned.
+
     Note:
         This hook will be called from child processes. Modifying global state
         will not affect the main process or other child processes.

src/ocrmypdf/subprocess.py

@@ -25,13 +25,21 @@ from ocrmypdf.exceptions import MissingDependencyError
 log = logging.getLogger(__name__)
 
 
-def run(args, *, env=None, **kwargs):
+def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
     """Wrapper around :py:func:`subprocess.run`
 
     The main purpose of this wrapper is to log subprocess output in an orderly
     fashion that indentifies the responsible subprocess. An additional
     task is that this function goes to greater lengths to find possible Windows
     locations of our dependencies when they are not on the system PATH.
+
+    Arguments should be identical to ``subprocess.run``, except for following:
+
+    Arguments:
+        logs_errors_to_stdout: If True, indicates that the process writes its error
+            messages to stdout rather than stderr, so stdout should be logged
+            if there is an error. If False, stderr is logged. Could be used with
+            stderr=STDOUT, stdout=PIPE for example.
     """
     if not env:
         env = os.environ
@@ -43,25 +51,34 @@ def run(args, *, env=None, **kwargs):
         args = _fix_windows_args(program, args, env)
 
     log.debug("Running: %s", args)
-    process_log = log.getChild('subprocess.' + os.path.basename(program))
+    process_log = log.getChild(os.path.basename(program))
-    if sys.version_info < (3, 7) and os.name == 'nt':
+    if sys.version_info < (3, 7):
-        # Can't use close_fds=True on Windows with Python 3.6 or older
+        if os.name == 'nt':
-        # https://bugs.python.org/issue19575, etc.
+            # Can't use close_fds=True on Windows with Python 3.6 or older
-        kwargs['close_fds'] = False
+            # https://bugs.python.org/issue19575, etc.
+            kwargs['close_fds'] = False
+        if 'text' in kwargs:
+            # Convert run(...text=) to run(...universal_newlines=) for Python 3.6
+            kwargs['universal_newlines'] = kwargs['text']
+            del kwargs['text']
 
     stderr = None
+    stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
     try:
         proc = subprocess_run(args, env=env, **kwargs)
     except CalledProcessError as e:
-        stderr = getattr(e, 'stderr', None)
+        stderr = getattr(e, stderr_name, None)
         raise
     else:
-        stderr = getattr(proc, 'stderr', None)
+        stderr = getattr(proc, stderr_name, None)
     finally:
         if process_log.isEnabledFor(logging.DEBUG) and stderr:
             with suppress(AttributeError, UnicodeDecodeError):
                 stderr = stderr.decode('utf-8', 'replace')
-            process_log.debug("stderr = %s", stderr)
+            if logs_errors_to_stdout:
+                process_log.debug("stdout/stderr = %s", stderr)
+            else:
+                process_log.debug("stderr = %s", stderr)
     return proc
 
 
@@ -107,7 +124,7 @@ def get_version(
         proc = run(
             args_prog,
             close_fds=True,
-            universal_newlines=True,
+            text=True,
             stdout=PIPE,
             stderr=STDOUT,
             check=True,

tests/conftest.py

@@ -128,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
 
 
 @pytest.helpers.register
-def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
+def run_ocrmypdf(input_file, output_file, *args, text=True):
     "Run ocrmypdf and let caller deal with results"
 
     p_args = (
@@ -151,7 +151,7 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
         p_args,
         stdout=PIPE,
         stderr=PIPE,
-        universal_newlines=universal_newlines,
+        universal_newlines=text,  # When dropping support for Python 3.6 change to text=
         env=env,
         check=False,
     )

tests/test_main.py

@@ -598,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
 
@@ -659,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
         assert p.returncode == ExitCode.ok, p.stderr

tests/test_metadata.py

@@ -302,8 +302,8 @@ def test_kodak_toc(resources, outpdf):
 
     p = pikepdf.open(outpdf)
 
-    if pikepdf.Name.First in p.root.Outlines:
+    if pikepdf.Name.First in p.Root.Outlines:
-        assert isinstance(p.root.Outlines.First, pikepdf.Dictionary)
+        assert isinstance(p.Root.Outlines.First, pikepdf.Dictionary)
 
 
 def test_metadata_fixup_warning(resources, outdir, caplog):

tests/test_rotation.py

@@ -15,6 +15,7 @@ from PIL import Image
 
 from ocrmypdf import leptonica
 from ocrmypdf._exec import ghostscript, tesseract
+from ocrmypdf._plugin_manager import get_plugin_manager
 from ocrmypdf.helpers import Resolution
 from ocrmypdf.pdfinfo import PdfInfo
 
@@ -240,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
         '--rotate-pages',
         '--rotate-pages-threshold',
         '0.001',
-        universal_newlines=False,
+        text=False,
     )
     err = err.decode('utf-8', errors='replace')
     assert p.returncode == 0, err
@@ -256,3 +257,33 @@ def test_tesseract_orientation(resources, tmp_path):
     tesseract.get_orientation(  # Test results of this are unreliable
         tmp_path / '000001.png', engine_mode='3', timeout=10
     )
+
+
+def test_rasterize_rotates(resources, tmp_path):
+    pm = get_plugin_manager([])
+
+    img = tmp_path / 'img90.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=90,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (123, 151), "Image not rotated"
+
+    img = tmp_path / 'img180.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=180,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (151, 123), "Image not rotated"

tests/test_validation.py

@@ -55,7 +55,7 @@ def test_old_ghostscript(caplog):
         vd._check_options(
             *make_opts_pm(language='chi_sim', output_type='pdfa'), {'chi_sim'}
         )
-        assert 'Ghostscript does not work correctly' in caplog.text
+        assert 'does not work correctly' in caplog.text
 
     with patch('ocrmypdf._exec.ghostscript.version', return_value='9.18'), patch(
         'ocrmypdf._exec.tesseract.has_textonly_pdf', return_value=True
@@ -137,9 +137,9 @@ def test_report_file_size(tmp_path, caplog):
     caplog.clear()
 
     waste_of_space = b'Dummy' * 5000
-    pdf.root.Dummy = waste_of_space
+    pdf.Root.Dummy = waste_of_space
     pdf.save(in_)
-    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.Root.Dummy2 = waste_of_space + waste_of_space
     pdf.save(out)
 
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(