v11.3.3 release notes

The parameters are equivalent but the latter is better named. Since
Python 3.6 doesn't support text= we use our wrapper to add it in that
place.

This is for subprocess.run.

.pre-commit-config.yaml

@@ -19,5 +19,5 @@ repos:
     rev: 19.10b0
     hooks:
       - id: black
-        language_version: python3.8
+        language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py

README.md

@@ -92,6 +92,9 @@ apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified lan
 
 # Arch Linux users
 pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
+
+# brew macOS users
+brew install tesseract-lang
 ```
 
 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.

azure-pipelines.yml

@@ -22,6 +22,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -59,6 +61,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -156,9 +160,8 @@ stages:
           #     versionSpec: "$(python.version)"
           - bash: |
               brew update
-              brew unlink python@2
               brew upgrade python
-              echo "Using Python `python3 --version`"
+              echo "Using `python3 --version`"
             displayName: "Update brew and Python"
           - bash: |
               brew install \

docs/api.rst

@@ -20,7 +20,8 @@ and largely have the same functions.
 
     import ocrmypdf
 
-    ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
+    if __name__ == '__main__':  # To ensure correct behavior on Windows
+        ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
 With a few exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
@@ -35,8 +36,9 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
 execution. To do this, it will:
 
 - create a monitoring thread
-- create worker processes (forking itself)
+- create worker processes (on Linux, forking itself; on Windows and macOS, by
-- manage the signal flags of worker processes
+  spawning)
+- manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
 The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
@@ -47,9 +49,9 @@ There is no currently no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
-OCRmyPDF does not.
+OCRmyPDF fails for any reason.
 
 Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
@@ -57,11 +59,10 @@ mapped file fails. OCRmyPDF may use memory mapping.
 
 .. warning::
 
-    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
-    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
-    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    not take at least one of these steps, process semantics will prevent
-    of these steps, Windows process semantics will prevent OCRmyPDF from working
+    OCRmyPDF from working correctly.
-    correctly.
 
 Logging
 -------

docs/release_notes.rst

@@ -12,6 +12,28 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+v11.3.3
+=======
+
+-  If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
+   conversion. (Possibly addresses #671.)
+
+v11.3.2
+=======
+
+-  Explicitly require pikepdf 2.0.0 or newer when running on Python 3.9. (There are
+   concerns about the stability of pybind11 2.5.x with Python 3.9, which is used in
+   pikepdf 1.x.)
+-  Fixed another issue related to page rotation.
+-  Fixed an issue where image marked as image masks were not properly considered
+   as optimization candidates.
+-  On some systems, unpaper seems to be unable to process the PNGs we offer it
+   as input. We now convert the input to PNM format, which unpaper always accepts.
+   Fixes #665 and #667.
+-  DPI sent to unpaper is now rounded to a more reasonable number of decimal digits.
+-  Debug and error messages from unpaper were being suppressed.
+-  Some documentation tweaks.
+
 v11.3.1
 =======
 

requirements/main.txt

@@ -1,12 +1,12 @@
 # requirements.txt can be used to replicate the developer's build environment
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
-cffi == 1.14.0
+cffi == 1.14.3
 coloredlogs == 14.0  # technically optional
-img2pdf == 0.3.6
+img2pdf == 0.4.0
-pdfminer.six == 20200517
+pdfminer.six == 20201018
-pikepdf == 1.16.1
+pikepdf == 2.0.0
 pluggy == 0.13.1
-Pillow == 7.1.2
+Pillow == 8.0.1
-reportlab == 3.5.42
+reportlab == 3.5.55
-tqdm == 4.46.1
+tqdm == 4.51.0

setup.py

@@ -73,7 +73,8 @@ setup(
         'coloredlogs >= 14.0',  # strictly optional
         'img2pdf >= 0.3.0, < 0.5',  # pure Python, so track HEAD closely
         'pdfminer.six >= 20191110, != 20200720, <= 20201018',
-        'pikepdf >= 1.14.0, < 3',
+        "pikepdf >= 1.14.0, < 3 ; python_version < '3.9'",
+        "pikepdf >= 2.0.0 ; python_version >= '3.9'",
         'Pillow >= 7.0.0',
         'pluggy >= 0.13.0, < 1.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/_exec/ghostscript.py

@@ -122,8 +122,6 @@ def rasterize_pdf(
         stderr = p.stderr.decode(errors='replace')
         if _gs_error_reported(stderr):
             log.error(stderr)
-        elif stderr:
-            log.debug(stderr)
 
     with Image.open(BytesIO(p.stdout)) as im:
         if rotation is not None:

src/ocrmypdf/_exec/tesseract.py

@@ -99,9 +99,7 @@ def get_languages():
 
     args_tess = ['tesseract', '--list-langs']
     try:
-        proc = run(
+        proc = run(args_tess, text=True, stdout=PIPE, stderr=STDOUT, check=True)
-            args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
-        )
         output = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(lang_error(e.output)) from e

src/ocrmypdf/_exec/unpaper.py

@@ -55,21 +55,21 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
                 "Failed to convert image to a supported format."
             ) from e
 
-        if im_modified or input_file.suffix != '.png':
+        if im_modified or input_file.suffix != '.pnm':
-            input_png = tmpdir / 'input.png'
+            input_pnm = tmpdir / 'input.pnm'
-            im.save(input_png, format='PNG', compress_level=1)
+            im.save(input_pnm, format='PPM')
         else:
             # No changes, PNG input, just use the file we already have
-            input_png = input_file
+            input_pnm = input_file
         output_pnm = tmpdir / f'output{suffix}'
-    return input_png, output_pnm
+    return input_pnm, output_pnm
 
 
 def run(input_file, output_file, dpi, mode_args):
-    args_unpaper = ['unpaper', '-v', '--dpi', str(dpi)] + mode_args
+    args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
 
     with TemporaryDirectory() as tmpdir:
-        input_png, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
+        input_pnm, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
 
         # To prevent any shenanigans from accepting arbitrary parameters in
         # --unpaper-args, we:
@@ -78,31 +78,25 @@ def run(input_file, output_file, dpi, mode_args):
         # 3) append absolute paths for the input and output file
         # This should ensure that a user cannot clobber some other file with
         # their unpaper arguments (whether intentionally or otherwise)
-        args_unpaper.extend([os.fspath(input_png), os.fspath(output_pnm)])
+        args_unpaper.extend([os.fspath(input_pnm), os.fspath(output_pnm)])
+        external_run(
+            args_unpaper,
+            close_fds=True,
+            check=True,
+            stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+            stdout=PIPE,  # and cannot send file output to stdout
+            cwd=tmpdir,
+            logs_errors_to_stdout=True,
+        )
         try:
-            proc = external_run(
+            with Image.open(output_pnm) as imout:
-                args_unpaper,
+                imout.save(output_file, dpi=(dpi, dpi))
-                check=True,
+        except (FileNotFoundError, OSError):
-                close_fds=True,
+            raise SubprocessOutputError(
-                universal_newlines=True,
+                "unpaper: failed to produce the expected output file. "
-                stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+                + " Called with: "
-                cwd=tmpdir,  # and cannot send file output to stdout
+                + str(args_unpaper)
-                stdout=PIPE,
+            ) from None
-            )
-        except CalledProcessError as e:
-            log.debug(e.stderr)
-            raise e from e
-        else:
-            log.debug(proc.stderr)
-            try:
-                with Image.open(output_pnm) as imout:
-                    imout.save(output_file, dpi=(dpi, dpi))
-            except (FileNotFoundError, OSError):
-                raise SubprocessOutputError(
-                    "unpaper: failed to produce the expected output file. "
-                    + " Called with: "
-                    + str(args_unpaper)
-                ) from None
 
 
 def validate_custom_args(args: str):

src/ocrmypdf/_pipeline.py

@@ -536,9 +536,6 @@ def create_ocr_image(image: Path, page_context: PageContext):
 
         # Pillow requires integer DPI
         dpi = tuple(round(coord) for coord in im.info['dpi'])
-        if page_context.pageinfo.rotation != 0:
-            log.info(f"Rotating {page_context.pageinfo.rotation}")
-            im = im.rotate(page_context.pageinfo.rotation)
         im.save(output_file, dpi=dpi)
     return output_file
 

src/ocrmypdf/_sync.py

@@ -307,7 +307,14 @@ class NeverRaise(Exception):
     pass  # pylint: disable=unnecessary-pass
 
 
-def configure_debug_logging(log_filename, prefix=''):
+def configure_debug_logging(log_filename, prefix: str = ''):
+    """
+    Create a debug log file at a specified location.
+
+    Arguments:
+        log_filename: Where to the put the log file.
+        prefix: The logging domain prefix that should be sent to the log.
+    """
     log_file_handler = logging.FileHandler(log_filename, delay=True)
     log_file_handler.setLevel(logging.DEBUG)
     formatter = logging.Formatter(

src/ocrmypdf/api.py

@@ -10,7 +10,7 @@ import os
 import sys
 from enum import IntEnum
 from pathlib import Path
-from typing import BinaryIO, Iterable, Union
+from typing import AnyStr, BinaryIO, Iterable, Optional, Union
 from warnings import warn
 
 from ocrmypdf._logging import PageNumberFilter, TqdmConsole
@@ -26,7 +26,8 @@ except ModuleNotFoundError:
     coloredlogs = None
 
 
-PathOrIO = Union[BinaryIO, os.PathLike, str, bytes]
+StrPath = Union[os.PathLike, AnyStr]
+PathOrIO = Union[BinaryIO, StrPath]
 
 
 class Verbosity(IntEnum):
@@ -202,7 +203,7 @@ def ocr(  # pylint: disable=unused-argument
     language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
-    sidecar: os.PathLike = None,
+    sidecar: Optional[StrPath] = None,
     jobs: int = None,
     use_threads: bool = None,
     title: str = None,
@@ -239,7 +240,7 @@ def ocr(  # pylint: disable=unused-argument
     user_words: os.PathLike = None,
     user_patterns: os.PathLike = None,
     fast_web_view: float = None,
-    plugins: Iterable[Union[str, Path]] = None,
+    plugins: Iterable[StrPath] = None,
     keep_temporary_files: bool = None,
     progress_bar: bool = None,
     **kwargs,
@@ -261,7 +262,7 @@ def ocr(  # pylint: disable=unused-argument
             read.
         output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
             interpreted as file system path to the output file. If the object
-            appears to be a writable stream (with methods such as ``.read()`` and
+            appears to be a writable stream (with methods such as ``.write()`` and
             ``.seek()``), the output will be written to this stream. If
             ``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
             (provided that standard output does not seem to be a terminal device).
@@ -298,18 +299,18 @@ def ocr(  # pylint: disable=unused-argument
     else:
         plugins = list(plugins)
 
-    parser = get_parser()
+    # No new variable names should be assigned until these two steps are run
-    _plugin_manager = get_plugin_manager(plugins)
+    create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
-    _plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
-
-    create_options_kwargs = {
-        k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
-    }
     create_options_kwargs.update(kwargs)
 
+    parser = get_parser()
+    create_options_kwargs['parser'] = parser
+    plugin_manager = get_plugin_manager(plugins)
+    plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
+
     if 'verbose' in kwargs:
         warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
 
     options = create_options(**create_options_kwargs)
-    check_options(options, _plugin_manager)
+    check_options(options, plugin_manager)
-    return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
+    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)

src/ocrmypdf/optimize.py

@@ -115,25 +115,29 @@ def extract_image_jbig2(
         and filtdp[0] != Name.JBIG2Decode
         and jbig2enc.available()
     ):
-        try:
+        # Save any colorspace associated with the image, so that we
-            # Save any colorspace associated with the image, so that we
+        # will export a pure 1-bit PNG with no palette or ICC profile.
-            # will export a pure 1-bit PNG with no palette or ICC profile.
+        # Showing the palette or ICC to jbig2enc will cause it to perform
-            # Showing the palette or ICC to jbig2enc will cause it to perform
+        # colorspace transform to 1bpp, which will conflict the palette or
-            # colorspace transform to 1bpp, which will conflict the palette or
+        # ICC if it exists.
-            # ICC if it exists.
+        colorspace = pim.obj.get(pikepdf.Name.ColorSpace, None)
-            colorspace = pim.obj.ColorSpace
+        if colorspace is not None or pim.image_mask:
-            # Set to DeviceGray temporarily; we already in 1 bpc.
+            try:
-            pim.obj.ColorSpace = pikepdf.Name.DeviceGray
+                # Set to DeviceGray temporarily; we already in 1 bpc.
-            imgname = root / f'{xref:08d}'
+                pim.obj.ColorSpace = pikepdf.Name.DeviceGray
-            with imgname.open('wb') as f:
+                imgname = root / f'{xref:08d}'
-                ext = pim.extract_to(stream=f)
+                with imgname.open('wb') as f:
-            imgname.rename(imgname.with_suffix(ext))
+                    ext = pim.extract_to(stream=f)
-        except pikepdf.UnsupportedImageTypeError:
+                imgname.rename(imgname.with_suffix(ext))
-            return None
+            except pikepdf.UnsupportedImageTypeError:
-        finally:
+                return None
-            # Restore image colorspace after temporarily setting it to DeviceGray
+            finally:
-            pim.obj.ColorSpace = colorspace
+                # Restore image colorspace after temporarily setting it to DeviceGray
-        return XrefExt(xref, ext)
+                if colorspace is not None:
+                    pim.obj.ColorSpace = colorspace
+                else:
+                    del pim.obj.ColorSpace
+            return XrefExt(xref, ext)
     return None
 
 

src/ocrmypdf/pdfinfo/info.py

@@ -818,12 +818,12 @@ class PdfInfo:
                 check_pages=check_pages,
                 detailed_analysis=detailed_analysis,
             )
-            self._needs_rendering = pdf.root.get('/NeedsRendering', False)
+            self._needs_rendering = pdf.Root.get('/NeedsRendering', False)
             self._has_acroform = False
-            if '/AcroForm' in pdf.root:
+            if '/AcroForm' in pdf.Root:
-                if len(pdf.root.AcroForm.get('/Fields', [])) > 0:
+                if len(pdf.Root.AcroForm.get('/Fields', [])) > 0:
                     self._has_acroform = True
-                elif '/XFA' in pdf.root.AcroForm:
+                elif '/XFA' in pdf.Root.AcroForm:
                     self._has_acroform = True
 
     @property

src/ocrmypdf/subprocess.py

@@ -25,13 +25,21 @@ from ocrmypdf.exceptions import MissingDependencyError
 log = logging.getLogger(__name__)
 
 
-def run(args, *, env=None, **kwargs):
+def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
     """Wrapper around :py:func:`subprocess.run`
 
     The main purpose of this wrapper is to log subprocess output in an orderly
     fashion that indentifies the responsible subprocess. An additional
     task is that this function goes to greater lengths to find possible Windows
     locations of our dependencies when they are not on the system PATH.
+
+    Arguments should be identical to ``subprocess.run``, except for following:
+
+    Arguments:
+        logs_errors_to_stdout: If True, indicates that the process writes its error
+            messages to stdout rather than stderr, so stdout should be logged
+            if there is an error. If False, stderr is logged. Could be used with
+            stderr=STDOUT, stdout=PIPE for example.
     """
     if not env:
         env = os.environ
@@ -43,25 +51,34 @@ def run(args, *, env=None, **kwargs):
         args = _fix_windows_args(program, args, env)
 
     log.debug("Running: %s", args)
-    process_log = log.getChild('subprocess.' + os.path.basename(program))
+    process_log = log.getChild(os.path.basename(program))
-    if sys.version_info < (3, 7) and os.name == 'nt':
+    if sys.version_info < (3, 7):
-        # Can't use close_fds=True on Windows with Python 3.6 or older
+        if os.name == 'nt':
-        # https://bugs.python.org/issue19575, etc.
+            # Can't use close_fds=True on Windows with Python 3.6 or older
-        kwargs['close_fds'] = False
+            # https://bugs.python.org/issue19575, etc.
+            kwargs['close_fds'] = False
+        if 'text' in kwargs:
+            # Convert run(...text=) to run(...universal_newlines=) for Python 3.6
+            kwargs['universal_newlines'] = kwargs['text']
+            del kwargs['text']
 
     stderr = None
+    stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
     try:
         proc = subprocess_run(args, env=env, **kwargs)
     except CalledProcessError as e:
-        stderr = getattr(e, 'stderr', None)
+        stderr = getattr(e, stderr_name, None)
         raise
     else:
-        stderr = getattr(proc, 'stderr', None)
+        stderr = getattr(proc, stderr_name, None)
     finally:
         if process_log.isEnabledFor(logging.DEBUG) and stderr:
             with suppress(AttributeError, UnicodeDecodeError):
                 stderr = stderr.decode('utf-8', 'replace')
-            process_log.debug("stderr = %s", stderr)
+            if logs_errors_to_stdout:
+                process_log.debug("stdout/stderr = %s", stderr)
+            else:
+                process_log.debug("stderr = %s", stderr)
     return proc
 
 
@@ -107,7 +124,7 @@ def get_version(
         proc = run(
             args_prog,
             close_fds=True,
-            universal_newlines=True,
+            text=True,
             stdout=PIPE,
             stderr=STDOUT,
             check=True,

tests/conftest.py

@@ -128,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
 
 
 @pytest.helpers.register
-def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
+def run_ocrmypdf(input_file, output_file, *args, text=True):
     "Run ocrmypdf and let caller deal with results"
 
     p_args = (
@@ -151,7 +151,7 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
         p_args,
         stdout=PIPE,
         stderr=PIPE,
-        universal_newlines=universal_newlines,
+        universal_newlines=text,  # When dropping support for Python 3.6 change to text=
         env=env,
         check=False,
     )

tests/test_main.py

@@ -598,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
 
@@ -659,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
         assert p.returncode == ExitCode.ok, p.stderr

tests/test_metadata.py

@@ -302,8 +302,8 @@ def test_kodak_toc(resources, outpdf):
 
     p = pikepdf.open(outpdf)
 
-    if pikepdf.Name.First in p.root.Outlines:
+    if pikepdf.Name.First in p.Root.Outlines:
-        assert isinstance(p.root.Outlines.First, pikepdf.Dictionary)
+        assert isinstance(p.Root.Outlines.First, pikepdf.Dictionary)
 
 
 def test_metadata_fixup_warning(resources, outdir, caplog):

tests/test_rotation.py

@@ -241,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
         '--rotate-pages',
         '--rotate-pages-threshold',
         '0.001',
-        universal_newlines=False,
+        text=False,
     )
     err = err.decode('utf-8', errors='replace')
     assert p.returncode == 0, err

tests/test_validation.py

@@ -137,9 +137,9 @@ def test_report_file_size(tmp_path, caplog):
     caplog.clear()
 
     waste_of_space = b'Dummy' * 5000
-    pdf.root.Dummy = waste_of_space
+    pdf.Root.Dummy = waste_of_space
     pdf.save(in_)
-    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.Root.Dummy2 = waste_of_space + waste_of_space
     pdf.save(out)
 
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(