v11.3.4 release notes

pikepdf 2.1.0 throws wrong type of exception in this case, so special-case it.

Closes #680
Closes #681

azure-pipelines.yml

@@ -22,6 +22,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -59,6 +61,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:

docs/api.rst

@@ -20,7 +20,8 @@ and largely have the same functions.
 
     import ocrmypdf
 
-    ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
+    if __name__ == '__main__':  # To ensure correct behavior on Windows
+        ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
 With a few exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
@@ -35,8 +36,9 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
 execution. To do this, it will:
 
 - create a monitoring thread
-- create worker processes (forking itself)
+- create worker processes (on Linux, forking itself; on Windows and macOS, by
-- manage the signal flags of worker processes
+  spawning)
+- manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
 The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
@@ -47,9 +49,9 @@ There is no currently no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
-OCRmyPDF does not.
+OCRmyPDF fails for any reason.
 
 Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
@@ -57,11 +59,10 @@ mapped file fails. OCRmyPDF may use memory mapping.
 
 .. warning::
 
-    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
-    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
-    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    not take at least one of these steps, process semantics will prevent
-    of these steps, Windows process semantics will prevent OCRmyPDF from working
+    OCRmyPDF from working correctly.
-    correctly.
 
 Logging
 -------

docs/batch.rst

@@ -127,7 +127,7 @@ Users may need to customize the script to meet their requirements.
     "OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
     "OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
     "OCR_DESKEW", "Apply deskew to crooked input PDFs"
-    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={"rotate_pages": true}'``.
+    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={""rotate_pages"": true}'``."
     "OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
     "OCR_LOGLEVEL", "Level of log messages to report"
 

docs/docker.rst

@@ -1,3 +1,5 @@
+.. _docker:
+
 =====================
 OCRmyPDF Docker image
 =====================

docs/index.rst

@@ -1,7 +1,7 @@
 OCRmyPDF documentation
 ======================
 
-OCRmyPDF adds an optical charcter recognition (OCR) text layer to scanned PDF
+OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
 files, allowing them to be searched.
 
 PDF is the best format for storing and exchanging scanned documents.

docs/installation.rst

@@ -604,7 +604,7 @@ However, the OCR-to-text-layer functionality is available.
 Docker
 ------
 
-You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
+You can also :ref:`Install the Docker <docker>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
 Installing on FreeBSD
@@ -630,7 +630,7 @@ Installing the Docker image
 For some users, installing the Docker image will be easier than
 installing all of OCRmyPDF's dependencies.
 
-See `OCRmyPDF Docker Image <docker>`__ for more information.
+See :ref:`docker` for more information.
 
 Installing with Python pip
 ==========================

docs/release_notes.rst

@@ -12,6 +12,22 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+v11.3.4
+=======
+
+-  Fixed an error message 'called readLinearizationData for file that is not
+   linearized' that may occur when pikepdf 2.1.0 is used. (Upgrading to pikepdf
+   2.1.1 also fixes the issue.)
+-  File watcher now automatically includes ``.PDF`` in addition to ``.pdf`` to
+   better support case sensitive file systems.
+-  Some documentation and comment improvements.
+
+v11.3.3
+=======
+
+-  If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
+   conversion. (Possibly addresses #671.)
+
 v11.3.2
 =======
 

misc/watcher.py

@@ -45,7 +45,7 @@ OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = int(os.getenv('OCR_POLL_NEW_FILE_SECONDS', '1'))
 USE_POLLING = bool(os.getenv('OCR_USE_POLLING', ''))
 LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
-PATTERNS = ['*.pdf']
+PATTERNS = ['*.pdf', '*.PDF']
 
 log = logging.getLogger('ocrmypdf-watcher')
 

src/ocrmypdf/_exec/ghostscript.py

@@ -25,24 +25,24 @@ from ocrmypdf.subprocess import get_version, run
 
 log = logging.getLogger(__name__)
 
+missing_gs_error = """
+---------------------------------------------------------------------
+This error normally occurs when ocrmypdf find can't Ghostscript.
+Please ensure Ghostscript is installed and its location is added to
+the system PATH environment variable.
+
+For details see:
+    https://ocrmypdf.readthedocs.io/en/latest/installation.html
+---------------------------------------------------------------------
+"""
+
 _gswin = None
 if os.name == 'nt':
     _gswin = which('gswin64c')
     if not _gswin:
         _gswin = which('gswin32c')
         if not _gswin:
-            raise MissingDependencyError(
+            raise MissingDependencyError(missing_gs_error)
-                """
-                ---------------------------------------------------------------------
-                This error normally occurs when ocrmypdf can't Ghostscript.  Please
-                ensure Ghostscript is installed and its location is added to the
-                system PATH environment variable.
-
-                For details see:
-                    https://ocrmypdf.readthedocs.io/en/latest/installation.html
-                ---------------------------------------------------------------------
-                """
-            )
     _gswin = Path(_gswin).stem
 
 GS = _gswin if _gswin else 'gs'
@@ -146,6 +146,9 @@ def generate_pdfa(
     pdf_version: str = '1.5',
     pdfa_part: str = '2',
 ):
+    # Ghostscript's compression is all or nothing. We can either force all images
+    # to JPEG, force all to Flate/PNG, or let it decide how to encode the images.
+    # In most case it's best to let it decide.
     compression_args = []
     if compression == 'jpeg':
         compression_args = [
@@ -173,8 +176,9 @@ def generate_pdfa(
     strategy = 'RGB' if version() >= '9.19' else '/RGB'
 
     if version() == '9.23':
-        # 9.23: new feature JPEG passthrough is broken in some cases, best to
+        # 9.23: added JPEG passthrough as a new feature, but with a bug that
-        # disable it always
+        # incorrectly formats some images. Fixed as of 9.24. So we disable this
+        # feature for 9.23.
         # https://bugs.ghostscript.com/show_bug.cgi?id=699216
         compression_args.append('-dPassThroughJPEGImages=false')
 

src/ocrmypdf/_exec/tesseract.py

@@ -99,9 +99,7 @@ def get_languages():
 
     args_tess = ['tesseract', '--list-langs']
     try:
-        proc = run(
+        proc = run(args_tess, text=True, stdout=PIPE, stderr=STDOUT, check=True)
-            args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
-        )
         output = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(lang_error(e.output)) from e

src/ocrmypdf/_exec/unpaper.py

@@ -13,10 +13,11 @@
 import logging
 import os
 import shlex
+from decimal import Decimal
 from pathlib import Path
-from subprocess import PIPE, STDOUT, CalledProcessError
+from subprocess import PIPE, STDOUT
 from tempfile import TemporaryDirectory
-from typing import Tuple
+from typing import List, Optional, Tuple, Union
 
 from PIL import Image
 
@@ -24,10 +25,12 @@ from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
 from ocrmypdf.subprocess import get_version
 from ocrmypdf.subprocess import run as external_run
 
+DecFloat = Union[Decimal, float]
+
 log = logging.getLogger(__name__)
 
 
-def version():
+def version() -> str:
     return get_version('unpaper')
 
 
@@ -53,7 +56,7 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
         except KeyError:
             raise MissingDependencyError(
                 "Failed to convert image to a supported format."
-            ) from e
+            ) from None
 
         if im_modified or input_file.suffix != '.pnm':
             input_pnm = tmpdir / 'input.pnm'
@@ -65,7 +68,9 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
     return input_pnm, output_pnm
 
 
-def run(input_file, output_file, dpi, mode_args):
+def run(
+    input_file: Path, output_file: Path, dpi: DecFloat, mode_args: List[str]
+) -> None:
     args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
 
     with TemporaryDirectory() as tmpdir:
@@ -83,7 +88,6 @@ def run(input_file, output_file, dpi, mode_args):
             args_unpaper,
             close_fds=True,
             check=True,
-            universal_newlines=True,
             stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
             stdout=PIPE,  # and cannot send file output to stdout
             cwd=tmpdir,
@@ -100,14 +104,19 @@ def run(input_file, output_file, dpi, mode_args):
             ) from None
 
 
-def validate_custom_args(args: str):
+def validate_custom_args(args: str) -> List[str]:
     unpaper_args = shlex.split(args)
     if any(('/' in arg or arg == '.' or arg == '..') for arg in unpaper_args):
         raise ValueError('No filenames allowed in --unpaper-args')
     return unpaper_args
 
 
-def clean(input_file, output_file, dpi, unpaper_args=None):
+def clean(
+    input_file: Path,
+    output_file: Path,
+    dpi: DecFloat,
+    unpaper_args: Optional[List[str]] = None,
+):
     default_args = [
         '--layout',
         'none',

src/ocrmypdf/_plugin_manager.py

@@ -82,7 +82,7 @@ def _setup_plugins(
         pm.register(module)
 
 
-def get_plugin_manager(plugins: List[str], builtins=True):
+def get_plugin_manager(plugins: List[Union[str, Path]], builtins=True):
     pm = OcrmypdfPluginManager(
         project_name='ocrmypdf',
         setup_func=partial(_setup_plugins, plugins=plugins, builtins=builtins),

src/ocrmypdf/_sync.py

@@ -55,6 +55,7 @@ from ocrmypdf._validation import (
 )
 from ocrmypdf.exceptions import ExitCode, ExitCodeException
 from ocrmypdf.helpers import (
+    NeverRaise,
     available_cpu_count,
     check_pdf,
     pikepdf_enable_mmap,
@@ -301,12 +302,6 @@ def exec_concurrent(context: PdfContext):
     copy_final(pdf, options.output_file, context)
 
 
-class NeverRaise(Exception):
-    """An exception that is never raised"""
-
-    pass  # pylint: disable=unnecessary-pass
-
-
 def configure_debug_logging(log_filename, prefix: str = ''):
     """
     Create a debug log file at a specified location.

src/ocrmypdf/api.py

@@ -299,18 +299,18 @@ def ocr(  # pylint: disable=unused-argument
     else:
         plugins = list(plugins)
 
-    parser = get_parser()
+    # No new variable names should be assigned until these two steps are run
-    _plugin_manager = get_plugin_manager(plugins)
+    create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
-    _plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
-
-    create_options_kwargs = {
-        k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
-    }
     create_options_kwargs.update(kwargs)
 
+    parser = get_parser()
+    create_options_kwargs['parser'] = parser
+    plugin_manager = get_plugin_manager(plugins)
+    plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
+
     if 'verbose' in kwargs:
         warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
 
     options = create_options(**create_options_kwargs)
-    check_options(options, _plugin_manager)
+    check_options(options, plugin_manager)
-    return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
+    return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)

src/ocrmypdf/helpers.py

@@ -58,6 +58,10 @@ class Resolution(namedtuple('Resolution', ('x', 'y'))):
         return f"Resolution({self.x}x{self.y} dpi)"
 
 
+class NeverRaise(Exception):
+    """An exception that is never raised"""
+
+
 def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike):
     """
     Helper function: relinks soft symbolic link if necessary
@@ -186,17 +190,25 @@ def check_pdf(input_file: Path) -> bool:
                 log.warning(msg)
 
         sio = StringIO()
-        linearize = None
+        linearize_msgs = ''
         try:
+            # If linearization is missing entirely, we do not complain. We do
+            # complain if linearization is present but incorrect.
             pdf.check_linearization(sio)
         except RuntimeError:
             pass
+        except (
+            getattr(pikepdf, 'ForeignObjectError')
+            if pikepdf.__version__ == '2.1.0'  # This version may throw wrong exception
+            else NeverRaise
+        ):
+            pass
         else:
-            linearize = sio.getvalue()
+            linearize_msgs = sio.getvalue()
-            if linearize:
+            if linearize_msgs:
-                log.warning(linearize)
+                log.warning(linearize_msgs)
 
-        if not messages and not linearize:
+        if not messages and not linearize_msgs:
             return True
         return False
     finally:

src/ocrmypdf/subprocess.py

@@ -52,10 +52,15 @@ def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
 
     log.debug("Running: %s", args)
     process_log = log.getChild(os.path.basename(program))
-    if sys.version_info < (3, 7) and os.name == 'nt':
+    if sys.version_info < (3, 7):
-        # Can't use close_fds=True on Windows with Python 3.6 or older
+        if os.name == 'nt':
-        # https://bugs.python.org/issue19575, etc.
+            # Can't use close_fds=True on Windows with Python 3.6 or older
-        kwargs['close_fds'] = False
+            # https://bugs.python.org/issue19575, etc.
+            kwargs['close_fds'] = False
+        if 'text' in kwargs:
+            # Convert run(...text=) to run(...universal_newlines=) for Python 3.6
+            kwargs['universal_newlines'] = kwargs['text']
+            del kwargs['text']
 
     stderr = None
     stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
@@ -119,7 +124,7 @@ def get_version(
         proc = run(
             args_prog,
             close_fds=True,
-            universal_newlines=True,
+            text=True,
             stdout=PIPE,
             stderr=STDOUT,
             check=True,

tests/conftest.py

@@ -128,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
 
 
 @pytest.helpers.register
-def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
+def run_ocrmypdf(input_file, output_file, *args, text=True):
     "Run ocrmypdf and let caller deal with results"
 
     p_args = (
@@ -151,7 +151,7 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
         p_args,
         stdout=PIPE,
         stderr=PIPE,
-        universal_newlines=universal_newlines,
+        universal_newlines=text,  # When dropping support for Python 3.6 change to text=
         env=env,
         check=False,
     )

tests/test_main.py

@@ -598,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
 
@@ -659,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
         assert p.returncode == ExitCode.ok, p.stderr

tests/test_rotation.py

@@ -241,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
         '--rotate-pages',
         '--rotate-pages-threshold',
         '0.001',
-        universal_newlines=False,
+        text=False,
     )
     err = err.decode('utf-8', errors='replace')
     assert p.returncode == 0, err