v12.7.2 release notes

When compiled without pybind11 2.8.0, pikepdf supplies a shim to implement
pikepdf._ObjectMapping.values() which has subtly different semantics
from a true dict-like objects; in particular it supports
next(objectmap.values())
where a standard dict requires
next(iter(objectmap.values()).

pybind11 2.8.0 now implements .values() properly, meaning some misuses of
protocol  in ocrmypdf fail.

If pybind11 < 2.8.0, pikepdf will
continue to offer its shim. If pybind11 >= 2.8.0, pikepdf does not add its shim.

Consequently no changes were needed in pikepdf.

Closes #843

.pre-commit-config.yaml

@@ -19,16 +19,16 @@ repos:
         language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py
   - repo: https://github.com/asottile/setup-cfg-fmt
-    rev: v1.17.0
+    rev: v1.19.0
     hooks:
       - id: setup-cfg-fmt
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.26.0
+    rev: v2.29.0
     hooks:
       - id: pyupgrade
         args: ["--py36-plus"]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.910
+    rev: v0.910-1
     hooks:
       - id: mypy
         additional_dependencies:

docs/api.rst

@@ -12,7 +12,7 @@ subprocess call anyway, as this provides isolation of its activities.
 Example
 =======
 
-OCRmyPDF one high-level function to run its main engine from an
+OCRmyPDF provides one high-level function to run its main engine from an
 application. The parameters are symmetric to the command line arguments
 and largely have the same functions.
 
@@ -23,7 +23,7 @@ and largely have the same functions.
     if __name__ == '__main__':  # To ensure correct behavior on Windows and macOS
         ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
-With a few exceptions, all of the command line arguments are available
+With some exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
 
 A few differences are that ``verbose`` and ``quiet`` are not available.
@@ -41,29 +41,29 @@ execution. To do this, it will:
 - manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
-The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
+The Python process that calls :func:`ocrmypdf.ocr()` must be sufficiently
 privileged to perform these actions.
 
 There currently is no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call :func:`ocrmypdf.ocr()` is suggested. That
 way your application will survive and remain interactive even if
 OCRmyPDF fails for any reason.
 
-Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
+Programs that call :func:`ocrmypdf.ocr()` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
 mapped file fails. OCRmyPDF may use memory mapping.
 
-``ocrmypdf.ocr()`` will take a threading lock to prevent multiple runs of itself
+:func:`ocrmypdf.ocr()` will take a threading lock to prevent multiple runs of itself
 in the same Python interpreter process. This is not thread-safe, because of how
 OCRmyPDF's plugins and Python's library import system work. If you need to parallelize
 OCRmyPDF, use processes.
 
 .. warning::
 
-    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
+    On Windows and macOS, the script that calls :func:`ocrmypdf.ocr()` must be
     protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
     not take at least one of these steps, process semantics will prevent
     OCRmyPDF from working correctly.
@@ -96,7 +96,7 @@ Exceptions
 
 OCRmyPDF may throw standard Python exceptions, ``ocrmypdf.exceptions.*``
 exceptions, some exceptions related to multiprocessing, and
-``KeyboardInterrupt``. The parent process should provide an exception
+:exc:`KeyboardInterrupt`. The parent process should provide an exception
 handler. OCRmyPDF will clean up its temporary files and worker processes
 automatically when an exception occurs.
 

docs/conf.py

@@ -31,9 +31,16 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['sphinx.ext.napoleon', 'sphinx_issues']
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.intersphinx',
+    'sphinx.ext.autosummary',
+    'sphinx.ext.napoleon',
+    'sphinx_issues',
+]
 
 # Extension settings
+intersphinx_mapping = {'https://docs.python.org/': None}
 napoleon_use_rtype = False
 issues_github_path = "jbarlow83/OCRmyPDF"
 
@@ -91,7 +98,7 @@ if on_rtd:
 from importlib_metadata import version as package_version
 
 # The full version, including alpha/beta/rc tags.
-release = package_version('ocrmypdf').version
+release = package_version('ocrmypdf')
 version = '.'.join(release.split('.')[:2])
 
 

docs/index.rst

@@ -1,6 +1,8 @@
 OCRmyPDF documentation
 ======================
 
+.. figure:: images/logo.svg
+
 OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
 files, allowing them to be searched.
 

docs/release_notes.rst

@@ -18,6 +18,41 @@ wish to use some of its features for working with PDFs.
    for Python 3.6 around that time. The change will be marked with a major
    release.
 
+v12.7.2
+=======
+
+-  Fixed "invalid version number" error for Tesseract packaging with nonstandard 
+   version "5.0.0-rc1.20211030".
+-  Fixed use of deprecated ``importlib.resources.read_binary``.
+-  Replace some uses of string paths with ``pathlib.Path``.
+-  Fixed a leaked file handle when using ``--output-type none``.
+-  Removed shims to support versions of pikepdf that are no longer supported.
+
+v12.7.1
+=======
+
+-  Declare support for pdfminer.six v20211012.
+
+v12.7.0
+=======
+
+-  Fixed test suite failure when using pikepdf 3.2.0 that was compiled with pybind11
+   2.8.0. :issue:`843`
+-  Improve advice to user about using ``--max-image-mpixels`` if OCR fails for this
+   reason.
+-  Minor documentation fixes. (Thanks to @mara004.)
+-  Don't require importlib-metadata and importlib-resources backports on versions of
+   Python where the standard library implementation is sufficient.
+   (Thanks to Marco Genasci.)
+
+v12.6.0
+=======
+
+-  Implemented ``--output-type=none`` to skip producing PDFs for applications that
+   only want sidecar files (:issue:`787`).
+-  Fixed ambiguities in descriptions of behavior of ``--jbig2-lossy``.
+-  Various improvements to documentation.
+
 v12.5.0
 =======
 

misc/batch.py

@@ -24,45 +24,39 @@
 import logging
 import os
 import sys
+from pathlib import Path
 
 import ocrmypdf
 
 # pylint: disable=logging-format-interpolation
 # pylint: disable=logging-not-lazy
 
-script_dir = os.path.dirname(os.path.realpath(__file__))
-print(script_dir + '/batch.py: Start')
+script_dir = Path(__file__).parent
 
 if len(sys.argv) > 1:
-    start_dir = sys.argv[1]
+    start_dir = Path(sys.argv[1])
 else:
-    start_dir = '.'
+    start_dir = Path('.')
 
 if len(sys.argv) > 2:
-    log_file = sys.argv[2]
+    log_file = Path(sys.argv[2])
 else:
-    log_file = script_dir + '/ocr-tree.log'
+    log_file = script_dir.with_name('ocr-tree.log')
 
 logging.basicConfig(
     level=logging.INFO,
     format='%(asctime)s %(message)s',
     filename=log_file,
-    filemode='w',
+    filemode='a',
 )
 
 ocrmypdf.configure_logging(ocrmypdf.Verbosity.default)
 
-for dir_name, subdirs, file_list in os.walk(start_dir):
-    logging.info(dir_name + '\n')
-    os.chdir(dir_name)
-    for filename in file_list:
-        file_ext = os.path.splitext(filename)[1]
-        if file_ext == '.pdf':
-            full_path = dir_name + '/' + filename
-            print(full_path)
-            result = ocrmypdf.ocr(filename, filename, deskew=True)
-            if result == ocrmypdf.ExitCode.already_done_ocr:
-                print("Skipped document because it already contained text")
-            elif result == ocrmypdf.ExitCode.ok:
-                print("OCR complete")
-            logging.info(result)
+for filename in start_dir.glob("**/*.py"):
+    logging.info(f"Processing {filename}")
+    result = ocrmypdf.ocr(filename, filename, deskew=True)
+    if result == ocrmypdf.ExitCode.already_done_ocr:
+        logging.error("Skipped document because it already contained text")
+    elif result == ocrmypdf.ExitCode.ok:
+        logging.info("OCR complete")
+    logging.info(result)

misc/completion/ocrmypdf.fish

@@ -54,6 +54,7 @@ function __fish_ocrmypdf_output_type
     echo -e "pdfa-1\t"(_ "output a PDF/A-1b")
     echo -e "pdfa-2\t"(_ "output a PDF/A-2b")
     echo -e "pdfa-3\t"(_ "output a PDF/A-3b")
+    echo -e "none\t"(_ "do not produce an output PDF (for example, if you only care about --sidecar)")
 end
 complete -c ocrmypdf -x -l output-type -a '(__fish_ocrmypdf_output_type)' -d "select PDF output options"
 

misc/synology.py

@@ -46,7 +46,7 @@ if len(sys.argv) > 1:
 else:
     start_dir = '.'
 
-for dir_name, subdirs, file_list in os.walk(start_dir):
+for dir_name, _subdirs, file_list in os.walk(start_dir):
     logging.info(dir_name)
     os.chdir(dir_name)
     for filename in file_list:

setup.cfg

@@ -28,6 +28,7 @@ classifiers =
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
+    Programming Language :: Python :: 3.10
     Topic :: Scientific/Engineering :: Image Recognition
     Topic :: Text Processing :: Indexing
     Topic :: Text Processing :: Linguistic
@@ -49,14 +50,14 @@ install_requires =
     cffi>=1.9.1  # must be a setup and install requirement
     coloredlogs>=14.0  # strictly optional
     img2pdf>=0.3.0,<0.5  # pure Python
-    importlib-metadata>=4  # until Python 3.8
-    importlib-resources>=5  # until Python 3.9
-    pdfminer.six!=20200720,>=20191110,<=20201018
+    pdfminer.six!=20200720,>=20191110,<=20211012
     pikepdf>=2.10.0
-    pikepdf<3;implementation_name=="pypy" and python_version=='3.6'
     pluggy>=0.13.0,<2
     reportlab>=3.5.66
     tqdm>=4
+    importlib-metadata>=4;python_version<'3.8'  # until Python 3.8
+    importlib-resources>=5;python_version<'3.9'  # until Python 3.9
+    pikepdf<3;implementation_name=="pypy" and python_version=='3.6'
 python_requires = >=3.6
 include_package_data = True
 package_dir =
@@ -64,8 +65,8 @@ package_dir =
 platforms = any
 setup_requires =
     cffi>=1.9.1  # to build the leptonica module
-    setuptools_scm
-    setuptools_scm_git_archive
+    setuptools-scm
+    setuptools-scm-git-archive
 zip_safe = False
 
 [options.packages.find]
@@ -107,3 +108,9 @@ test = pytest
 [check-manifest]
 ignore =
     .github
+
+[flake8]
+ignore = D203,F401,W503,E501,E203,F841
+exclude = .git,__pycache__,docs/conf.py,build,dist,.venv,.venvpp,.eggs,tmp,src/ocrmypdf/lib/
+max-complexity = 10
+max-line-length = 100

src/ocrmypdf/_exec/tesseract.py

@@ -8,9 +8,7 @@
 """Interface to Tesseract executable"""
 
 import logging
-import os
 import re
-import shutil
 from collections import namedtuple
 from distutils.version import StrictVersion
 from os import fspath
@@ -20,7 +18,6 @@ from typing import List, Optional
 
 from PIL import Image
 
-from ocrmypdf.api import StrPath
 from ocrmypdf.exceptions import (
     MissingDependencyError,
     SubprocessOutputError,
@@ -62,8 +59,8 @@ class TesseractVersion(StrictVersion):
         r'''
         ^(\d+) \. (\d+) (\. (\d+))?  # groups: 1/major, 2/minor, 3/[skip], 4/patch
         [-]?  # optional hyphen separator
-        (?:(alpha|beta|rc|dev)?[.\-\ ]?(\d+)?)?  # 5/prerelease, 6/prerelease_num
-        (?:-(\d+)-g[0-9a-f]+)?  # untagged git version
+        (?: ((?:alpha|beta|rc|dev)\d*)? [.\-\ ]? (\d+)? )?  # 5/prerelease, 6/prerelease_num
+        (?:(?:-\d+)?-g[0-9a-f]+)?  # untagged git version
         $
         ''',
         re.VERBOSE | re.ASCII,
@@ -74,7 +71,7 @@ class TesseractVersion(StrictVersion):
             super().parse(vstring)
         except TypeError as e:
             if 'int() argument must be a string' in str(e):
-                super().parse(vstring + '0')
+                super().parse(vstring + '-0')
 
 
 def version():
@@ -251,7 +248,7 @@ def generate_hocr(
 
     # Reminder: test suite tesseract test plugins will break after any changes
     # to the number of order parameters here
-    args_tesseract.extend([os.fspath(input_file), os.fspath(prefix), 'hocr', 'txt'])
+    args_tesseract.extend([fspath(input_file), fspath(prefix), 'hocr', 'txt'])
     args_tesseract.extend(tessconfig)
     try:
         p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
@@ -274,7 +271,7 @@ def generate_hocr(
         # The sidecar text file will get the suffix .txt; rename it to
         # whatever caller wants it named
         if prefix.with_suffix('.txt').exists():
-            shutil.move(prefix.with_suffix('.txt'), output_text)
+            prefix.with_suffix('.txt').replace(output_text)
 
 
 def use_skip_page(output_pdf, output_text):
@@ -321,18 +318,18 @@ def generate_pdf(
     if user_patterns:
         args_tesseract.extend(['--user-patterns', user_patterns])
 
-    prefix = os.path.splitext(output_pdf)[0]  # Tesseract appends suffixes
+    prefix = output_pdf.parent / Path(output_pdf.stem)
 
     # Reminder: test suite tesseract test plugins might break after any changes
     # to the number of order parameters here
 
-    args_tesseract.extend([os.fspath(input_file), os.fspath(prefix), 'pdf', 'txt'])
+    args_tesseract.extend([fspath(input_file), fspath(prefix), 'pdf', 'txt'])
     args_tesseract.extend(tessconfig)
     try:
         p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
         stdout = p.stdout
-        if os.path.exists(prefix + '.txt'):
-            shutil.move(prefix + '.txt', output_text)
+        if prefix.with_suffix('.txt').exists():
+            prefix.with_suffix('.txt').replace(output_text)
     except TimeoutExpired:
         page_timedout(timeout)
         use_skip_page(output_pdf, output_text)

src/ocrmypdf/_exec/unpaper.py

@@ -96,12 +96,12 @@ def run(
         try:
             with Image.open(output_pnm) as imout:
                 imout.save(output_file, dpi=(dpi, dpi))
-        except (FileNotFoundError, OSError):
+        except OSError as e:
             raise SubprocessOutputError(
                 "unpaper: failed to produce the expected output file. "
                 + " Called with: "
                 + str(args_unpaper)
-            ) from None
+            ) from e
 
 
 def validate_custom_args(args: str) -> List[str]:

src/ocrmypdf/_sync.py

@@ -293,12 +293,13 @@ def exec_concurrent(context: PdfContext, executor: Executor):
     # Merge layers to one single pdf
     pdf = ocrgraft.finalize()
 
-    # PDF/A and metadata
-    log.info("Postprocessing...")
-    pdf = post_process(pdf, context, executor)
+    if options.output_type != 'none':
+        # PDF/A and metadata
+        log.info("Postprocessing...")
+        pdf = post_process(pdf, context, executor)
 
-    # Copy PDF file to destination
-    copy_final(pdf, options.output_file, context)
+        # Copy PDF file to destination
+        copy_final(pdf, options.output_file, context)
 
 
 def configure_debug_logging(log_filename: Path, prefix: str = ''):
@@ -399,7 +400,7 @@ def run_pipeline(options, *, plugin_manager, api=False):
                 return ExitCode.invalid_output_pdf
             report_output_file_size(options, start_input_file, options.output_file)
 
-    except (KeyboardInterrupt if not api else NeverRaise) as e:
+    except (KeyboardInterrupt if not api else NeverRaise):
         if options.verbose >= 1:
             log.exception("KeyboardInterrupt")
         else:
@@ -413,7 +414,14 @@ def run_pipeline(options, *, plugin_manager, api=False):
         else:
             log.error(type(e).__name__)
         return e.exit_code
-    except (Exception if not api else NeverRaise) as e:  # pylint: disable=broad-except
+    except (PIL.Image.DecompressionBombError if not api else NeverRaise) as e:
+        log.exception(
+            "A decompression bomb error was encountered while executing the "
+            "pipeline. Use the argument --max-image-mpixels to raise the maximum "
+            "image pixel limit."
+        )
+        return ExitCode.other_error
+    except (Exception if not api else NeverRaise):  # pylint: disable=broad-except
         log.exception("An exception occurred while executing the pipeline")
         return ExitCode.other_error
     finally:

src/ocrmypdf/_validation.py

@@ -26,12 +26,7 @@ from ocrmypdf.exceptions import (
     MissingDependencyError,
     OutputFileAccessError,
 )
-from ocrmypdf.helpers import (
-    is_file_writable,
-    is_iterable_notstr,
-    monotonic,
-    safe_symlink,
-)
+from ocrmypdf.helpers import is_file_writable, monotonic, safe_symlink, samefile
 from ocrmypdf.hocrtransform import HOCR_OK_LANGS
 from ocrmypdf.subprocess import check_external_program
 
@@ -68,7 +63,7 @@ def check_options_languages(options, ocr_engine_languages):
     missing_languages = options.languages - ocr_engine_languages
     if missing_languages:
         msg = (
-            f"OCR engine does not have language data for the following "
+            "OCR engine does not have language data for the following "
             "requested languages: \n"
         )
         msg += '\n'.join(lang for lang in missing_languages)
@@ -80,12 +75,18 @@ def check_options_output(options):
     is_latin = options.languages.issubset(HOCR_OK_LANGS)
 
     if options.pdf_renderer.startswith('hocr') and not is_latin:
-        msg = (
+        log.warning(
             "The 'hocr' PDF renderer is known to cause problems with one "
             "or more of the languages in your document.  Use "
-            "--pdf-renderer auto (the default) to avoid this issue."
+            "`--pdf-renderer auto` (the default) to avoid this issue."
+        )
+
+    if options.output_type == 'none' and options.output_file != os.devnull:
+        raise BadArgsError(
+            "Since you specified `--pdf-renderer none`, the output file "
+            f"{options.output_file} cannot be produced. Set the output file to "
+            f"{os.devnull} to suppress this message."
         )
-        log.warning(msg)
 
     lossless_reconstruction = False
     if not any(
@@ -112,6 +113,10 @@ def check_options_sidecar(options):
             raise BadArgsError(
                 "--sidecar filename must be specified when output file is stdout."
             )
+        elif options.output_file == os.devnull:
+            raise BadArgsError(
+                "--sidecar filename must be specified when output file is /dev/null or NUL."
+            )
         options.sidecar = options.output_file + '.txt'
     if options.sidecar == options.input_file or options.sidecar == options.output_file:
         raise BadArgsError(
@@ -155,10 +160,12 @@ def _pages_from_ranges(ranges: str) -> Set[int]:
             try:
                 new_pages = list(range(int(start) - 1, int(end)))
                 if not new_pages:
-                    raise BadArgsError(f"invalid page subrange '{start}-{end}'")
+                    raise BadArgsError(
+                        f"invalid page subrange '{start}-{end}'"
+                    ) from None
                 pages.extend(new_pages)
             except ValueError:
-                raise BadArgsError("invalid page range") from None
+                raise BadArgsError(f"invalid page subrange '{g}'") from None
 
     if not pages:
         raise BadArgsError(

src/ocrmypdf/_version.py

@@ -5,7 +5,10 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-from importlib_metadata import version as _package_version
+try:
+    from importlib_metadata import version as _package_version
+except ImportError:
+    from importlib.metadata import version as _package_version
 
 PROGRAM_NAME = 'ocrmypdf'
 

src/ocrmypdf/api.py

@@ -15,10 +15,7 @@ from pathlib import Path
 from typing import AnyStr, BinaryIO, Iterable, Optional, Union
 from warnings import warn
 
-from ocrmypdf._logging import (  # pylint: disable=unused-import
-    PageNumberFilter,
-    TqdmConsole,
-)
+from ocrmypdf._logging import PageNumberFilter, TqdmConsole
 from ocrmypdf._plugin_manager import get_plugin_manager
 from ocrmypdf._sync import run_pipeline
 from ocrmypdf._validation import check_options
@@ -338,3 +335,17 @@ def ocr(  # pylint: disable=unused-argument
         options = create_options(**create_options_kwargs)
         check_options(options, plugin_manager)
         return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)
+
+
+__all__ = [
+    'PageNumberFilter',
+    'TqdmConsole',
+    'Verbosity',
+    'check_options',
+    'configure_logging',
+    'create_options',
+    'get_parser',
+    'get_plugin_manager',
+    'ocr',
+    'run_pipeline',
+]

src/ocrmypdf/builtin_plugins/concurrency.py

@@ -21,7 +21,7 @@ import sys
 import threading
 from contextlib import suppress
 from multiprocessing.pool import Pool, ThreadPool
-from typing import Callable, Iterable, Optional, Tuple, Type, Union
+from typing import Callable, Iterable, Type, Union
 
 from tqdm import tqdm
 

src/ocrmypdf/builtin_plugins/tesseract_ocr.py

@@ -11,7 +11,6 @@ import os
 from ocrmypdf import hookimpl
 from ocrmypdf._exec import tesseract
 from ocrmypdf.cli import numeric
-from ocrmypdf.exceptions import MissingDependencyError
 from ocrmypdf.helpers import clamp
 from ocrmypdf.pluginspec import OcrEngine
 from ocrmypdf.subprocess import check_external_program

src/ocrmypdf/cli.py

@@ -147,7 +147,7 @@ Online documentation is located at:
     )
     parser.add_argument(
         '--output-type',
-        choices=['pdfa', 'pdf', 'pdfa-1', 'pdfa-2', 'pdfa-3'],
+        choices=['pdfa', 'pdf', 'pdfa-1', 'pdfa-2', 'pdfa-3', 'none'],
         default='pdfa',
         help="Choose output type. 'pdfa' creates a PDF/A-2b compliant file for "
         "long term archiving (default, recommended) but may not suitable "
@@ -155,7 +155,8 @@ Online documentation is located at:
         "also has problems with full Unicode text. 'pdf' attempts to "
         "preserve file contents as much as possible. 'pdf-a1' creates a "
         "PDF/A1-b file. 'pdf-a2' is equivalent to 'pdfa'. 'pdf-a3' creates a "
-        "PDF/A3-b file.",
+        "PDF/A3-b file. 'none' will produce no output, which may be helpful if "
+        "only the --sidecar is desired.",
     )
 
     # Use null string '\0' as sentinel to indicate the user supplied no argument,
@@ -340,8 +341,9 @@ Online documentation is located at:
             "Control how PDF is optimized after processing:"
             "0 - do not optimize; "
             "1 - do safe, lossless optimizations (default); "
-            "2 - do some lossy optimizations; "
-            "3 - do aggressive lossy optimizations (including lossy JBIG2)"
+            "2 - do lossy JPEG and JPEG2000 optimizations; "
+            "3 - do more aggressive lossy JPEG and JPEG2000 optimizations. "
+            "To enable lossy JBIG2, see --jbig2-lossy."
         ),
     )
     optimizing.add_argument(
@@ -379,7 +381,8 @@ Online documentation is located at:
         action='store_true',
         help=(
             "Enable JBIG2 lossy mode (better compression, not suitable for some "
-            "use cases - see documentation)."
+            "use cases - see documentation). Only takes effect if --optimize 1 or "
+            "higher is also enabled."
         ),
     )
     optimizing.add_argument(

src/ocrmypdf/helpers.py

@@ -221,15 +221,7 @@ def check_pdf(input_file: Path) -> bool:
                 # If linearization is missing entirely, we do not complain. We do
                 # complain if linearization is present but incorrect.
                 pdf.check_linearization(sio)
-            except RuntimeError:
-                pass
-            except (
-                # Workaround for a problematic pikepdf version
-                # pragma: no cover
-                getattr(pikepdf, 'ForeignObjectError')
-                if pikepdf.__version__ == '2.1.0'
-                else NeverRaise
-            ):
+            except (RuntimeError, pikepdf.ForeignObjectError):
                 pass
             else:
                 linearize_msgs = sio.getvalue()

src/ocrmypdf/optimize.py

@@ -97,7 +97,7 @@ def extract_image_filter(
         return None  # Don't mess with wide gamut images
 
     if filtdp[0] == Name.JPXDecode:
-        log.debug(f"Skipping JPEG2000 iamge, xref {xref}")
+        log.debug(f"Skipping JPEG2000 image, xref {xref}")
         return None  # Don't do JPEG2000
 
     if filtdp[0] == Name.CCITTFaxDecode and filtdp[1].get('/K', 0) >= 0:
@@ -454,7 +454,7 @@ def _transcode_png(pike: Pdf, filename: Path, xref: Xref) -> bool:
         img2pdf.convert(fspath(filename), outputstream=f)
 
     with Pdf.open(output) as pdf_image:
-        foreign_image = next(pdf_image.pages[0].images.values())
+        foreign_image = next(iter(pdf_image.pages[0].images.values()))
         local_image = pike.copy_foreign(foreign_image)
 
         im_obj = pike.get_object(xref, 0)
@@ -533,12 +533,15 @@ def transcode_pngs(
         _transcode_png(pike, filename, xref)
 
 
+DEFAULT_EXECUTOR = SerialExecutor()
+
+
 def optimize(
     input_file: Path,
     output_file: Path,
     context,
     save_settings,
-    executor: Executor = SerialExecutor(),
+    executor: Executor = DEFAULT_EXECUTOR,
 ) -> None:
     options = context.options
     if options.optimize == 0:
@@ -582,7 +585,10 @@ def optimize(
     log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
 
     if savings < 0:
-        log.info("Image optimization did not improve the file - discarded")
+        log.info(
+            "Image optimization did not improve the file - "
+            "optimizations will not be used"
+        )
         # We still need to save the file
         with Pdf.open(input_file) as pike:
             pike.remove_unreferenced_resources()

src/ocrmypdf/pdfa.py

@@ -13,7 +13,11 @@ import base64
 from pathlib import Path
 from typing import Dict, Iterator, Union
 
-import importlib_resources
+try:
+    from importlib_resources import files as package_files
+except ImportError:
+    from importlib.resources import files as package_files
+
 import pikepdf
 import pkg_resources  # deprecated
 
@@ -104,9 +108,7 @@ def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
     if icc != 'sRGB':
         raise NotImplementedError("Only supporting sRGB")
 
-    bytes_icc_profile = importlib_resources.read_binary(
-        'ocrmypdf.data', SRGB_ICC_PROFILE_NAME
-    )
+    bytes_icc_profile = (package_files('ocrmypdf.data') / SRGB_ICC_PROFILE).read_bytes()
     ps = '\n'.join(_make_postscript(icc, bytes_icc_profile, 3))
 
     # We should have encoded everything to pure ASCII by this point, and

src/ocrmypdf/pdfinfo/info.py

@@ -9,7 +9,7 @@
 import atexit
 import logging
 import re
-from collections import defaultdict, namedtuple
+from collections import defaultdict
 from contextlib import ExitStack
 from decimal import Decimal
 from enum import Enum
@@ -449,7 +449,7 @@ def _image_xobjects(container) -> Iterator[Tuple[Object, str]]:
     xobjs = resources['/XObject'].as_dict()
     for xobj in xobjs:
         candidate: Object = xobjs[xobj]
-        if not '/Subtype' in candidate:
+        if '/Subtype' not in candidate:
             continue
         if candidate['/Subtype'] == '/Image':
             pdfimage = candidate
@@ -877,6 +877,9 @@ class PageInfo:
         )
 
 
+DEFAULT_EXECUTOR = SerialExecutor()
+
+
 class PdfInfo:
     """Get summary information about a PDF"""
 
@@ -888,7 +891,7 @@ class PdfInfo:
         progbar: bool = False,
         max_workers: int = None,
         check_pages=None,
-        executor: Executor = SerialExecutor(),
+        executor: Executor = DEFAULT_EXECUTOR,
     ):
         self._infile = infile
         if check_pages is None:

src/ocrmypdf/subprocess/__init__.py

@@ -15,7 +15,6 @@ from collections.abc import Mapping
 from contextlib import suppress
 from distutils.version import LooseVersion, Version
 from functools import lru_cache
-from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError, CompletedProcess, Popen
 from subprocess import run as subprocess_run
 from typing import Callable, Optional, Type, Union
@@ -27,7 +26,9 @@ from ocrmypdf.exceptions import MissingDependencyError
 log = logging.getLogger(__name__)
 
 
-def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
+def run(
+    args, *, env=None, logs_errors_to_stdout: bool = False, **kwargs
+) -> CompletedProcess:
     """Wrapper around :py:func:`subprocess.run`
 
     The main purpose of this wrapper is to log subprocess output in an orderly
@@ -65,7 +66,9 @@ def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
     return proc
 
 
-def run_polling_stderr(args, *, callback, check=False, env=None, **kwargs):
+def run_polling_stderr(
+    args, *, callback: Callable[[str], None], check: bool = False, env=None, **kwargs
+) -> CompletedProcess:
     """Run a process like ``ocrmypdf.subprocess.run``, and poll stderr.
 
     Every line of produced by stderr will be forwarded to the callback function.
@@ -83,6 +86,8 @@ def run_polling_stderr(args, *, callback, check=False, env=None, **kwargs):
     with Popen(args, env=env, **kwargs) as proc:
         lines = []
         while proc.poll() is None:
+            if proc.stderr is None:
+                continue
             for msg in iter(proc.stderr.readline, ''):
                 if process_log.isEnabledFor(logging.DEBUG):
                     process_log.debug(msg.strip())
@@ -102,7 +107,7 @@ def _fix_process_args(args, env, kwargs):
         env = os.environ
 
     # Search in spoof path if necessary
-    program = args[0]
+    program = str(args[0])
 
     if os.name == 'nt':
         from ocrmypdf.subprocess._windows import fix_windows_args

src/ocrmypdf/subprocess/_windows.py

@@ -9,9 +9,9 @@ import os
 import shutil
 import sys
 from distutils.version import LooseVersion
-from itertools import chain, filterfalse
+from itertools import chain
 from pathlib import Path
-from typing import Any, Callable, Iterator, Optional, Tuple, TypeVar, cast
+from typing import Any, Callable, Iterable, Iterator, Set, Tuple, TypeVar
 
 try:
     import winreg
@@ -113,7 +113,7 @@ SHIMS = [
 ]
 
 
-def fix_windows_args(program, args, env):
+def fix_windows_args(program: str, args, env):
     """Adjust our desired program and command line arguments for use on Windows"""
 
     if sys.version_info < (3, 8):
@@ -137,14 +137,12 @@ def fix_windows_args(program, args, env):
     return args
 
 
-def unique_everseen(iterable, key=None):
-    "List unique elements, preserving order. Remember all elements ever seen."
+def unique_everseen(iterable: Iterable[T], key: Callable[[T], T]) -> Iterator[T]:
+    "List unique elements, preserving order."
     # unique_everseen('AAAABBBCCDAABBB') --> A B C D
     # unique_everseen('ABBCcAD', str.lower) --> A B C D
-    seen = set()
+    seen: Set[T] = set()
     seen_add = seen.add
-    if key is None:
-        key = lambda x: x
     for element in iterable:
         k = key(element)
         if k not in seen:

tests/conftest.py

@@ -69,6 +69,11 @@ def outpdf(tmp_path):
     return tmp_path / 'out.pdf'
 
 
+@pytest.fixture(scope="function")
+def outtxt(tmp_path):
+    return tmp_path / 'out.txt'
+
+
 @pytest.fixture(scope="function")
 def no_outpdf(tmp_path):
     """This just documents the fact that a test is not expected to produce

tests/test_image_input.py

@@ -87,4 +87,4 @@ def test_jpeg_in_jpeg_out(resources, outpdf):
         'tests/plugins/tesseract_noop.py',
     )
     with pikepdf.open(outpdf) as pdf:
-        assert next(pdf.pages[0].images.values()).Filter == pikepdf.Name.DCTDecode
+        assert next(iter(pdf.pages[0].images.values())).Filter == pikepdf.Name.DCTDecode

tests/test_main.py

@@ -745,14 +745,13 @@ def test_pdfa_n(pdfa_level, resources, outpdf):
     assert pdfa_info['conformance'] == f'PDF/A-{pdfa_level}B'
 
 
-@pytest.mark.skipif(
-    PIL.__version__ < '5.0.0', reason="Pillow < 5.0.0 doesn't raise the exception"
-)
-@pytest.mark.slow
-def test_decompression_bomb(resources, outpdf):
+def test_decompression_bomb_error(resources, outpdf):
     p, _out, err = run_ocrmypdf(resources / 'hugemono.pdf', outpdf)
-    assert 'decompression bomb' in err
+    assert 'decompression bomb' in err and '--max-image-mpixels' in err
 
+
+@pytest.mark.slow
+def test_decompression_bomb_succeeds(resources, outpdf):
     p, _out, err = run_ocrmypdf(
         resources / 'hugemono.pdf', outpdf, '--max-image-mpixels', '2000'
     )
@@ -881,3 +880,29 @@ def test_image_dpi_threshold(resources, outpdf):
         'tests/plugins/tesseract_noop.py',
     )
     assert outpdf.exists()
+
+
+def test_outputtype_none_bad_setup(resources, outpdf):
+    p, _out, err = run_ocrmypdf(
+        resources / 'trivial.pdf',
+        outpdf,
+        '--output-type=none',
+        '--plugin',
+        'tests/plugins/tesseract_noop.py',
+    )
+    assert p.returncode == ExitCode.bad_args
+    assert 'Set the output file to' in err
+
+
+def test_outputtype_none(resources, outtxt):
+    p, _out, err = run_ocrmypdf(
+        resources / 'trivial.pdf',
+        os.devnull,
+        '--output-type=none',
+        '--sidecar',
+        outtxt,
+        '--plugin',
+        'tests/plugins/tesseract_noop.py',
+    )
+    assert p.returncode == ExitCode.ok
+    assert outtxt.exists()

tests/test_metadata.py

@@ -287,7 +287,7 @@ def test_srgb_in_unicode_path(tmp_path):
 
 
 def test_kodak_toc(resources, outpdf):
-    _output = check_ocrmypdf(
+    check_ocrmypdf(
         resources / 'kcs.pdf',
         outpdf,
         '--output-type',

tests/test_validation.py

@@ -6,6 +6,7 @@
 
 
 import logging
+import os
 from unittest.mock import patch
 
 import pikepdf
@@ -237,6 +238,13 @@ def test_version_comparison():
         need_version='4.0.0',
         version_parser=TesseractVersion,
     )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '5.0.0-rc1.20211030',
+        need_version='4.0.0',
+        version_parser=TesseractVersion,
+    )
     vd.check_external_program(
         program="tesseract",
         package="tesseract",
@@ -298,3 +306,8 @@ def test_sidecar_equals_output(resources, no_outpdf):
     op = no_outpdf
     with pytest.raises(BadArgsError, match=r'--sidecar'):
         run_ocrmypdf_api(resources / 'trivial.pdf', op, '--sidecar', op)
+
+
+def test_devnull_sidecar(resources):
+    with pytest.raises(BadArgsError, match=r'--sidecar.*NUL'):
+        run_ocrmypdf_api(resources / 'trivial.pdf', os.devnull, '--sidecar')