v12.7.2 release notes

When compiled without pybind11 2.8.0, pikepdf supplies a shim to implement
pikepdf._ObjectMapping.values() which has subtly different semantics
from a true dict-like objects; in particular it supports
next(objectmap.values())
where a standard dict requires
next(iter(objectmap.values()).

pybind11 2.8.0 now implements .values() properly, meaning some misuses of
protocol  in ocrmypdf fail.

If pybind11 < 2.8.0, pikepdf will
continue to offer its shim. If pybind11 >= 2.8.0, pikepdf does not add its shim.

Consequently no changes were needed in pikepdf.

Closes #843

.pre-commit-config.yaml

@@ -19,16 +19,16 @@ repos:
         language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py
   - repo: https://github.com/asottile/setup-cfg-fmt
-    rev: v1.17.0
+    rev: v1.19.0
     hooks:
       - id: setup-cfg-fmt
   - repo: https://github.com/asottile/pyupgrade
-    rev: v2.26.0
+    rev: v2.29.0
     hooks:
       - id: pyupgrade
         args: ["--py36-plus"]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v0.910
+    rev: v0.910-1
     hooks:
       - id: mypy
         additional_dependencies:

docs/api.rst

@@ -12,7 +12,7 @@ subprocess call anyway, as this provides isolation of its activities.
 Example
 =======
 
-OCRmyPDF one high-level function to run its main engine from an
+OCRmyPDF provides one high-level function to run its main engine from an
 application. The parameters are symmetric to the command line arguments
 and largely have the same functions.
 
@@ -23,7 +23,7 @@ and largely have the same functions.
     if __name__ == '__main__':  # To ensure correct behavior on Windows and macOS
         ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
-With a few exceptions, all of the command line arguments are available
+With some exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
 
 A few differences are that ``verbose`` and ``quiet`` are not available.
@@ -41,29 +41,29 @@ execution. To do this, it will:
 - manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
-The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
+The Python process that calls :func:`ocrmypdf.ocr()` must be sufficiently
 privileged to perform these actions.
 
 There currently is no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call :func:`ocrmypdf.ocr()` is suggested. That
 way your application will survive and remain interactive even if
 OCRmyPDF fails for any reason.
 
-Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
+Programs that call :func:`ocrmypdf.ocr()` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
 mapped file fails. OCRmyPDF may use memory mapping.
 
-``ocrmypdf.ocr()`` will take a threading lock to prevent multiple runs of itself
+:func:`ocrmypdf.ocr()` will take a threading lock to prevent multiple runs of itself
 in the same Python interpreter process. This is not thread-safe, because of how
 OCRmyPDF's plugins and Python's library import system work. If you need to parallelize
 OCRmyPDF, use processes.
 
 .. warning::
 
-    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
+    On Windows and macOS, the script that calls :func:`ocrmypdf.ocr()` must be
     protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
     not take at least one of these steps, process semantics will prevent
     OCRmyPDF from working correctly.
@@ -96,7 +96,7 @@ Exceptions
 
 OCRmyPDF may throw standard Python exceptions, ``ocrmypdf.exceptions.*``
 exceptions, some exceptions related to multiprocessing, and
-``KeyboardInterrupt``. The parent process should provide an exception
+:exc:`KeyboardInterrupt`. The parent process should provide an exception
 handler. OCRmyPDF will clean up its temporary files and worker processes
 automatically when an exception occurs.
 

docs/conf.py

@@ -40,6 +40,7 @@ extensions = [
 ]
 
 # Extension settings
+intersphinx_mapping = {'https://docs.python.org/': None}
 napoleon_use_rtype = False
 issues_github_path = "jbarlow83/OCRmyPDF"
 

docs/release_notes.rst

@@ -18,6 +18,33 @@ wish to use some of its features for working with PDFs.
    for Python 3.6 around that time. The change will be marked with a major
    release.
 
+v12.7.2
+=======
+
+-  Fixed "invalid version number" error for Tesseract packaging with nonstandard 
+   version "5.0.0-rc1.20211030".
+-  Fixed use of deprecated ``importlib.resources.read_binary``.
+-  Replace some uses of string paths with ``pathlib.Path``.
+-  Fixed a leaked file handle when using ``--output-type none``.
+-  Removed shims to support versions of pikepdf that are no longer supported.
+
+v12.7.1
+=======
+
+-  Declare support for pdfminer.six v20211012.
+
+v12.7.0
+=======
+
+-  Fixed test suite failure when using pikepdf 3.2.0 that was compiled with pybind11
+   2.8.0. :issue:`843`
+-  Improve advice to user about using ``--max-image-mpixels`` if OCR fails for this
+   reason.
+-  Minor documentation fixes. (Thanks to @mara004.)
+-  Don't require importlib-metadata and importlib-resources backports on versions of
+   Python where the standard library implementation is sufficient.
+   (Thanks to Marco Genasci.)
+
 v12.6.0
 =======
 

misc/batch.py

@@ -24,45 +24,39 @@
 import logging
 import os
 import sys
+from pathlib import Path
 
 import ocrmypdf
 
 # pylint: disable=logging-format-interpolation
 # pylint: disable=logging-not-lazy
 
-script_dir = os.path.dirname(os.path.realpath(__file__))
-print(script_dir + '/batch.py: Start')
+script_dir = Path(__file__).parent
 
 if len(sys.argv) > 1:
-    start_dir = sys.argv[1]
+    start_dir = Path(sys.argv[1])
 else:
-    start_dir = '.'
+    start_dir = Path('.')
 
 if len(sys.argv) > 2:
-    log_file = sys.argv[2]
+    log_file = Path(sys.argv[2])
 else:
-    log_file = script_dir + '/ocr-tree.log'
+    log_file = script_dir.with_name('ocr-tree.log')
 
 logging.basicConfig(
     level=logging.INFO,
     format='%(asctime)s %(message)s',
     filename=log_file,
-    filemode='w',
+    filemode='a',
 )
 
 ocrmypdf.configure_logging(ocrmypdf.Verbosity.default)
 
-for dir_name, _subdirs, file_list in os.walk(start_dir):
-    logging.info(dir_name + '\n')
-    os.chdir(dir_name)
-    for filename in file_list:
-        file_ext = os.path.splitext(filename)[1]
-        if file_ext == '.pdf':
-            full_path = dir_name + '/' + filename
-            print(full_path)
-            result = ocrmypdf.ocr(filename, filename, deskew=True)
-            if result == ocrmypdf.ExitCode.already_done_ocr:
-                print("Skipped document because it already contained text")
-            elif result == ocrmypdf.ExitCode.ok:
-                print("OCR complete")
-            logging.info(result)
+for filename in start_dir.glob("**/*.py"):
+    logging.info(f"Processing {filename}")
+    result = ocrmypdf.ocr(filename, filename, deskew=True)
+    if result == ocrmypdf.ExitCode.already_done_ocr:
+        logging.error("Skipped document because it already contained text")
+    elif result == ocrmypdf.ExitCode.ok:
+        logging.info("OCR complete")
+    logging.info(result)

setup.cfg

@@ -28,6 +28,7 @@ classifiers =
     Programming Language :: Python :: 3.7
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
+    Programming Language :: Python :: 3.10
     Topic :: Scientific/Engineering :: Image Recognition
     Topic :: Text Processing :: Indexing
     Topic :: Text Processing :: Linguistic
@@ -49,14 +50,14 @@ install_requires =
     cffi>=1.9.1  # must be a setup and install requirement
     coloredlogs>=14.0  # strictly optional
     img2pdf>=0.3.0,<0.5  # pure Python
-    importlib-metadata>=4  # until Python 3.8
-    importlib-resources>=5  # until Python 3.9
-    pdfminer.six!=20200720,>=20191110,<=20201018
+    pdfminer.six!=20200720,>=20191110,<=20211012
     pikepdf>=2.10.0
-    pikepdf<3;implementation_name=="pypy" and python_version=='3.6'
     pluggy>=0.13.0,<2
     reportlab>=3.5.66
     tqdm>=4
+    importlib-metadata>=4;python_version<'3.8'  # until Python 3.8
+    importlib-resources>=5;python_version<'3.9'  # until Python 3.9
+    pikepdf<3;implementation_name=="pypy" and python_version=='3.6'
 python_requires = >=3.6
 include_package_data = True
 package_dir =
@@ -64,8 +65,8 @@ package_dir =
 platforms = any
 setup_requires =
     cffi>=1.9.1  # to build the leptonica module
-    setuptools_scm
-    setuptools_scm_git_archive
+    setuptools-scm
+    setuptools-scm-git-archive
 zip_safe = False
 
 [options.packages.find]
@@ -112,4 +113,4 @@ ignore =
 ignore = D203,F401,W503,E501,E203,F841
 exclude = .git,__pycache__,docs/conf.py,build,dist,.venv,.venvpp,.eggs,tmp,src/ocrmypdf/lib/
 max-complexity = 10
-max-line-length = 100
+max-line-length = 100

src/ocrmypdf/_exec/tesseract.py

@@ -8,9 +8,7 @@
 """Interface to Tesseract executable"""
 
 import logging
-import os
 import re
-import shutil
 from collections import namedtuple
 from distutils.version import StrictVersion
 from os import fspath
@@ -61,8 +59,8 @@ class TesseractVersion(StrictVersion):
         r'''
         ^(\d+) \. (\d+) (\. (\d+))?  # groups: 1/major, 2/minor, 3/[skip], 4/patch
         [-]?  # optional hyphen separator
-        (?:(alpha|beta|rc|dev)?[.\-\ ]?(\d+)?)?  # 5/prerelease, 6/prerelease_num
-        (?:-(\d+)-g[0-9a-f]+)?  # untagged git version
+        (?: ((?:alpha|beta|rc|dev)\d*)? [.\-\ ]? (\d+)? )?  # 5/prerelease, 6/prerelease_num
+        (?:(?:-\d+)?-g[0-9a-f]+)?  # untagged git version
         $
         ''',
         re.VERBOSE | re.ASCII,
@@ -73,7 +71,7 @@ class TesseractVersion(StrictVersion):
             super().parse(vstring)
         except TypeError as e:
             if 'int() argument must be a string' in str(e):
-                super().parse(vstring + '0')
+                super().parse(vstring + '-0')
 
 
 def version():
@@ -250,7 +248,7 @@ def generate_hocr(
 
     # Reminder: test suite tesseract test plugins will break after any changes
     # to the number of order parameters here
-    args_tesseract.extend([os.fspath(input_file), os.fspath(prefix), 'hocr', 'txt'])
+    args_tesseract.extend([fspath(input_file), fspath(prefix), 'hocr', 'txt'])
     args_tesseract.extend(tessconfig)
     try:
         p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
@@ -273,7 +271,7 @@ def generate_hocr(
         # The sidecar text file will get the suffix .txt; rename it to
         # whatever caller wants it named
         if prefix.with_suffix('.txt').exists():
-            shutil.move(prefix.with_suffix('.txt'), output_text)
+            prefix.with_suffix('.txt').replace(output_text)
 
 
 def use_skip_page(output_pdf, output_text):
@@ -320,18 +318,18 @@ def generate_pdf(
     if user_patterns:
         args_tesseract.extend(['--user-patterns', user_patterns])
 
-    prefix = os.path.splitext(output_pdf)[0]  # Tesseract appends suffixes
+    prefix = output_pdf.parent / Path(output_pdf.stem)
 
     # Reminder: test suite tesseract test plugins might break after any changes
     # to the number of order parameters here
 
-    args_tesseract.extend([os.fspath(input_file), os.fspath(prefix), 'pdf', 'txt'])
+    args_tesseract.extend([fspath(input_file), fspath(prefix), 'pdf', 'txt'])
     args_tesseract.extend(tessconfig)
     try:
         p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
         stdout = p.stdout
-        if os.path.exists(prefix + '.txt'):
-            shutil.move(prefix + '.txt', output_text)
+        if prefix.with_suffix('.txt').exists():
+            prefix.with_suffix('.txt').replace(output_text)
     except TimeoutExpired:
         page_timedout(timeout)
         use_skip_page(output_pdf, output_text)

src/ocrmypdf/_sync.py

@@ -290,10 +290,10 @@ def exec_concurrent(context: PdfContext, executor: Executor):
         # Copy text file to destination
         copy_final(text, options.sidecar, context)
 
-    if options.output_type != 'none':
-        # Merge layers to one single pdf
-        pdf = ocrgraft.finalize()
+    # Merge layers to one single pdf
+    pdf = ocrgraft.finalize()
 
+    if options.output_type != 'none':
         # PDF/A and metadata
         log.info("Postprocessing...")
         pdf = post_process(pdf, context, executor)
@@ -414,6 +414,13 @@ def run_pipeline(options, *, plugin_manager, api=False):
         else:
             log.error(type(e).__name__)
         return e.exit_code
+    except (PIL.Image.DecompressionBombError if not api else NeverRaise) as e:
+        log.exception(
+            "A decompression bomb error was encountered while executing the "
+            "pipeline. Use the argument --max-image-mpixels to raise the maximum "
+            "image pixel limit."
+        )
+        return ExitCode.other_error
     except (Exception if not api else NeverRaise):  # pylint: disable=broad-except
         log.exception("An exception occurred while executing the pipeline")
         return ExitCode.other_error

src/ocrmypdf/_validation.py

@@ -160,10 +160,12 @@ def _pages_from_ranges(ranges: str) -> Set[int]:
             try:
                 new_pages = list(range(int(start) - 1, int(end)))
                 if not new_pages:
-                    raise BadArgsError(f"invalid page subrange '{start}-{end}'")
+                    raise BadArgsError(
+                        f"invalid page subrange '{start}-{end}'"
+                    ) from None
                 pages.extend(new_pages)
             except ValueError:
-                raise BadArgsError("invalid page range") from None
+                raise BadArgsError(f"invalid page subrange '{g}'") from None
 
     if not pages:
         raise BadArgsError(

src/ocrmypdf/_version.py

@@ -5,7 +5,10 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-from importlib_metadata import version as _package_version
+try:
+    from importlib_metadata import version as _package_version
+except ImportError:
+    from importlib.metadata import version as _package_version
 
 PROGRAM_NAME = 'ocrmypdf'
 

src/ocrmypdf/helpers.py

@@ -221,15 +221,7 @@ def check_pdf(input_file: Path) -> bool:
                 # If linearization is missing entirely, we do not complain. We do
                 # complain if linearization is present but incorrect.
                 pdf.check_linearization(sio)
-            except RuntimeError:
-                pass
-            except (
-                # Workaround for a problematic pikepdf version
-                # pragma: no cover
-                pikepdf.ForeignObjectError
-                if pikepdf.__version__ == '2.1.0'
-                else NeverRaise
-            ):
+            except (RuntimeError, pikepdf.ForeignObjectError):
                 pass
             else:
                 linearize_msgs = sio.getvalue()

src/ocrmypdf/optimize.py

@@ -454,7 +454,7 @@ def _transcode_png(pike: Pdf, filename: Path, xref: Xref) -> bool:
         img2pdf.convert(fspath(filename), outputstream=f)
 
     with Pdf.open(output) as pdf_image:
-        foreign_image = next(pdf_image.pages[0].images.values())
+        foreign_image = next(iter(pdf_image.pages[0].images.values()))
         local_image = pike.copy_foreign(foreign_image)
 
         im_obj = pike.get_object(xref, 0)
@@ -585,7 +585,10 @@ def optimize(
     log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
 
     if savings < 0:
-        log.info("Image optimization did not improve the file - discarded")
+        log.info(
+            "Image optimization did not improve the file - "
+            "optimizations will not be used"
+        )
         # We still need to save the file
         with Pdf.open(input_file) as pike:
             pike.remove_unreferenced_resources()

src/ocrmypdf/pdfa.py

@@ -13,7 +13,11 @@ import base64
 from pathlib import Path
 from typing import Dict, Iterator, Union
 
-import importlib_resources
+try:
+    from importlib_resources import files as package_files
+except ImportError:
+    from importlib.resources import files as package_files
+
 import pikepdf
 import pkg_resources  # deprecated
 
@@ -104,9 +108,7 @@ def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
     if icc != 'sRGB':
         raise NotImplementedError("Only supporting sRGB")
 
-    bytes_icc_profile = importlib_resources.read_binary(
-        'ocrmypdf.data', SRGB_ICC_PROFILE_NAME
-    )
+    bytes_icc_profile = (package_files('ocrmypdf.data') / SRGB_ICC_PROFILE).read_bytes()
     ps = '\n'.join(_make_postscript(icc, bytes_icc_profile, 3))
 
     # We should have encoded everything to pure ASCII by this point, and

tests/test_image_input.py

@@ -87,4 +87,4 @@ def test_jpeg_in_jpeg_out(resources, outpdf):
         'tests/plugins/tesseract_noop.py',
     )
     with pikepdf.open(outpdf) as pdf:
-        assert next(pdf.pages[0].images.values()).Filter == pikepdf.Name.DCTDecode
+        assert next(iter(pdf.pages[0].images.values())).Filter == pikepdf.Name.DCTDecode

tests/test_main.py

@@ -745,14 +745,13 @@ def test_pdfa_n(pdfa_level, resources, outpdf):
     assert pdfa_info['conformance'] == f'PDF/A-{pdfa_level}B'
 
 
-@pytest.mark.skipif(
-    PIL.__version__ < '5.0.0', reason="Pillow < 5.0.0 doesn't raise the exception"
-)
-@pytest.mark.slow
-def test_decompression_bomb(resources, outpdf):
+def test_decompression_bomb_error(resources, outpdf):
     p, _out, err = run_ocrmypdf(resources / 'hugemono.pdf', outpdf)
-    assert 'decompression bomb' in err
+    assert 'decompression bomb' in err and '--max-image-mpixels' in err
 
+
+@pytest.mark.slow
+def test_decompression_bomb_succeeds(resources, outpdf):
     p, _out, err = run_ocrmypdf(
         resources / 'hugemono.pdf', outpdf, '--max-image-mpixels', '2000'
     )

tests/test_validation.py

@@ -238,6 +238,13 @@ def test_version_comparison():
         need_version='4.0.0',
         version_parser=TesseractVersion,
     )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '5.0.0-rc1.20211030',
+        need_version='4.0.0',
+        version_parser=TesseractVersion,
+    )
     vd.check_external_program(
         program="tesseract",
         package="tesseract",