v12.0.2 release notes

Closes Error when trying --remove-background on pdf #769

.docker/Dockerfile

@@ -79,6 +79,5 @@ COPY --from=builder /app/misc/watcher.py /app/
 COPY --from=builder /app/setup.cfg /app/setup.py /app/README.md /app/
 COPY --from=builder /app/requirements /app/requirements
 COPY --from=builder /app/tests /app/tests
-COPY --from=builder /app/src /app/src
 
 ENTRYPOINT ["/usr/local/bin/ocrmypdf"]

docs/cookbook.rst

@@ -58,9 +58,11 @@ portrait pages.
 You can increase (decrease) the parameter ``--rotate-pages-threshold``
 to make page rotation more (less) aggressive. The threshold number is the ratio
 of how confidence the OCR engine is that the document image should be changed,
-compared to kept the same. A value of ``15.0`` is the default, and is fairly
+compared to kept the same. The default value is quite conservative; on some files
-conservative. A value of ``2.0`` will produce more rotations, and more false
+it may not attempt rotations at all unless it is very confident that the current
-positives.
+rotation is wrong. A lower value of ``2.0`` will produce more rotations, and
+more false positives. Run with ``-v1`` to see the confidence level for each
+page to see if there may be a better value for your files.
 
 If the page is "just a little off horizontal", like a crooked picture,
 then you want ``--deskew``. ``--rotate-pages`` is for when the cardinal

docs/docker.rst

@@ -103,16 +103,41 @@ Adding languages to the Docker image
 By default the Docker image includes English, German, Simplified Chinese,
 French, Portuguese and Spanish, the most popular languages for OCRmyPDF
 users based on feedback. You may add other languages by creating a new
-Dockerfile based on the public one:
+Dockerfile based on the public one.
 
 .. code-block:: dockerfile
 
    FROM jbarlow83/ocrmypdf
 
-   # Add French
+   # Example: add Italian
-   RUN apt install tesseract-ocr-fra
+   RUN apt install tesseract-ocr-ita
 
-You can also copy training data to ``/usr/share/tesseract-ocr/<tesseract version>/tessdata``.
+To install language packs (training data) such as the 
+`tessdata_best <https://github.com/tesseract-ocr/tessdata_best>`_ suite or
+custom data, you first need to determine the version of Tesseract data files, which
+may differ from the Tesseract program version. Use this command to determine the data
+file version:
+
+.. code-block:: bash
+
+   docker run -i --rm --entrypoint /bin/ls jbarlow83/ocrmypdf /usr/share/tesseract-ocr
+
+As of 2021, the data file version is probably ``4.00``.
+
+You can then add new data with either a Dockerfile:
+
+.. code-block:: dockerfile
+
+   FROM jbarlow83/ocrmypdf
+
+   # Example: add a tessdata_best file
+   COPY chi_tra_vert.traineddata /usr/share/tesseract-ocr/<data version>/tessdata/
+
+Alternately, you can copy training data into a Docker container as follows:
+
+.. code-block:: bash
+
+   docker cp mycustomtraining.traineddata name_of_container:/usr/share/tesseract-ocr/<tesseract version>/tessdata/
 
 Executing the test suite
 ========================

docs/languages.rst

@@ -12,9 +12,9 @@ languages <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.
 Languages are identified by standardized three-letter codes (called ISO 639-2 Alpha-3).
 Tesseract's documentation also lists the three-letter code for your language.
 Some are anglicized, e.g. Spanish is ``spa`` rather than ``esp``, while others
-are not, e.g. German is ``deu``.
+are not, e.g. German is ``deu`` and French is ``fra``.
 
-After you have installed a language pack, you can use it ``ocrmypdf -l <language>``,
+After you have installed a language pack, you can use it with ``ocrmypdf -l <language>``,
 for example ``ocrmypdf -l spa``. For multilingual documents, you can specify
 all languages to be expected, e.g. ``ocrmypdf -l eng+fra`` for English and French.
 English is assumed by default unless other language(s) are specified.
@@ -35,8 +35,8 @@ Debian and Ubuntu users
 
 You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
 to what languages it should search for. Multiple languages can be
-requested using either ``-l eng+fre`` (English and French) or
+requested using either ``-l eng+fra`` (English and French) or
-``-l eng -l fre``.
+``-l eng -l fra``.
 
 Fedora users
 ============
@@ -51,8 +51,8 @@ Fedora users
 
 You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as
 to what languages it should search for. Multiple languages can be
-requested using either ``-l eng+fre`` (English and French) or
+requested using either ``-l eng+fra`` (English and French) or
-``-l eng -l fre``.
+``-l eng -l fra``.
 
 macOS users
 ===========

docs/release_notes.rst

@@ -12,6 +12,20 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
+
+v12.0.2
+=======
+
+-  Fix exception thrown when using ``--remove-background`` on files containing small
+   images (#769).
+-  Improve documentation for description of adding language packs to the Docker image
+   and corrected name of French language pack.  
+
+v12.0.1
+=======
+
+-  Fix "invalid version number" for untagged tesseract versions (#770).
+
 v12.0.0
 =======
 
@@ -25,12 +39,15 @@ v12.0.0
    Ghostscript to create a PDF/A. Generally this is faster than performing a
    color conversion, which is not always necessary.
 -  OCR text is now packaged in a Form XObject. This makes it easier to isolate
-   OCR from other document content. However, some poor implemented PDF text
+   OCR from other document content. However, some poorly implemented PDF text
-   extraction algorithms may fail to find the text.
+   extraction algorithms may fail to detect the text.
 -  Many API functions have stricter parameter checking or expect keyword arguments
    were they previously did not.
 -  Some deprecated functions in ``ocrmypdf.optimize`` were removed.
--  The ``ocrmypdf.leptonica`` module is now deprecated.
+-  The ``ocrmypdf.leptonica`` module is now deprecated, due to difficulties with
+   the current strategy of ABI binding on newer platforms like Apple Silicon.
+   It will be removed and replaced, either by repackaging Leptonica as an
+   independent library using or using a different image processing library.
 -  Continuous integration moved to GitHub Actions.
 -  We no longer depend on ``pytest_helpers_namespace`` for testing.
 
@@ -46,8 +63,29 @@ v12.0.0
    way OCRmyPDF outputs its messages.
 -  New plugin hook: ``filter_pdf_page``, for modifying individual PDF
    pages produced by OCRmyPDF.
+-  OCRmyPDF now runs on nonstandard execution environments that do not have
+   interprocess semaphores, such as AWS Lambda and Android Termux. If the environment
+   does not have semaphores, OCRmyPDF will automatically select an alternate
+   process executor that does not use semaphores.
+-  Continuous integration moved to GitHub Actions.
 -  We now generate an ARM64-compatible Docker image alongside the x64 image.
-   Thanks to @andkrause for contributing the change and @0x326 for review comments.
+   Thanks to @andkrause for doing most of the work in a pull request several months
+   ago, which we were finally able to integrate now. Also thanks to @0x326 for
+   review comments.
+
+**Fixes**
+
+-  Fixed a possible deadlock on attempting to flush ``sys.stderr`` when older
+   versions of Leptonica are in use.
+-  Some worker processes inherited resources from their parents such as log
+   handlers that may have also lead to deadlocks. These resources are now released.
+-  Improvements to test coverage.
+-  Removed vestiges of support for Tesseract versions older than 4.0.0-beta1 (
+   which ships with Ubuntu 18.04).
+-  OCRmyPDF can now parse all of Tesseract version numbers, since several
+   schemes have been in use.
+-  Fixed an issue with parsing PDFs that contain images drawn at a scale of 0. (#761)
+-  Removed a frequently repeated message about disabling mmap.
 
 v11.7.3
 =======

src/ocrmypdf/_exec/tesseract.py

@@ -61,6 +61,7 @@ class TesseractVersion(StrictVersion):
         ^(\d+) \. (\d+) (\. (\d+))?  # groups: 1/major, 2/minor, 3/[skip], 4/patch
         [-]?  # optional hyphen separator
         (?:(alpha|beta|rc|dev)[.\-\ ]?(\d+)?)?  # 5/prerelease, 6/prerelease_num
+        (?:-(\d+)-g[0-9a-f]+)?  # untagged git version
         $
         ''',
         re.VERBOSE | re.ASCII,

src/ocrmypdf/_plugin_manager.py

@@ -73,10 +73,19 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
                 module = importlib.import_module(name)
                 self.register(module)
 
-        # 2. Register setuptools plugins
+        # 2. Install semfree if needed
+        try:
+            # pylint: disable=import-outside-toplevel
+            from multiprocessing.synchronize import SemLock
+
+            del SemLock
+        except ImportError:
+            self.register(importlib.import_module('ocrmypdf.extra_plugins.semfree'))
+
+        # 3. Register setuptools plugins
         self.load_setuptools_entrypoints('ocrmypdf')
 
-        # 3. Register plugins specified on command line
+        # 4. Register plugins specified on command line
         for name in self.__plugins:
             if isinstance(name, Path) or name.endswith('.py'):
                 # Import by filename

src/ocrmypdf/_validation.py

@@ -341,7 +341,17 @@ def create_input_file(options, work_folder: Path) -> Tuple[Path, str]:
             safe_symlink(options.input_file, target)
             return target, os.fspath(options.input_file)
         except FileNotFoundError:
-            raise InputFileError(f"File not found - {options.input_file}")
+            msg = f"File not found - {options.input_file}"
+            if Path('/.dockerenv').exists():  # pragma: no cover
+                msg += (
+                    "\nDocker cannot your working directory unless you "
+                    "explicitly share it with the Docker container and set up"
+                    "permissions correctly.\n"
+                    "You may find it easier to use stdin/stdout:"
+                    "\n"
+                    "\tdocker run -i --rm jbarlow83/ocrmypdf - - <input.pdf >output.pdf\n"
+                )
+            raise InputFileError(msg)
 
 
 def check_requested_output_file(options):

src/ocrmypdf/extra_plugins/semfree.py

@@ -5,22 +5,21 @@
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
-"""Alternate executor to support OCRmyPDF in AWS Lambda.
+"""Semaphore-free alternate executor.
 
-AWS Lambda does not support the standard multiprocessing module because it
+There are two popular environments that do not fully support the standard Python
-environment.
+multiprocessing module: AWS Lambda, and Termux (a terminal emulator for Android).
 
-This alternate executor avoids that. However, it has drawbacks. Most notably,
+This alternate executor divvies up work among worker processes before processing,
-it divvies up work among worker processes at the beginning, to avoid coordinating
+rather than having each worker consume work from a shared queue when they finish
-effort in ways that require shared queues and semaphores. In this implementation,
+their task. This means workers have no need to coordinate with each other. Each
-there is no shared queue, so no possible lock contention between workers. The
+worker communicates only with the main process.
-main process has a shared pipe with each worker.
 
-If some tasks are larger than others, some workers will may fall far behind
+This is not without drawbacks. If the tasks are not "even" in size, which cannot
-while others have deep queues. The last worker may end up with fewer tasks.
+be guaranteed, some workers may end up with too much work while others are idle.
+It is less efficient than the standard implementation, so not th edefault.
 """
 
-
 import logging
 import logging.handlers
 import signal
@@ -30,7 +29,6 @@ from itertools import islice, repeat, takewhile, zip_longest
 from multiprocessing import Pipe, Process
 from multiprocessing.connection import Connection, wait
 from typing import Callable, Iterable, Iterator
-from unittest.mock import Mock
 
 from ocrmypdf import Executor, hookimpl
 from ocrmypdf._concurrent import NullProgressBar

src/ocrmypdf/helpers.py

@@ -15,7 +15,7 @@ from collections.abc import Iterable
 from contextlib import suppress
 from functools import wraps
 from io import StringIO
-from math import isclose
+from math import isclose, isfinite
 from pathlib import Path
 from typing import Any, Sequence
 
@@ -39,6 +39,10 @@ class Resolution(namedtuple('Resolution', ('x', 'y'))):
     def is_square(self) -> bool:
         return isclose(self.x, self.y, rel_tol=1e-3)
 
+    @property
+    def is_finite(self) -> bool:
+        return isfinite(self.x) and isfinite(self.y)
+
     def take_max(self, vals, yvals=None):
         if yvals is not None:
             return Resolution(max(self.x, *vals), max(self.y, *yvals))

src/ocrmypdf/leptonica.py

@@ -116,8 +116,8 @@ class _LeptonicaErrorTrap_Redirect:
         try:
             # It would make sense to do sys.stderr.flush() here, but that can deadlock
             # due to https://bugs.python.org/issue6721. So don't flush. Pretend
-            # there's nothing important sys.stderr. If the user cared they would
+            # there's nothing important in sys.stderr. If the user cared they would
-            # be use Leptonica 1.79 or later anyway and avoid this mess.
+            # be using Leptonica 1.79 or later anyway to avoid this mess.
             self.copy_of_stderr = os.dup(sys.stderr.fileno())
             os.dup2(self.tmpfile.fileno(), sys.stderr.fileno(), inheritable=False)
         except AttributeError:
@@ -192,11 +192,17 @@ class _LeptonicaErrorTrap_Queue:
         if 'Error' in output:
             if 'image file not found' in output:
                 raise FileNotFoundError()
-            if 'pixWrite: stream not opened' in output:
+            elif 'pixWrite: stream not opened' in output:
                 raise LeptonicaIOError()
-            if 'index not valid' in output:
+            elif 'index not valid' in output:
                 raise IndexError()
-            raise LeptonicaError(output)
+            elif 'pixGetInvBackgroundMap: w and h must be >= 5' in output:
+                logger.warning(
+                    "Leptonica attempted to remove background from a low resolution - "
+                    "you may want to review in a PDF viewer"
+                )
+            else:
+                raise LeptonicaError(output)
         return False
 
 
@@ -656,6 +662,9 @@ class Pix(LeptonicaObject):
         bg_val=200,
         smooth_kernel=(2, 1),
     ):
+        if self.width < tile_size[0] or self.height < tile_size[1]:
+            logger.info("Skipped pixMaskedThreshOnBackgroundNorm on small image")
+            return self
         # Background norm doesn't work on color mapped Pix, so remove colormap
         target_pix = self.remove_colormap(lept.REMOVE_CMAP_BASED_ON_SRC)
         with _LeptonicaErrorTrap():

src/ocrmypdf/pdfinfo/info.py

@@ -14,7 +14,7 @@ from contextlib import ExitStack
 from decimal import Decimal
 from enum import Enum
 from functools import partial
-from math import hypot, isclose
+from math import hypot, inf, isclose
 from os import PathLike
 from pathlib import Path
 from typing import Container, Iterator, Optional, Tuple, Union
@@ -256,17 +256,15 @@ def _get_dpi(ctm_shorthand, image_size) -> Resolution:
     a, b, c, d, _, _ = ctm_shorthand
 
     # Calculate the width and height of the image in PDF units
-    image_drawn_width = hypot(a, b)
+    image_drawn = hypot(a, b), hypot(c, d)
-    image_drawn_height = hypot(c, d)
 
-    # The scale of the image is pixels per unit of default user space (1/72")
+    def calc(drawn, pixels, inches_per_pt=72.0):
-    scale_w = image_size[0] / image_drawn_width
+        # The scale of the image is pixels per unit of default user space (1/72")
-    scale_h = image_size[1] / image_drawn_height
+        scale = pixels / drawn if drawn != 0 else inf
-
+        dpi = scale * inches_per_pt
-    # DPI = scale * 72
+        return dpi
-    dpi_w = scale_w * 72.0
-    dpi_h = scale_h * 72.0
 
+    dpi_w, dpi_h = (calc(image_drawn[n], image_size[n]) for n in range(2))
     return Resolution(dpi_w, dpi_h)
 
 
@@ -365,6 +363,10 @@ class ImageInfo:
     def enc(self):
         return self._enc
 
+    @property
+    def renderable(self):
+        return self.dpi.is_finite and self.width >= 0 and self.height >= 0
+
     @property
     def dpi(self):
         return _get_dpi(self._shorthand, (self._width, self._height))
@@ -734,7 +736,9 @@ class PageInfo:
 
         self._dpi = None
         if self._images:
-            dpi = Resolution(0.0, 0.0).take_max(image.dpi for image in self._images)
+            dpi = Resolution(0.0, 0.0).take_max(
+                image.dpi for image in self._images if image.renderable
+            )
             self._dpi = dpi
             self._width_pixels = int(round(dpi.x * float(self._width_inches)))
             self._height_pixels = int(round(dpi.y * float(self._height_inches)))

tests/test_pdfinfo.py

@@ -17,6 +17,7 @@ from reportlab.pdfgen.canvas import Canvas
 
 from ocrmypdf import pdfinfo
 from ocrmypdf.exceptions import InputFileError
+from ocrmypdf.helpers import Resolution
 from ocrmypdf.pdfinfo import Colorspace, Encoding
 from ocrmypdf.pdfinfo.layout import PDFPage
 
@@ -200,3 +201,23 @@ def test_pages_issue700(monkeypatch, resources):
             progbar=False,
             max_workers=1,
         )
+
+
+def test_image_scale0(resources, outpdf):
+    with pikepdf.open(resources / 'cmyk.pdf') as cmyk:
+        xobj = pikepdf.Page(cmyk.pages[0]).as_form_xobject()
+
+        p = pikepdf.Pdf.new()
+        p.add_blank_page(page_size=(72, 72))
+        objname = pikepdf.Page(p.pages[0]).add_resource(
+            p.copy_foreign(xobj), pikepdf.Name.XObject, pikepdf.Name.Im0
+        )
+        print(objname)
+        p.pages[0].Contents = pikepdf.Stream(
+            p, b"q 0 0 0 0 0 0 cm %s Do Q" % bytes(objname)
+        )
+        p.save(outpdf)
+
+    pi = pdfinfo.PdfInfo(outpdf, detailed_analysis=True, progbar=False, max_workers=1)
+    assert not pi.pages[0]._images[0].dpi.is_finite
+    assert pi.pages[0].dpi == Resolution(0, 0)

tests/test_validation.py

@@ -237,6 +237,13 @@ def test_version_comparison():
         need_version='4.0.0',
         version_parser=TesseractVersion,
     )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '4.1.1-rc2-25-g9707',
+        need_version='4.0.0',
+        version_parser=TesseractVersion,
+    )
     with pytest.raises(MissingDependencyError):
         vd.check_external_program(
             program="dummy_fails",