v9.8.2 release notes

Use more realistic test data

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,40 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+What command line or API call were you trying to run?
+
+```bash
+ocrmypdf  ...arguments... input.pdf output.pdf
+```
+
+Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
+
+**Example file**
+Include an input PDF or image that demonstrates your issue.
+
+Please provide an input file with no personal or confidential information. At your option you may `GPG-encrypt the file <https://github.com/jbarlow83/OCRmyPDF/wiki>` for OCRmyPDF's author only.
+
+Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
+
+(Exceptions: Issues with installation, command line argument parsing, test suite failures.Issues without example files usually cannot be resolved.)
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System**
+ - OS: [e.g. Linux, Windows, macOS]
+ - OCRmyPDF Version: ``ocrmypdf --version``
+ - How did you install ocrmypdf? Did you use a system package manager, `pip`, or a Docker image?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

azure-pipelines.yml

@@ -153,12 +153,13 @@ stages:
           matrix:
             Python37:
               python.version: "3.7"
-            Python38:
-              python.version: "3.8"
+            # Python38:
+            #   python.version: "3.8"
         steps:
-          - task: UsePythonVersion@0
-            inputs:
-              versionSpec: "$(python.version)"
+          # https://github.com/actions/virtual-environments/issues/664
+          # - task: UsePythonVersion@0
+          #   inputs:
+          #     versionSpec: "$(python.version)"
           - bash: |
               brew update
               brew unlink python@2
@@ -169,6 +170,7 @@ stages:
                 leptonica \
                 openjpeg \
                 pngquant \
+                python \
                 tesseract \
                 unpaper
             displayName: "Install system packages"

docs/api.rst

@@ -51,6 +51,14 @@ Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
 OCRmyPDF does not.
 
+.. warning::
+
+    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    of these steps, Windows fork semantics will prevent OCRmyPDF from working
+    correct.
+
 Logging
 -------
 

docs/batch.rst

@@ -202,6 +202,13 @@ Alternatives
 -  `Watchman <https://facebook.github.io/watchman/>`__ is a more
    powerful alternative to ``watchmedo``.
 
+AWS Lambda is not viable
+------------------------
+
+AWS Lambda and its equivalents have low limits on execution time and payload
+size, relative to OCRmyPDF's needs. As of this writing, the request/response
+payload for AWS Lambda was 6 MB, which means many PDFs will not fit.
+
 macOS Automator
 ===============
 

docs/docker.rst

@@ -90,10 +90,8 @@ Docker volume:
 
 .. code-block:: bash
 
-   docker run --rm -v $(pwd):/data ocrmypdf /data/input.pdf /data/output.pdf
-
-(However, when done this way, ``output.pdf`` may be owned by the root
-user.)
+   alias docker_ocrmypdf='docker run --rm  -i --user "$(id -u):$(id -g)" --workdir /data -v "$PWD:/data" ocrmypdf'
+   docker_ocrmypdf /data/input.pdf /data/output.pdf
 
 .. _docker-lang-packs:
 

docs/index.rst

@@ -27,6 +27,7 @@ image processing and OCR to existing PDFs.
    advanced
    batch
    security
+   performance
    errors
 
 .. toctree::

docs/installation.rst

@@ -137,11 +137,12 @@ Installing the latest version on Ubuntu 18.04 LTS
 -------------------------------------------------
 
 Ubuntu 18.04 includes ocrmypdf 6.1.2 - you can install that with ``apt``, but
-it is quite old now. To install a more recent version, first install several
-system dependencies:
+it is quite old now. To install a more recent version, uninstall the old version
+of ocrmypdf, and install the following dependencies:
 
 .. code-block:: bash
 
+    sudo apt-get -y remove ocrmypdf
     sudo apt-get -y update
     sudo apt-get -y install \
         ghostscript \
@@ -326,20 +327,7 @@ standard tooling needed to build packages, such as a compiler and binary tools.
 
    sudo pacman -S base-devel
 
-The OCRmyPDF package depends on `the python-pdfminer.six AUR package
-<https://aur.archlinux.org/packages/python-pdfminer.six/>`__.  Dependencies on
-AUR packages are not automatically resolved, so this package must be manually
-installed first.
-
-.. code-block:: bash
-
-   curl -O https://aur.archlinux.org/cgit/aur.git/snapshot/python-pdfminer.six.tar.gz
-   tar xvzf python-pdfminer.six.tar.gz
-   cd python-pdfminer.six
-   makepkg -sri
-
-With that complete you can then repeat the same series of steps for the
-OCRmyPDF package.
+Now you are ready to install the OCRmyPDF package.
 
 .. code-block:: bash
 
@@ -373,11 +361,10 @@ page.
     fine without it but will produce larger output files. The encoder is
     available from `the jbig2enc-git AUR package
     <https://aur.archlinux.org/packages/jbig2enc-git/>`__ and may be installed
-    using the same series of steps as for the installation of the pdfminer.six
-    and OCRmyPDF AUR packages. Alternatively, it may be built manually from
-    source following the instructions in `Installing the JBIG2 encoder
-    <jbig2>`__.  If JBIG2 is installed, OCRmyPDF 7.0.0 and later will
-    automatically detect it.
+    using the same series of steps as for the installation OCRmyPDF AUR
+    package. Alternatively, it may be built manually from source following the
+    instructions in `Installing the JBIG2 encoder <jbig2>`__.  If JBIG2 is
+    installed, OCRmyPDF 7.0.0 and later will automatically detect it.
 
 Alpine Linux
 ------------
@@ -442,7 +429,10 @@ languages you can optionally install them all:
 Manual installation on macOS
 ----------------------------
 
-These instructions probably work on all macOS supported by Homebrew.
+These instructions probably work on all macOS supported by Homebrew, and are
+for installing a more current version of OCRmyPDF than is available from
+Homebrew. Note that the Homebrew versions usually track the release versions
+fairly closely.
 
 If it's not already present, `install Homebrew <http://brew.sh/>`__.
 
@@ -453,14 +443,8 @@ Update Homebrew:
     brew update
 
 Install or upgrade the required Homebrew packages, if any are missing.
-To do this, download the ``Brewfile`` that lists all of the dependencies
-to the current directory, and run ``brew bundle`` to process them
-(installing or upgrading as needed). ``Brewfile`` is a plain text file.
-
-.. code-block:: bash
-
-    wget https://github.com/jbarlow83/OCRmyPDF/raw/master/.travis/Brewfile
-    brew bundle
+To do this, use ``brew edit ocrmypdf`` to obtain a recent list of Homebrew
+dependencies. You could also check the ``azure-pipelines.yml``.
 
 This will include the English, French, German and Spanish language
 packs. If you need other languages you can optionally install them all:
@@ -591,6 +575,53 @@ Docker
 You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
+Installing on Cygwin64 under Windows
+====================================
+
+First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::
+
+    python36 (or later)
+    python3?-devel
+    python3?-pip
+    python3?-lxml
+    python3?-imaging
+
+       (where 3? means match the version of python3 you installed)
+
+    gcc-g++
+    ghostscript (<=9.50 or >=9.52-2 see note below)
+    libexempi3
+    libexempi-devel
+    libffi6
+    libffi-devel
+    pngquant
+    qpdf
+    libqpdf-devel
+    tesseract-ocr
+    tesseract-ocr-devel
+
+.. note::
+
+    The Cygwin package for Ghostscript in versions 9.52 and
+    9.52-1 contained a bug that caused an exception to occur when
+    ocrmypdf invoked gs.  Make sure you have either 9.50 (or earlier)
+    or 9.52-2 (or later).
+
+Then open a Cygwin terminal (i.e. ``mintty``), run the following commands. Note
+that if you are using the version of ``pip`` that was installed with the Cygwin
+Python package, the command name will be ``pip3``.  If you have since updated
+``pip`` (with, for instance ``pip3 install --upgrade pip``) the the command is
+likely just ``pip`` instead of ``pip3``:
+
+.. code-block:: bash
+
+    pip3 install wheel
+    pip3 install ocrmypdf
+
+The optional dependency "unpaper" that is currently not available under Cygwin.
+Without it, certain options such as ``--clean`` will produce an error message.
+However, the OCR-to-text-layer functionality is available.
+
 Installing with Python pip
 ==========================
 

docs/performance.rst

@@ -0,0 +1,22 @@
+===========
+Performance
+===========
+
+Some users have noticed that current versions of OCRmyPDF do not run as quickly
+as some older versions (specifically 6.x and older). This is because OCRmyPDF
+added image optimization as a postprocessing step, and it is enabled by default.
+
+Speed
+=====
+
+If running OCRmyPDF quickly is your main goal, you can use settings such as:
+
+* ``--optimize 0`` to disable file size optimization
+* ``--output-type pdf`` to disable PDF/A generation
+* ``--fast-web-view 0`` to disable fast web view optimization
+* ``--skip-big`` to skip large images, if some pages have large images
+
+You can also avoid:
+
+* ``--force-ocr``
+* Image preprocessing

docs/release_notes.rst

@@ -13,6 +13,45 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.8.2
+======
+
+-  Fixed an issue where OCRmyPDF would ignore text inside Form XObject when
+   making certain decisions about whether a document already had text.
+-  Fixed file size increase warning to take overhead of small files into account.
+-  Added instructions for installing on Cygwin.
+
+v9.8.1
+======
+
+-  Fixed an issue where unexpected files in the ``%PROGRAMFILES%\gs`` directory
+   (Windows) caused an exception.
+-  Mark pdfminer.six 20200517 as supported.
+-  If jbig2enc is missing and optimization is requested, a warning is issued
+   instead of an error, which was the intended behavior.
+-  Documentation updates.
+
+v9.8.0
+======
+
+-  Fixed issue where only the first PNG (FlateDecode) image in a file would be
+   considered for optimization. File sizes should be improved from here on.
+-  Fixed a startup crash when the chosen language was Japanese (#543).
+-  Added options to configure polling and log level to watcher.py.
+
+v9.7.2
+======
+
+-  Fixed an issue with ``ocrmypdf.ocr(...language=)`` not accepting a list of
+   languages as documented.
+-  Updated setup.py to confirm that pdfminer.six version 20200402 is supported.
+
+v9.7.1
+======
+
+-  Fixed version check failing when used with qpdf 10.0.0.
+-  Added some missing type annotations.
+-  Updated documentation to warn about need for "ifmain" guard and Windows.
 
 v9.7.0
 ======

misc/watcher.py

@@ -25,6 +25,7 @@ from pathlib import Path
 import pikepdf
 from watchdog.events import PatternMatchingEventHandler
 from watchdog.observers import Observer
+from watchdog.observers.polling import PollingObserver
 
 import ocrmypdf
 
@@ -37,7 +38,8 @@ ON_SUCCESS_DELETE = bool(os.getenv('OCR_ON_SUCCESS_DELETE', False))
 DESKEW = bool(os.getenv('OCR_DESKEW', False))
 OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = os.getenv('OCR_POLL_NEW_FILE_SECONDS', 1)
-LOGLEVEL = os.environ.get('OCR_LOGLEVEL', 'INFO').upper()
+USE_POLLING = bool(os.getenv('OCR_USE_POLLING', False))
+LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
 PATTERNS = ['*.pdf']
 
 log = logging.getLogger('ocrmypdf-watcher')
@@ -112,6 +114,7 @@ def main():
     ocrmypdf.configure_logging(
         verbosity=ocrmypdf.Verbosity.default, manage_root_logger=True
     )
+    log.setLevel(LOGLEVEL)
     log.info(
         f"Starting OCRmyPDF watcher with config:\n"
         f"Input Directory: {INPUT_DIRECTORY}\n"
@@ -126,6 +129,7 @@ def main():
         f"DESKEW: {DESKEW}\n"
         f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
+        f"USE_POLLING: {USE_POLLING}\n"
         f"LOGLEVEL: {LOGLEVEL}\n"
     )
 
@@ -134,7 +138,10 @@ def main():
         sys.exit(1)
 
     handler = HandleObserverEvent(patterns=PATTERNS)
-    observer = Observer()
+    if USE_POLLING:
+        observer = PollingObserver()
+    else:
+        observer = Observer()
     observer.schedule(handler, INPUT_DIRECTORY, recursive=True)
     observer.start()
     try:

requirements/dev.txt

@@ -1,2 +0,0 @@
-twine >= 1.8.1
-coverage >= 4.5

requirements/main.txt

@@ -2,9 +2,9 @@
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
 cffi == 1.14.0
-img2pdf == 0.3.3
-pdfminer.six == 20200124
-pikepdf == 1.10.2
-Pillow == 7.0.0
+img2pdf == 0.3.4
+pdfminer.six == 20200402
+pikepdf == 1.11.1
+Pillow == 7.1.1
 reportlab == 3.5.34
-tqdm == 4.42.1
+tqdm == 4.45.0

setup.py

@@ -62,7 +62,7 @@ setup(
     long_description_content_type='text/markdown',
     url='https://github.com/jbarlow83/OCRmyPDF',
     author='James R. Barlow',
-    author_email='jim@purplerock.ca',
+    author_email='james@purplerock.ca',
     packages=find_packages('src', exclude=["tests", "tests.*"]),
     package_dir={'': 'src'},
     keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200124',
+        'pdfminer.six >= 20181108, <= 20200517',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/_validation.py

@@ -24,6 +24,7 @@ import sys
 from pathlib import Path
 from shutil import copyfileobj
 
+import pikepdf
 import PIL
 
 from ._unicodefun import verify_python3_env
@@ -410,8 +411,15 @@ def report_output_file_size(options, input_file, output_file):
         input_size = Path(input_file).stat().st_size
     except FileNotFoundError:
         return  # Outputting to stream or something
+    with pikepdf.open(output_file) as p:
+        # Overhead constants obtained by estimating amount of data added by OCR
+        # PDF/A conversion, and possible XMP metadata addition, with compression
+        FILE_OVERHEAD = 4000
+        OCR_PER_PAGE_OVERHEAD = 3000
+        reasonable_overhead = FILE_OVERHEAD + OCR_PER_PAGE_OVERHEAD * len(p.pages)
     ratio = output_size / input_size
-    if ratio < 1.35 or input_size < 25000:
+    reasonable_ratio = output_size / (input_size + reasonable_overhead)
+    if reasonable_ratio < 1.35 or input_size < 25000:
         return  # Seems fine
 
     reasons = []

src/ocrmypdf/api.py

@@ -21,7 +21,7 @@ import sys
 from contextlib import suppress
 from enum import IntEnum
 from pathlib import Path
-from typing import Dict, List
+from typing import Dict, Iterable
 
 from tqdm import tqdm
 
@@ -67,7 +67,11 @@ class Verbosity(IntEnum):
     debug_all = 2  #: More detailed debugging from ocrmypdf and dependent modules
 
 
-def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=False):
+def configure_logging(
+    verbosity: Verbosity,
+    progress_bar_friendly: bool = True,
+    manage_root_logger: bool = False,
+):
     """Set up logging.
 
     Library users may wish to use this function if they want their log output to be
@@ -128,7 +132,7 @@ def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=
     return log
 
 
-def create_options(*, input_file, output_file, **kwargs):
+def create_options(*, input_file: os.PathLike, output_file: os.PathLike, **kwargs):
     cmdline = []
     deferred = []
 
@@ -150,6 +154,12 @@ def create_options(*, input_file, output_file, **kwargs):
                 cmdline.append(f"--{cmd_style_arg}")
             continue
 
+        if isinstance(val, Iterable) and not isinstance(val, str):
+            for elem in val:
+                cmdline.append(f"--{cmd_style_arg}")
+                cmdline.append(elem)
+            continue
+
         # We have a parameter
         cmdline.append(f"--{cmd_style_arg}")
         if isinstance(val, (int, float)):
@@ -180,7 +190,7 @@ def ocr(  # pylint: disable=unused-argument
     input_file: os.PathLike,
     output_file: os.PathLike,
     *,
-    language: List[str] = None,
+    language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
     sidecar: os.PathLike = None,
@@ -210,7 +220,7 @@ def ocr(  # pylint: disable=unused-argument
     jbig2_page_group_size: int = None,
     pages: str = None,
     max_image_mpixels: float = None,
-    tesseract_config: List[str] = None,
+    tesseract_config: Iterable[str] = None,
     tesseract_pagesegmode: int = None,
     tesseract_oem: int = None,
     pdf_renderer=None,

src/ocrmypdf/exec/__init__.py

@@ -23,7 +23,9 @@ import re
 import shutil
 import sys
 from collections.abc import Mapping
+from distutils.version import LooseVersion
 from functools import lru_cache
+from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError
 from subprocess import run as subprocess_run
 
@@ -137,24 +139,25 @@ def shim_paths_with_program_files(env=None):
     program_files = env.get('PROGRAMFILES', '')
     if not program_files:
         return env.get('PATH', '')
-    paths = []
-    try:
-        for dirname in os.listdir(program_files):
-            if dirname.lower() == 'tesseract-ocr':
-                paths.append(os.path.join(program_files, dirname))
-            elif dirname.lower() == 'gs':
-                try:
-                    latest_gs = max(
-                        os.listdir(os.path.join(program_files, dirname)),
-                        key=lambda d: float(d[2:]),
-                    )
-                except (FileNotFoundError, NotADirectoryError):
-                    continue
-                paths.append(os.path.join(program_files, dirname, latest_gs, 'bin'))
-    except EnvironmentError:
-        pass
-    paths.extend(path for path in os.get_exec_path(env) if path not in set(paths))
-    return os.pathsep.join(paths)
+
+    def path_walker():
+        for path in Path(program_files).iterdir():
+            if not path.is_dir():
+                continue
+            if path.name.lower() == 'tesseract-ocr':
+                yield path
+            elif path.name.lower() == 'gs':
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
+
+    paths = sorted(
+        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
+    )
+    paths.extend(
+        Path(str_path)
+        for str_path in os.get_exec_path(env)
+        if Path(str_path) not in set(paths)
+    )
+    return os.pathsep.join(str(p) for p in paths)
 
 
 missing_program = '''
@@ -232,10 +235,10 @@ def _error_trailer(program, package, **kwargs):
 
 
 def _error_missing_program(program, package, required_for, recommended):
-    if required_for:
+    if recommended:
+        log.warning(missing_recommend_program.format(**locals()))
+    elif required_for:
         log.error(missing_optional_program.format(**locals()))
-    elif recommended:
-        log.info(missing_recommend_program.format(**locals()))
     else:
         log.error(missing_program.format(**locals()))
     _error_trailer(**locals())
@@ -270,7 +273,15 @@ def check_external_program(
             raise MissingDependencyError()
         return
 
-    if found_version < need_version:
+    def remove_leading_v(s):
+        if s.startswith('v'):
+            return s[1:]
+        return s
+
+    found_version = remove_leading_v(found_version)
+    need_version = remove_leading_v(need_version)
+
+    if found_version and LooseVersion(found_version) < LooseVersion(need_version):
         _error_old_version(program, package, need_version, found_version, required_for)
         if not recommended:
             raise MissingDependencyError()

src/ocrmypdf/exec/tesseract.py

@@ -77,20 +77,14 @@ def has_textonly_pdf(tesseract_env=None, langs=None):
     args_tess = tess_base_args(langs, engine_mode=None) + ['--print-parameters', 'pdf']
     params = ''
     try:
-        proc = run(
-            args_tess,
-            check=True,
-            universal_newlines=True,
-            stdout=PIPE,
-            stderr=STDOUT,
-            env=tesseract_env,
-        )
+        # print-parameters can return non-UTF8 if the parameters are so initialized
+        proc = run(args_tess, check=True, stdout=PIPE, stderr=STDOUT, env=tesseract_env)
         params = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(
             "Could not --print-parameters from tesseract"
         ) from e
-    if 'textonly_pdf' in params:
+    if b'textonly_pdf' in params:
         return True
     return False
 

src/ocrmypdf/helpers.py

@@ -28,7 +28,7 @@ from pathlib import Path
 log = logging.getLogger(__name__)
 
 
-def safe_symlink(input_file, soft_link_name, *args, **kwargs):
+def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike, *args, **kwargs):
     """
     Helper function: relinks soft symbolic link if necessary
     """
@@ -76,12 +76,12 @@ def is_iterable_notstr(thing):
     return isinstance(thing, Iterable) and not isinstance(thing, str)
 
 
-def monotonic(L):
+def monotonic(L: Iterable):
     """Does list increase monotonically?"""
     return all(b > a for a, b in zip(L, L[1:]))
 
 
-def page_number(input_file):
+def page_number(input_file: os.PathLike):
     """Get one-based page number implied by filename (000002.pdf -> 2)"""
     return int(os.path.basename(os.fspath(input_file))[0:6])
 
@@ -97,7 +97,7 @@ def available_cpu_count():
     return 1
 
 
-def is_file_writable(test_file):
+def is_file_writable(test_file: os.PathLike):
     """Intentionally racy test if target is writable.
 
     We intend to write to the output file if and only if we succeed and

src/ocrmypdf/optimize.py

@@ -421,9 +421,9 @@ def transcode_pngs(pike, images, image_name_fn, root, log, options):
             )
             continue
         if compdata.type == leptonica.lept.L_FLATE_ENCODE:
-            return rewrite_png(pike, im_obj, compdata, log)
+            rewrite_png(pike, im_obj, compdata, log)
         elif compdata.type == leptonica.lept.L_G4_ENCODE:
-            return rewrite_png_as_g4(pike, im_obj, compdata, log)
+            rewrite_png_as_g4(pike, im_obj, compdata, log)
 
 
 def rewrite_png_as_g4(pike, im_obj, compdata, log):

src/ocrmypdf/pdfinfo/info.py

@@ -22,7 +22,7 @@ from collections import defaultdict, namedtuple
 from decimal import Decimal
 from enum import Enum
 from math import hypot, isclose
-from os import fspath
+from os import PathLike, fspath
 from pathlib import Path
 from warnings import warn
 
@@ -40,7 +40,7 @@ logger = logging.getLogger()
 Colorspace = Enum('Colorspace', 'gray rgb cmyk lab icc index sep devn pattern jpeg2000')
 
 Encoding = Enum(
-    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate ' + 'runlength'
+    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate runlength'
 )
 
 FRIENDLY_COLORSPACE = {
@@ -558,7 +558,7 @@ def simplify_textboxes(miner, textbox_getter):
         yield TextboxInfo(box.bbox, visible, corrupt)
 
 
-def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
+def _pdf_get_pageinfo(pdf, pageno: int, infile: PathLike, xmltext: str):
     pageinfo = {}
     pageinfo['pageno'] = pageno
     pageinfo['images'] = []

src/ocrmypdf/pdfinfo/layout.py

@@ -269,7 +269,9 @@ class TextPositionTracker(PDFLayoutAnalyzer):
 
 def get_page_analysis(infile, pageno, pscript5_mode):
     rman = pdfminer.pdfinterp.PDFResourceManager(caching=True)
-    dev = TextPositionTracker(rman, laparams=LAParams())
+    dev = TextPositionTracker(
+        rman, laparams=LAParams(all_texts=True, detect_vertical=True)
+    )
     interp = pdfminer.pdfinterp.PDFPageInterpreter(rman, dev)
 
     if pscript5_mode:

tests/spoof/tesseract_badutf8.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-# © 2017 James R. Barlow: github.com/jbarlow83
-#
-# Permission is hereby granted, free of charge, to any person obtaining a
-# copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be included
-# in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-import sys
-
-"""Tesseract bad utf8 spoof
-
-In 'hocr' mode or 'pdf' mode, return error code 1 and some non-Unicode
-text because tesseract seems to do that in some cases related to
-language pack version mismatches
-
-"""
-
-
-VERSION_STRING = '''tesseract 4.0.0
- leptonica-1.77.0
-  libjpeg 9c : libpng 1.6.35 : libtiff 4.0.10 : zlib 1.2.11 : libopenjp2 2.3.0
- Found AVX2
- Found AVX
- Found SSE
-SPOOFED
-'''
-
-# Japanese "Invalid UTF-8" encoded in Shift JIS
-BAD_UTF8 = b'\x96\xb3\x8c\xf8\x82\xc8UTF-8\x0a'
-
-
-def main():
-    if sys.argv[1] == '--version':
-        print(VERSION_STRING, file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[1] == '--list-langs':
-        print('List of available languages (1):\neng', file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[-2] == '--print-parameters':
-        print("Some parameters", file=sys.stderr)
-        print("textonly_pdf\t1\tSome help text")
-        sys.exit(0)
-    elif sys.argv[-2] in ('hocr', 'pdf'):
-        sys.stdout.buffer.write(BAD_UTF8)
-        sys.exit(1)
-    elif sys.argv[-1] == 'stdout':
-        # input file is at sys.argv[-2] but we don't look at it
-        print(
-            """Orientation: 0
-Orientation in degrees: 0
-Orientation confidence: 100.00
-Script: 1
-Script confidence: 100.00""",
-            file=sys.stderr,
-        )
-    else:
-        print("Spoof doesn't understand arguments", file=sys.stderr)
-        print(sys.argv, file=sys.stderr)
-        sys.exit(1)
-
-    sys.exit(0)
-
-
-if __name__ == '__main__':
-    main()

tests/test_api.py

@@ -59,3 +59,10 @@ def test_tqdm_console():
 
     log.info("done")
     assert not before_pbar("done")
+
+
+def test_language_list():
+    with pytest.raises(
+        (ocrmypdf.exceptions.InputFileError, ocrmypdf.exceptions.MissingDependencyError)
+    ):
+        ocrmypdf.ocr('doesnotexist.pdf', '_.pdf', language=['eng', 'deu'])

tests/test_helpers.py

@@ -17,6 +17,7 @@
 
 import logging
 import multiprocessing
+import os
 from pathlib import Path
 from unittest.mock import MagicMock
 
@@ -95,3 +96,21 @@ class TestFileIsWritable:
         pathmock.exists.return_value = True
         pathmock.is_file.side_effect = PermissionError
         assert not helpers.is_file_writable(pathmock)
+
+
+def test_shim_paths(tmp_path):
+    progfiles = tmp_path / 'Program Files'
+    progfiles.mkdir()
+    (progfiles / 'tesseract-ocr').mkdir()
+    (progfiles / 'gs' / '9.51' / 'bin').mkdir(parents=True)
+    (progfiles / 'gs' / '9.52' / 'bin').mkdir(parents=True)
+    syspath = tmp_path / 'bin'
+    env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
+    from ocrmypdf.exec import shim_paths_with_program_files
+
+    result_str = shim_paths_with_program_files(env=env)
+    results = result_str.split(os.pathsep)
+    assert results[0].endswith('tesseract-ocr')
+    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
+    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
+    assert results[3] == str(syspath)

tests/test_optimize.py

@@ -18,10 +18,12 @@
 import logging
 from os import fspath
 from pathlib import Path
+from unittest.mock import patch
 
+import img2pdf
 import pikepdf
 import pytest
-from PIL import Image
+from PIL import Image, ImageDraw
 
 from ocrmypdf import optimize as opt
 from ocrmypdf.exec import jbig2enc, pngquant
@@ -130,3 +132,41 @@ def test_flate_to_jbig2(resources, outdir, spoof_tesseract_noop):
     pdf = pikepdf.open(outdir / 'out.pdf')
     pim = pikepdf.PdfImage(next(iter(pdf.pages[0].images.values())))
     assert pim.filters[0] == '/JBIG2Decode'
+
+
+def test_multiple_pngs(resources, outdir, spoof_tesseract_noop):
+    with Path.open(outdir / 'in.pdf', 'wb') as inpdf:
+        img2pdf.convert(
+            fspath(resources / 'baiona_colormapped.png'),
+            fspath(resources / 'baiona_gray.png'),
+            with_pdfrw=False,
+            outputstream=inpdf,
+        )
+
+    def mockquant(input_file, output_file, _quality_min, _quality_max):
+        with Image.open(input_file) as im:
+            draw = ImageDraw.Draw(im)
+            draw.rectangle((0, 0, im.width, im.height), fill=128)
+            im.save(output_file)
+
+    with patch('ocrmypdf.optimize.pngquant.quantize', new=mockquant):
+        check_ocrmypdf(
+            outdir / 'in.pdf',
+            outdir / 'out.pdf',
+            '--optimize',
+            '3',
+            '--jobs',
+            '1',
+            '--use-threads',
+            '--output-type',
+            'pdf',
+            env=spoof_tesseract_noop,
+        )
+
+    with pikepdf.open(outdir / 'in.pdf') as inpdf, pikepdf.open(
+        outdir / 'out.pdf'
+    ) as outpdf:
+        for n in range(len(inpdf.pages)):
+            inim = next(iter(inpdf.pages[n].images.values()))
+            outim = next(iter(outpdf.pages[n].images.values()))
+            assert len(outim.read_raw_bytes()) < len(inim.read_raw_bytes()), n

tests/test_stdio.py

@@ -33,11 +33,6 @@ run_ocrmypdf_api = pytest.helpers.run_ocrmypdf
 spoof = pytest.helpers.spoof
 
 
-@pytest.fixture
-def spoof_tess_bad_utf8(tmp_path_factory):
-    return spoof(tmp_path_factory, tesseract='tesseract_badutf8.py')
-
-
 def test_stdin(spoof_tesseract_noop, ocrmypdf_exec, resources, outpdf):
     input_file = str(resources / 'francais.pdf')
     output_file = str(outpdf)

tests/test_validation.py

@@ -19,6 +19,7 @@ import logging
 import os
 from unittest.mock import patch
 
+import pikepdf
 import pytest
 
 import ocrmypdf._validation as vd
@@ -111,15 +112,20 @@ def test_output_tty():
 def test_report_file_size(tmp_path, caplog):
     in_ = tmp_path / 'a.pdf'
     out = tmp_path / 'b.pdf'
-    in_.write_bytes(b'123')
-    out.write_bytes(b'')
+    pdf = pikepdf.new()
+    pdf.save(in_)
+    pdf.save(out)
     opts = make_opts()
     vd.report_output_file_size(opts, in_, out)
     assert caplog.text == ''
     caplog.clear()
 
-    os.truncate(in_, 25001)
-    os.truncate(out, 50000)
+    waste_of_space = b'Dummy' * 5000
+    pdf.root.Dummy = waste_of_space
+    pdf.save(in_)
+    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.save(out)
+
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(
         'ocrmypdf._validation.pngquant.available', return_value=True
     ):
@@ -176,3 +182,58 @@ def test_language_warning(caplog):
         vd.check_options_languages(opts)
         assert opts.language == ['eng']
         assert 'assuming --language' in caplog.text
+
+
+def test_version_comparison():
+    vd.check_external_program(
+        program="dummy_basic",
+        package="dummy",
+        version_checker=lambda: '9.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="dummy_doubledigit",
+        package="dummy",
+        version_checker=lambda: '10.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '4.0.0-beta.1',
+        need_version='4.0.0',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: 'v5.0.0-alpha.20200201',
+        need_version='4.0.0',
+    )
+    with pytest.raises(MissingDependencyError):
+        vd.check_external_program(
+            program="dummy_fails",
+            package="dummy",
+            version_checker=lambda: '1.0',
+            need_version='2.0',
+        )
+
+
+def test_optional_program_recommended(caplog):
+    caplog.clear()
+
+    def raiser():
+        raise FileNotFoundError('jbig2')
+
+    with caplog.at_level(logging.WARNING):
+        vd.check_external_program(
+            program="jbig2",
+            package="jbig2enc",
+            version_checker=raiser,
+            need_version='42',
+            required_for='this test case',
+            recommended=True,
+        )
+        assert any(
+            (loglevel == logging.WARNING and "recommended" in msg)
+            for _logger_name, loglevel, msg in caplog.record_tuples
+        )