pytest picky about list vs tuple

Fixes #523

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+What command line or API call were you trying to run?
+
+```bash
+ocrmypdf  ...arguments... input.pdf output.pdf
+```
+
+Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
+
+**Example file**
+Please include an example *input* PDF (or image). The input file is more helpful.
+
+If possible, use an input file with no personal or confidential information. At your option you may GPG-encrypt the file for OCRmyPDF's author only.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System**
+ - OS: [e.g. Linux, Windows, macOS]
+ - OCRmyPDF Version: ``ocrmypdf --version``
+ - How did you install ocrmypdf? Did you use a system package manager, `pip`, or a Docker image?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

docs/api.rst

@@ -51,6 +51,14 @@ Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
 OCRmyPDF does not.
 
+.. warning::
+
+    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    of these steps, Windows fork semantics will prevent OCRmyPDF from working
+    correct.
+
 Logging
 -------
 

docs/batch.rst

@@ -99,11 +99,45 @@ and all inquiries are appreciated.
 Hot (watched) folders
 =====================
 
+Watched folders with watcher.py
+-------------------------------
+
+OCRmyPDF has a folder watcher called watcher.py, which is currently included in source
+distributions but not part of the main program. It may be used natively or may run
+in a Docker container. Native instances tend to give better performance. watcher.py
+works on all platforms.
+
+Users may need to customize the script to meet their requirements.
+
+.. code-block:: bash
+
+    pip3 install -r requirements/watcher.txt
+
+    env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
+        OCR_OUTPUT_DIRECTORY=/mnt/output-pdfs \
+        OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
+        python3 watcher.py
+
+.. csv-table:: watcher.py environment variables
+    :header: "Environment variable", "Description"
+    :widths: 50, 50
+
+    "OCR_INPUT_DIRECTORY", "Set input directory to monitor (recursive)"
+    "OCR_OUTPUT_DIRECTORY", "Set output directory (should not be under input)"
+    "OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
+    "OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
+    "OCR_DESKEW", "Apply deskew to crooked input PDFs"
+    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``"
+    "OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
+    "OCR_LOGLEVEL", "Level of log messages to report"
+
+One could configure a networked scanner or scanning computer to drop files in the
+watched folder.
+
 Watched folders with Docker
 ---------------------------
 
-The OCRmyPDF Docker image includes a watcher service. This service can
-be launched as follows:
+The watcher service is included in the OCRmyPDF Docker image. To run it:
 
 .. code-block:: bash
 
@@ -127,9 +161,9 @@ convert it to a OCRed PDF in ``/output/``. The parameters to this image are:
 
     "``-v <path to files to convert>:/input``", "Files placed in this location will be OCRed"
     "``-v <path to store results>:/output``", "This is where OCRed files will be stored"
-    "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "This will place files in the output in {output}/{year}/{month}/{filename}"
-    "``-e OCR_ON_SUCCESS_DELETE=1``", "This will delete the input file if the exit code is 0 (OK)"
-    "``-e OCR_DESKEW=1``", "This will enable deskew for crooked PDFs"
+    "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "Define environment variable OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1"
+    "``-e OCR_ON_SUCCESS_DELETE=1``", "Define environment variable"
+    "``-e OCR_DESKEW=1``", "Define environment variable"
     "``-e PYTHONBUFFERED=1``", "This will force STDOUT to be unbuffered and allow you to see messages in docker logs"
 
 This service relies on polling to check for changes to the filesystem. It
@@ -143,56 +177,6 @@ service is always available.
     :language: yaml
     :caption: misc/docker-compose.example.yml
 
-Watched folders with watcher.py
--------------------------------
-
-The watcher service may also be run natively, without Docker:
-
-.. code-block:: bash
-
-    pip3 install -r requirements/watcher.txt
-
-    env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
-        OCR_OUTPUT_DIRECTORY=/mnt/output-pdfs \
-        OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
-        python3 watcher.py
-
-Watched folders with CLI
-------------------------
-
-To set up a "hot folder" that will trigger OCR for every file inserted,
-use a program like Python
-`watchdog <https://pypi.python.org/pypi/watchdog>`__ (supports all major
-OS).
-
-One could then configure a scanner to automatically place scanned files
-in a hot folder, so that they will be queued for OCR and copied to the
-destination.
-
-.. code-block:: bash
-
-   pip install watchdog
-
-watchdog installs the command line program ``watchmedo``, which can be
-told to run ``ocrmypdf`` on any .pdf added to the current directory
-(``.``) and place the result in the previously created ``out/`` folder.
-
-.. code-block:: bash
-
-   cd hot-folder
-   mkdir out
-   watchmedo shell-command \
-       --patterns="*.pdf" \
-       --ignore-directories \
-       --command='ocrmypdf "${watch_src_path}" "out/${watch_src_path}" ' \
-       .  # don't forget the final dot
-
-On file servers, you could configure watchmedo as a system service so it
-will run all the time.
-
-For more complex behavior you can write a Python script around to use
-the watchdog API. You can refer to the watcher.py script as an example.
-
 Caveats
 -------
 
@@ -218,6 +202,13 @@ Alternatives
 -  `Watchman <https://facebook.github.io/watchman/>`__ is a more
    powerful alternative to ``watchmedo``.
 
+AWS Lambda is not viable
+------------------------
+
+AWS Lambda and its equivalents have low limits on execution time and payload
+size, relative to OCRmyPDF's needs. As of this writing, the request/response
+payload for AWS Lambda was 6 MB, which means many PDFs will not fit.
+
 macOS Automator
 ===============
 

docs/docker.rst

@@ -90,10 +90,8 @@ Docker volume:
 
 .. code-block:: bash
 
-   docker run --rm -v $(pwd):/data ocrmypdf /data/input.pdf /data/output.pdf
-
-(However, when done this way, ``output.pdf`` may be owned by the root
-user.)
+   alias docker_ocrmypdf='docker run --rm  -i --user "$(id -u):$(id -g)" --workdir /data -v "$PWD:/data" ocrmypdf'
+   docker_ocrmypdf /data/input.pdf /data/output.pdf
 
 .. _docker-lang-packs:
 

docs/installation.rst

@@ -232,7 +232,7 @@ environment variable contains ``$HOME/.local/bin``.
 .. code-block:: bash
 
     export PATH=$HOME/.local/bin:$PATH
-    pip3 install --user ocrmypdf
+    pip3.6 install --user ocrmypdf
 
 To add JBIG2 encoding, see :ref:`jbig2`.
 
@@ -574,7 +574,7 @@ Installing on Windows Subsystem for Linux
 
 .. code-block:: powershell
 
-    wsl sudo ln -s  /home/user/.local/bin/ocrmypdf /usr/local/bin/ocrmypdf
+    wsl sudo ln -s  /home/$USER/.local/bin/ocrmypdf /usr/local/bin/ocrmypdf
 
 Then confirm that the expected version from PyPI (|latest|) is installed:
 

docs/release_notes.rst

@@ -13,6 +13,35 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.7.2
+======
+
+-  Fixed an issue with ``ocrmypdf.ocr(...language=)`` not accepting a list of
+   languages as documented.
+-  Updated setup.py to confirm that pdfminer.six version 20200402 is supported.
+
+v9.7.1
+======
+
+-  Fixed version check failing when used with qpdf 10.0.0.
+-  Added some missing type annotations.
+-  Updated documentation to warn about need for "ifmain" guard and Windows.
+
+v9.7.0
+======
+
+-  Fixed an error in watcher.py if ``OCR_JSON_SETTINGS`` was not defined.
+-  Ghostscript 9.51 is now blacklisted, due to numerous problems with this version.
+-  Added a workaround for a problem with "txtwrite" in Ghostscript 9.52.
+-  Fixed an issue where the incorrect number of threads used was shown when
+   ``OMP_THREAD_LIMIT`` was manipulated.
+-  Removed a possible performance bottlenecks for files that use hundreds to
+   thousands of images on the same page.
+-  Documentation improvements.
+-  Optimization will now be applied to some monochrome images that have a color
+   profile defined instead of only black and white.
+-  ICC profiles are consulted when determining the simplified colorspace of an
+   image.
 
 v9.6.1
 ======

misc/watcher.py

@@ -14,8 +14,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import json
 import logging
 import os
+import sys
 import time
 from datetime import datetime
 from pathlib import Path
@@ -33,6 +35,7 @@ OUTPUT_DIRECTORY = os.getenv('OCR_OUTPUT_DIRECTORY', '/output')
 OUTPUT_DIRECTORY_YEAR_MONTH = bool(os.getenv('OCR_OUTPUT_DIRECTORY_YEAR_MONTH', False))
 ON_SUCCESS_DELETE = bool(os.getenv('OCR_ON_SUCCESS_DELETE', False))
 DESKEW = bool(os.getenv('OCR_DESKEW', False))
+OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = os.getenv('OCR_POLL_NEW_FILE_SECONDS', 1)
 LOGLEVEL = os.environ.get('OCR_LOGLEVEL', 'INFO').upper()
 PATTERNS = ['*.pdf']
@@ -87,7 +90,10 @@ def execute_ocrmypdf(file_path):
         return
     log.info(f'Attempting to OCRmyPDF to: {output_path}')
     exit_code = ocrmypdf.ocr(
-        input_file=file_path, output_file=output_path, deskew=DESKEW
+        input_file=file_path,
+        output_file=output_path,
+        deskew=DESKEW,
+        **OCR_JSON_SETTINGS,
     )
     if exit_code == 0 and ON_SUCCESS_DELETE:
         log.info(f'OCR is done. Deleting: {file_path}')
@@ -118,10 +124,15 @@ def main():
         f"OUTPUT_DIRECTORY_YEAR_MONTH: {OUTPUT_DIRECTORY_YEAR_MONTH}\n"
         f"ON_SUCCESS_DELETE: {ON_SUCCESS_DELETE}\n"
         f"DESKEW: {DESKEW}\n"
+        f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
         f"LOGLEVEL: {LOGLEVEL}\n"
     )
 
+    if 'input_file' in OCR_JSON_SETTINGS or 'output_file' in OCR_JSON_SETTINGS:
+        log.error('OCR_JSON_SETTINGS should not specify input file or output file')
+        sys.exit(1)
+
     handler = HandleObserverEvent(patterns=PATTERNS)
     observer = Observer()
     observer.schedule(handler, INPUT_DIRECTORY, recursive=True)

requirements/main.txt

@@ -3,8 +3,8 @@
 # installation
 cffi == 1.14.0
 img2pdf == 0.3.3
-pdfminer.six == 20200124
-pikepdf == 1.10.1
+pdfminer.six == 20200402
+pikepdf == 1.10.2
 Pillow == 7.0.0
 reportlab == 3.5.34
 tqdm == 4.42.1

setup.py

@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200124',
+        'pdfminer.six >= 20181108, <= 20200402',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/_sync.py

@@ -245,6 +245,10 @@ def exec_concurrent(context):
     if context.options.tesseract_env is None:
         context.options.tesseract_env = os.environ.copy()
     context.options.tesseract_env.setdefault('OMP_THREAD_LIMIT', str(tess_threads))
+    try:
+        tess_threads = int(context.options.tesseract_env['OMP_THREAD_LIMIT'])
+    except ValueError:  # OMP_THREAD_LIMIT initialized to non-numeric
+        context.log.error("Environment variable OMP_THREAD_LIMIT is not numeric")
     if tess_threads > 1:
         context.log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
 
@@ -357,10 +361,11 @@ def run_pipeline(options, api=False):
         options.jobs = available_cpu_count()
 
     work_folder = mkdtemp(prefix="com.github.ocrmypdf.")
+    debug_log_handler = None
     if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
         'PYTEST_CURRENT_TEST', ''
     ):
-        configure_debug_logging(Path(work_folder) / "debug.log")
+        debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
 
     try:
         check_requested_output_file(options)
@@ -428,6 +433,12 @@ def run_pipeline(options, api=False):
         log.exception("An exception occurred while executing the pipeline")
         return ExitCode.other_error
     finally:
+        if debug_log_handler:
+            try:
+                debug_log_handler.close()
+                log.removeHandler(debug_log_handler)
+            except EnvironmentError as e:
+                print(e, file=sys.stderr)
         cleanup_working_files(work_folder, options)
 
     return ExitCode.ok

src/ocrmypdf/_validation.py

@@ -466,11 +466,12 @@ def check_dependency_versions(options):
         version_checker=ghostscript.version,
         need_version='9.15',  # limited by Travis CI / Ubuntu 14.04 backports
     )
-    if ghostscript.version() == '9.24':
+    gs_version = ghostscript.version()
+    if gs_version in ('9.24', '9.51'):
         raise MissingDependencyError(
-            "Ghostscript 9.24 contains serious regressions and is not "
-            "supported. Please upgrade to Ghostscript 9.25 or use an older "
-            "version."
+            f"Ghostscript {gs_version} contains serious regressions and is not "
+            "supported. Please upgrade to a newer version, or downgrade to the "
+            "previous version."
         )
     check_external_program(
         program='qpdf',

src/ocrmypdf/api.py

@@ -21,7 +21,7 @@ import sys
 from contextlib import suppress
 from enum import IntEnum
 from pathlib import Path
-from typing import Dict, List
+from typing import Dict, Iterable
 
 from tqdm import tqdm
 
@@ -67,7 +67,11 @@ class Verbosity(IntEnum):
     debug_all = 2  #: More detailed debugging from ocrmypdf and dependent modules
 
 
-def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=False):
+def configure_logging(
+    verbosity: Verbosity,
+    progress_bar_friendly: bool = True,
+    manage_root_logger: bool = False,
+):
     """Set up logging.
 
     Library users may wish to use this function if they want their log output to be
@@ -128,7 +132,7 @@ def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=
     return log
 
 
-def create_options(*, input_file, output_file, **kwargs):
+def create_options(*, input_file: os.PathLike, output_file: os.PathLike, **kwargs):
     cmdline = []
     deferred = []
 
@@ -150,6 +154,12 @@ def create_options(*, input_file, output_file, **kwargs):
                 cmdline.append(f"--{cmd_style_arg}")
             continue
 
+        if isinstance(val, Iterable) and not isinstance(val, str):
+            for elem in val:
+                cmdline.append(f"--{cmd_style_arg}")
+                cmdline.append(elem)
+            continue
+
         # We have a parameter
         cmdline.append(f"--{cmd_style_arg}")
         if isinstance(val, (int, float)):
@@ -180,7 +190,7 @@ def ocr(  # pylint: disable=unused-argument
     input_file: os.PathLike,
     output_file: os.PathLike,
     *,
-    language: List[str] = None,
+    language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
     sidecar: os.PathLike = None,
@@ -210,7 +220,7 @@ def ocr(  # pylint: disable=unused-argument
     jbig2_page_group_size: int = None,
     pages: str = None,
     max_image_mpixels: float = None,
-    tesseract_config: List[str] = None,
+    tesseract_config: Iterable[str] = None,
     tesseract_pagesegmode: int = None,
     tesseract_oem: int = None,
     pdf_renderer=None,

src/ocrmypdf/cli.py

@@ -20,6 +20,8 @@ import argparse
 from ._version import PROGRAM_NAME as _PROGRAM_NAME
 from ._version import __version__ as _VERSION
 
+__all__ = ['parser']
+
 
 def numeric(basetype, min_=None, max_=None):
     """Validator for numeric params"""

src/ocrmypdf/exec/__init__.py

@@ -23,6 +23,7 @@ import re
 import shutil
 import sys
 from collections.abc import Mapping
+from distutils.version import LooseVersion
 from functools import lru_cache
 from subprocess import PIPE, STDOUT, CalledProcessError
 from subprocess import run as subprocess_run
@@ -270,7 +271,15 @@ def check_external_program(
             raise MissingDependencyError()
         return
 
-    if found_version < need_version:
+    def remove_leading_v(s):
+        if s.startswith('v'):
+            return s[1:]
+        return s
+
+    found_version = remove_leading_v(found_version)
+    need_version = remove_leading_v(need_version)
+
+    if LooseVersion(found_version) < LooseVersion(need_version):
         _error_old_version(program, package, need_version, found_version, required_for)
         if not recommended:
             raise MissingDependencyError()

src/ocrmypdf/helpers.py

@@ -28,7 +28,7 @@ from pathlib import Path
 log = logging.getLogger(__name__)
 
 
-def safe_symlink(input_file, soft_link_name, *args, **kwargs):
+def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike, *args, **kwargs):
     """
     Helper function: relinks soft symbolic link if necessary
     """
@@ -76,12 +76,12 @@ def is_iterable_notstr(thing):
     return isinstance(thing, Iterable) and not isinstance(thing, str)
 
 
-def monotonic(L):
+def monotonic(L: Iterable):
     """Does list increase monotonically?"""
     return all(b > a for a, b in zip(L, L[1:]))
 
 
-def page_number(input_file):
+def page_number(input_file: os.PathLike):
     """Get one-based page number implied by filename (000002.pdf -> 2)"""
     return int(os.path.basename(os.fspath(input_file))[0:6])
 
@@ -97,7 +97,7 @@ def available_cpu_count():
     return 1
 
 
-def is_file_writable(test_file):
+def is_file_writable(test_file: os.PathLike):
     """Intentionally racy test if target is writable.
 
     We intend to write to the output file if and only if we succeed and

src/ocrmypdf/lib/compile_leptonica.py

@@ -443,6 +443,12 @@ pixReadBarcodes(PIXA     *pixa,
                 SARRAY  **psaw,
                 l_int32 debugflag);
 
+PIX *
+pixGenHalftoneMask(PIX      *pixs,
+                        PIX     **ppixtext,
+                        l_int32  *phtfound,
+                        PIXA     *pixadb);
+
 l_int32
 l_generateCIDataForPdf(const char *fname,
                        PIX *pix,

src/ocrmypdf/optimize.py

@@ -155,6 +155,17 @@ def extract_image_generic(*, pike, root, log, image, xref, options):
         # generating a PNG from compressed data
         pim.as_pil_image().save(png_name(root, xref))
         return xref, '.png'
+    elif (
+        not pim.indexed
+        and pim.colorspace == Name.ICCBased
+        and pim.bits_per_component == 1
+        and not options.jbig2_lossy
+    ):
+        # We can losslessly optimize 1-bit images to CCITT or JBIG2 without
+        # paying any attention to the ICC profile, provided we're not doing
+        # lossy JBIG2
+        pim.as_pil_image().save(png_name(root, xref))
+        return xref, '.png'
 
     return None
 

src/ocrmypdf/pdfinfo/info.py

@@ -18,11 +18,11 @@
 
 import logging
 import re
-from collections import namedtuple
+from collections import defaultdict, namedtuple
 from decimal import Decimal
 from enum import Enum
 from math import hypot, isclose
-from os import fspath
+from os import PathLike, fspath
 from pathlib import Path
 from warnings import warn
 
@@ -30,17 +30,17 @@ import pikepdf
 from pikepdf import PdfMatrix
 from tqdm import tqdm
 
-from ocrmypdf.exceptions import EncryptedPdfError, MissingDependencyError
-
-from . import ghosttext
-from .layout import get_page_analysis, get_text_boxes
+from ocrmypdf.exceptions import EncryptedPdfError
+from ocrmypdf.exec import ghostscript
+from ocrmypdf.pdfinfo import ghosttext
+from ocrmypdf.pdfinfo.layout import get_page_analysis, get_text_boxes
 
 logger = logging.getLogger()
 
 Colorspace = Enum('Colorspace', 'gray rgb cmyk lab icc index sep devn pattern jpeg2000')
 
 Encoding = Enum(
-    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate ' + 'runlength'
+    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate runlength'
 )
 
 FRIENDLY_COLORSPACE = {
@@ -98,7 +98,7 @@ XobjectSettings = namedtuple('XobjectSettings', ['name', 'shorthand', 'stack_dep
 InlineSettings = namedtuple('InlineSettings', ['iimage', 'shorthand', 'stack_depth'])
 
 ContentsInfo = namedtuple(
-    'ContentsInfo', ['xobject_settings', 'inline_images', 'found_vector']
+    'ContentsInfo', ['xobject_settings', 'inline_images', 'found_vector', 'name_index']
 )
 
 TextboxInfo = namedtuple('TextboxInfo', ['bbox', 'is_visible', 'is_corrupt'])
@@ -151,6 +151,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
     ctm = PdfMatrix(initial_shorthand)
     xobject_settings = []
     inline_images = []
+    name_index = defaultdict(lambda: [])
     found_vector = False
     vector_ops = set('S s f F f* B B* b b*'.split())
     image_ops = set('BI ID EI q Q Do cm'.split())
@@ -185,6 +186,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
                 name=image_name, shorthand=ctm.shorthand, stack_depth=len(stack)
             )
             xobject_settings.append(settings)
+            name_index[image_name].append(settings)
         elif operator == 'INLINE IMAGE':  # BI/ID/EI are grouped into this
             iimage = operands[0]
             inline = InlineSettings(
@@ -198,6 +200,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
         xobject_settings=xobject_settings,
         inline_images=inline_images,
         found_vector=found_vector,
+        name_index=name_index,
     )
 
 
@@ -303,14 +306,22 @@ class ImageInfo:
         if self._enc == Encoding.jpeg2000:
             self._color = Colorspace.jpeg2000
 
-        self._comp = FRIENDLY_COMP.get(self._color, '?')
+        if self._color == Colorspace.icc:
+            # Check the ICC profile to determine actual colorspace
+            pim_icc = pim.icc
+            if pim_icc.profile.xcolor_space == 'GRAY':
+                self._comp = 1
+            elif pim_icc.profile.xcolor_space == 'CMYK':
+                self._comp = 4
+            else:
+                self._comp = 3
+        else:
+            self._comp = FRIENDLY_COMP.get(self._color, '?')
 
-        # Bit of a hack... infer grayscale if component count is uncertain
-        # but encoding must be monochrome. This happens if a monochrome image
-        # has an ICC profile attached. Better solution would be to examine
-        # the ICC profile.
-        if self._comp == '?' and self._enc in (Encoding.ccitt, Encoding.jbig2):
-            self._comp = FRIENDLY_COMP[Colorspace.gray]
+            # Bit of a hack... infer grayscale if component count is uncertain
+            # but encoding only supports monochrome.
+            if self._comp == '?' and self._enc in (Encoding.ccitt, Encoding.jbig2):
+                self._comp = FRIENDLY_COMP[Colorspace.gray]
 
     @property
     def name(self):
@@ -411,13 +422,9 @@ def _find_regular_images(container, contentsinfo):
     """
 
     for pdfimage, xobj in _image_xobjects(container):
-
-        # For each image that is drawn on this, check if we drawing the
-        # current image - yes this is O(n^2), but n == 1 almost always
-        for draw in contentsinfo.xobject_settings:
-            if draw.name != xobj:
-                continue
-
+        if xobj not in contentsinfo.name_index:
+            continue
+        for draw in contentsinfo.name_index[xobj]:
             if draw.stack_depth == 0 and _is_unit_square(draw.shorthand):
                 # At least one PDF in the wild (and test suite) draws an image
                 # when the graphics stack depth is 0, meaning that the image
@@ -551,7 +558,7 @@ def simplify_textboxes(miner, textbox_getter):
         yield TextboxInfo(box.bbox, visible, corrupt)
 
 
-def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
+def _pdf_get_pageinfo(pdf, pageno: int, infile: PathLike, xmltext: str):
     pageinfo = {}
     pageinfo['pageno'] = pageno
     pageinfo['images'] = []
@@ -611,25 +618,28 @@ def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
 
 def _pdf_get_all_pageinfo(infile, detailed_analysis=False, log=None, progbar=False):
     pdf = pikepdf.open(infile)  # Do not close in this function
-    if pdf.is_encrypted:
-        pdf.close()
-        raise EncryptedPdfError()  # Triggered by encryption with empty passwd
-    if detailed_analysis:
-        pages_xml = None
-    else:
-        pages_xml = ghosttext.extract_text_xml(infile, pdf, pageno=None, log=log)
+    try:
+        if pdf.is_encrypted:
+            raise EncryptedPdfError()  # Triggered by encryption with empty passwd
+        if detailed_analysis:
+            pages_xml = None
+        else:
+            pages_xml = ghosttext.extract_text_xml(infile, pdf, pageno=None, log=log)
 
-    pages = []
-    for n, _ in tqdm(
-        enumerate(pdf.pages),
-        total=len(pdf.pages),
-        desc="Scan",
-        unit='page',
-        disable=not progbar,
-    ):
-        page_xml = pages_xml[n] if pages_xml else None
-        page = PageInfo(pdf, n, infile, page_xml, detailed_analysis)
-        pages.append(page)
+        pages = []
+        for n, _ in tqdm(
+            enumerate(pdf.pages),
+            total=len(pdf.pages),
+            desc="Scan",
+            unit='page',
+            disable=not progbar,
+        ):
+            page_xml = pages_xml[n] if pages_xml else None
+            page = PageInfo(pdf, n, infile, page_xml, detailed_analysis)
+            pages.append(page)
+    except Exception:
+        pdf.close()
+        raise
 
     return pages, pdf
 
@@ -750,6 +760,8 @@ class PdfInfo:
 
     def __init__(self, infile, detailed_page_analysis=False, log=logger, progbar=False):
         self._infile = infile
+        if ghostscript.version() in ('9.52',):
+            detailed_page_analysis = True  # txtwrite doesn't work in these versions
         self._pages, pdf = _pdf_get_all_pageinfo(
             infile, detailed_page_analysis, log=log, progbar=progbar
         )
@@ -805,10 +817,14 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('infile')
     args = parser.parse_args()
-    info = _pdf_get_all_pageinfo(args.infile)
+    pagesinfo, pdfinfo = _pdf_get_all_pageinfo(args.infile)
     from pprint import pprint
 
-    pprint(info)
+    pprint(pdfinfo)
+    for page in pagesinfo:
+        pprint(page)
+        for im in page.images:
+            pprint(im)
 
 
 if __name__ == '__main__':

tests/conftest.py

@@ -97,10 +97,48 @@ assert ast.parse(WINDOWS_SHIM_TEMPLATE.format(spoofer=repr(r"C:\\Temp\\file.py")
 def spoof(tmp_path_factory, **kwargs):
     """Modify PATH to override subprocess executables
 
-    spoof(program1='replacement', ...)
+    spoof(tmp_path_factory, program1='replacement', ...)
 
-    Creates temporary directory with symlinks to targets.
+    For the test suite we need a way override executables, so that we can
+    substitute desired results such as errors or just speed up OCR.
 
+    On POSIXish platforms we create a temporary folder with overrides that
+    are symlinks to the executables we want to run. We do not actually override
+    PATH. We also set an environment variable _OCRMYPDF_TEST_PATH, which
+    OCRmyPDF's subprocess wrapper will check before they use regular PATH. The
+    output is a folder full of executables we are overriding. We can override
+    multiple executables. The end result is a folder we can use in a PATH-style
+    lookup to override some executables:
+
+        /tmp/abcxyz/tesseract -> ocrmypdf/tests/resources/spoof/tesseract_crash.py
+        /tmp/abcxyz/gs -> ocrmypdf/tests/resources/spoof/gs_backflip.py
+
+    Windows needs extra help from us because usually, only the Administrator
+    can create symlinks. Instead we create small Python scripts that call
+    the programs we want, implementing the effect of a symlink. This is cleaner
+    than creating Windows executables or trying to use non-Python scripts.
+    The temporary folder generated for Windows could like:
+
+        %TEMP%\abcxyz\tesseract.py:
+            (script that runs ocrmypdf/tests/resources/spoof/tesseract_crash.py)
+        %TEMP%\abcxyz\gswin32c.py:
+            (script that runs ocrmypdf/tests/resources/spoof/gs_backflip.py)
+        %TEMP%\abcxyz\gswin64c.py:
+            (script that runs ocrmypdf/tests/resources/spoof/gs_backflip.py)
+
+    We also address one quirk here, that Ghostscript may be known as gswin32c
+    or gswin64c, depending on what the user installed (regardless of Windows
+    itself). On POSIX, Ghostscript is just 'gs'.  We handle the special case here
+    too.
+
+    All of this is intimately dependent on the machinery in ocrmypdf.exec.run().
+    In particular, for Windows, that code has to know that if there is a .py
+    file, it needs to run it with Python, since Windows does not like being
+    asked to execute files.
+
+    We don't overload PATH directly because we have some tests where we call
+    ocrmypdf as a subprocess (to exercise the command line interface) and some
+    tests where we call it as an API.
     """
     env = os.environ.copy()
     slug = '-'.join(v.replace('.py', '') for v in sorted(kwargs.values()))

tests/test_api.py

@@ -59,3 +59,10 @@ def test_tqdm_console():
 
     log.info("done")
     assert not before_pbar("done")
+
+
+def test_language_list():
+    with pytest.raises(
+        (ocrmypdf.exceptions.InputFileError, ocrmypdf.exceptions.MissingDependencyError)
+    ):
+        ocrmypdf.ocr('doesnotexist.pdf', '_.pdf', language=['eng', 'deu'])

tests/test_graft.py

@@ -25,25 +25,30 @@ import ocrmypdf
 
 
 def test_no_glyphless_graft(resources, outdir):
-    pdf = pikepdf.open(resources / 'francais.pdf')
-    pdf_aspect = pikepdf.open(resources / 'aspect.pdf')
-    pdf_cmyk = pikepdf.open(resources / 'cmyk.pdf')
-    pdf.pages.extend(pdf_aspect.pages)
-    pdf.pages.extend(pdf_cmyk.pages)
-    pdf.save(outdir / 'test.pdf')
+    with pikepdf.open(resources / 'francais.pdf') as pdf, pikepdf.open(
+        resources / 'aspect.pdf'
+    ) as pdf_aspect, pikepdf.open(resources / 'cmyk.pdf') as pdf_cmyk:
+        pdf.pages.extend(pdf_aspect.pages)
+        pdf.pages.extend(pdf_cmyk.pages)
+        pdf.save(outdir / 'test.pdf')
 
     with patch('ocrmypdf._graft.MAX_REPLACE_PAGES', 2):
         ocrmypdf.ocr(
-            outdir / 'test.pdf', outdir / 'out.pdf', deskew=True, tesseract_timeout=0
+            outdir / 'test.pdf',
+            outdir / 'out.pdf',
+            deskew=True,
+            tesseract_timeout=0,
+            force_ocr=True,
         )
+    # This test needs asserts
 
 
 def test_links(resources, outpdf):
     ocrmypdf.ocr(
         resources / 'link.pdf', outpdf, redo_ocr=True, oversample=200, output_type='pdf'
     )
-    pdf = pikepdf.open(outpdf)
-    p1 = pdf.pages[0]
-    p2 = pdf.pages[1]
-    assert p1.Annots[0].A.D[0].objgen == p2.objgen
-    assert p2.Annots[0].A.D[0].objgen == p1.objgen
+    with pikepdf.open(outpdf) as pdf:
+        p1 = pdf.pages[0]
+        p2 = pdf.pages[1]
+        assert p1.Annots[0].A.D[0].objgen == p2.objgen
+        assert p2.Annots[0].A.D[0].objgen == p1.objgen

tests/test_pdfinfo.py

@@ -26,6 +26,7 @@ from PIL import Image
 from reportlab.pdfgen.canvas import Canvas
 
 from ocrmypdf import pdfinfo
+from ocrmypdf.exec import ghostscript
 from ocrmypdf.pdfinfo import Colorspace, Encoding
 
 # pylint: disable=protected-access
@@ -183,6 +184,9 @@ def test_ocr_detection(resources):
 @pytest.mark.parametrize(
     'testfile', ('truetype_font_nomapping.pdf', 'type3_font_nomapping.pdf')
 )
+@pytest.mark.xfail(
+    ghostscript.version() in ('9.52',), reason="gs 9.52 txtwrite doesn't work"
+)
 def test_corrupt_font_detection(resources, testfile):
     filename = resources / testfile
     with pytest.raises(NotImplementedError):

tests/test_validation.py

@@ -176,3 +176,37 @@ def test_language_warning(caplog):
         vd.check_options_languages(opts)
         assert opts.language == ['eng']
         assert 'assuming --language' in caplog.text
+
+
+def test_version_comparison():
+    vd.check_external_program(
+        program="dummy_basic",
+        package="dummy",
+        version_checker=lambda: '9.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="dummy_doubledigit",
+        package="dummy",
+        version_checker=lambda: '10.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '4.0.0-beta.1',
+        need_version='4.0.0',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: 'v5.0.0-alpha.20200201',
+        need_version='4.0.0',
+    )
+    with pytest.raises(MissingDependencyError):
+        vd.check_external_program(
+            program="dummy_fails",
+            package="dummy",
+            version_checker=lambda: '1.0',
+            need_version='2.0',
+        )