v9.8.2 release notes

Use more realistic test data

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,40 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+What command line or API call were you trying to run?
+
+```bash
+ocrmypdf  ...arguments... input.pdf output.pdf
+```
+
+Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
+
+**Example file**
+Include an input PDF or image that demonstrates your issue.
+
+Please provide an input file with no personal or confidential information. At your option you may `GPG-encrypt the file <https://github.com/jbarlow83/OCRmyPDF/wiki>` for OCRmyPDF's author only.
+
+Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
+
+(Exceptions: Issues with installation, command line argument parsing, test suite failures.Issues without example files usually cannot be resolved.)
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System**
+ - OS: [e.g. Linux, Windows, macOS]
+ - OCRmyPDF Version: ``ocrmypdf --version``
+ - How did you install ocrmypdf? Did you use a system package manager, `pip`, or a Docker image?

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,17 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

azure-pipelines.yml

@@ -153,12 +153,13 @@ stages:
           matrix:
             Python37:
               python.version: "3.7"
-            Python38:
-              python.version: "3.8"
+            # Python38:
+            #   python.version: "3.8"
         steps:
-          - task: UsePythonVersion@0
-            inputs:
-              versionSpec: "$(python.version)"
+          # https://github.com/actions/virtual-environments/issues/664
+          # - task: UsePythonVersion@0
+          #   inputs:
+          #     versionSpec: "$(python.version)"
           - bash: |
               brew update
               brew unlink python@2
@@ -169,6 +170,7 @@ stages:
                 leptonica \
                 openjpeg \
                 pngquant \
+                python \
                 tesseract \
                 unpaper
             displayName: "Install system packages"

docs/api.rst

@@ -51,6 +51,14 @@ Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
 OCRmyPDF does not.
 
+.. warning::
+
+    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
+    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
+    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
+    of these steps, Windows fork semantics will prevent OCRmyPDF from working
+    correct.
+
 Logging
 -------
 

docs/batch.rst

@@ -99,11 +99,45 @@ and all inquiries are appreciated.
 Hot (watched) folders
 =====================
 
+Watched folders with watcher.py
+-------------------------------
+
+OCRmyPDF has a folder watcher called watcher.py, which is currently included in source
+distributions but not part of the main program. It may be used natively or may run
+in a Docker container. Native instances tend to give better performance. watcher.py
+works on all platforms.
+
+Users may need to customize the script to meet their requirements.
+
+.. code-block:: bash
+
+    pip3 install -r requirements/watcher.txt
+
+    env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
+        OCR_OUTPUT_DIRECTORY=/mnt/output-pdfs \
+        OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
+        python3 watcher.py
+
+.. csv-table:: watcher.py environment variables
+    :header: "Environment variable", "Description"
+    :widths: 50, 50
+
+    "OCR_INPUT_DIRECTORY", "Set input directory to monitor (recursive)"
+    "OCR_OUTPUT_DIRECTORY", "Set output directory (should not be under input)"
+    "OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
+    "OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
+    "OCR_DESKEW", "Apply deskew to crooked input PDFs"
+    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``"
+    "OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
+    "OCR_LOGLEVEL", "Level of log messages to report"
+
+One could configure a networked scanner or scanning computer to drop files in the
+watched folder.
+
 Watched folders with Docker
 ---------------------------
 
-The OCRmyPDF Docker image includes a watcher service. This service can
-be launched as follows:
+The watcher service is included in the OCRmyPDF Docker image. To run it:
 
 .. code-block:: bash
 
@@ -127,9 +161,9 @@ convert it to a OCRed PDF in ``/output/``. The parameters to this image are:
 
     "``-v <path to files to convert>:/input``", "Files placed in this location will be OCRed"
     "``-v <path to store results>:/output``", "This is where OCRed files will be stored"
-    "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "This will place files in the output in {output}/{year}/{month}/{filename}"
-    "``-e OCR_ON_SUCCESS_DELETE=1``", "This will delete the input file if the exit code is 0 (OK)"
-    "``-e OCR_DESKEW=1``", "This will enable deskew for crooked PDFs"
+    "``-e OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1``", "Define environment variable OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1"
+    "``-e OCR_ON_SUCCESS_DELETE=1``", "Define environment variable"
+    "``-e OCR_DESKEW=1``", "Define environment variable"
     "``-e PYTHONBUFFERED=1``", "This will force STDOUT to be unbuffered and allow you to see messages in docker logs"
 
 This service relies on polling to check for changes to the filesystem. It
@@ -143,56 +177,6 @@ service is always available.
     :language: yaml
     :caption: misc/docker-compose.example.yml
 
-Watched folders with watcher.py
--------------------------------
-
-The watcher service may also be run natively, without Docker:
-
-.. code-block:: bash
-
-    pip3 install -r requirements/watcher.txt
-
-    env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
-        OCR_OUTPUT_DIRECTORY=/mnt/output-pdfs \
-        OCR_OUTPUT_DIRECTORY_YEAR_MONTH=1 \
-        python3 watcher.py
-
-Watched folders with CLI
-------------------------
-
-To set up a "hot folder" that will trigger OCR for every file inserted,
-use a program like Python
-`watchdog <https://pypi.python.org/pypi/watchdog>`__ (supports all major
-OS).
-
-One could then configure a scanner to automatically place scanned files
-in a hot folder, so that they will be queued for OCR and copied to the
-destination.
-
-.. code-block:: bash
-
-   pip install watchdog
-
-watchdog installs the command line program ``watchmedo``, which can be
-told to run ``ocrmypdf`` on any .pdf added to the current directory
-(``.``) and place the result in the previously created ``out/`` folder.
-
-.. code-block:: bash
-
-   cd hot-folder
-   mkdir out
-   watchmedo shell-command \
-       --patterns="*.pdf" \
-       --ignore-directories \
-       --command='ocrmypdf "${watch_src_path}" "out/${watch_src_path}" ' \
-       .  # don't forget the final dot
-
-On file servers, you could configure watchmedo as a system service so it
-will run all the time.
-
-For more complex behavior you can write a Python script around to use
-the watchdog API. You can refer to the watcher.py script as an example.
-
 Caveats
 -------
 
@@ -218,6 +202,13 @@ Alternatives
 -  `Watchman <https://facebook.github.io/watchman/>`__ is a more
    powerful alternative to ``watchmedo``.
 
+AWS Lambda is not viable
+------------------------
+
+AWS Lambda and its equivalents have low limits on execution time and payload
+size, relative to OCRmyPDF's needs. As of this writing, the request/response
+payload for AWS Lambda was 6 MB, which means many PDFs will not fit.
+
 macOS Automator
 ===============
 

docs/docker.rst

@@ -90,10 +90,8 @@ Docker volume:
 
 .. code-block:: bash
 
-   docker run --rm -v $(pwd):/data ocrmypdf /data/input.pdf /data/output.pdf
-
-(However, when done this way, ``output.pdf`` may be owned by the root
-user.)
+   alias docker_ocrmypdf='docker run --rm  -i --user "$(id -u):$(id -g)" --workdir /data -v "$PWD:/data" ocrmypdf'
+   docker_ocrmypdf /data/input.pdf /data/output.pdf
 
 .. _docker-lang-packs:
 

docs/index.rst

@@ -27,6 +27,7 @@ image processing and OCR to existing PDFs.
    advanced
    batch
    security
+   performance
    errors
 
 .. toctree::

docs/installation.rst

@@ -137,11 +137,12 @@ Installing the latest version on Ubuntu 18.04 LTS
 -------------------------------------------------
 
 Ubuntu 18.04 includes ocrmypdf 6.1.2 - you can install that with ``apt``, but
-it is quite old now. To install a more recent version, first install several
-system dependencies:
+it is quite old now. To install a more recent version, uninstall the old version
+of ocrmypdf, and install the following dependencies:
 
 .. code-block:: bash
 
+    sudo apt-get -y remove ocrmypdf
     sudo apt-get -y update
     sudo apt-get -y install \
         ghostscript \
@@ -232,7 +233,7 @@ environment variable contains ``$HOME/.local/bin``.
 .. code-block:: bash
 
     export PATH=$HOME/.local/bin:$PATH
-    pip3 install --user ocrmypdf
+    pip3.6 install --user ocrmypdf
 
 To add JBIG2 encoding, see :ref:`jbig2`.
 
@@ -326,20 +327,7 @@ standard tooling needed to build packages, such as a compiler and binary tools.
 
    sudo pacman -S base-devel
 
-The OCRmyPDF package depends on `the python-pdfminer.six AUR package
-<https://aur.archlinux.org/packages/python-pdfminer.six/>`__.  Dependencies on
-AUR packages are not automatically resolved, so this package must be manually
-installed first.
-
-.. code-block:: bash
-
-   curl -O https://aur.archlinux.org/cgit/aur.git/snapshot/python-pdfminer.six.tar.gz
-   tar xvzf python-pdfminer.six.tar.gz
-   cd python-pdfminer.six
-   makepkg -sri
-
-With that complete you can then repeat the same series of steps for the
-OCRmyPDF package.
+Now you are ready to install the OCRmyPDF package.
 
 .. code-block:: bash
 
@@ -373,11 +361,10 @@ page.
     fine without it but will produce larger output files. The encoder is
     available from `the jbig2enc-git AUR package
     <https://aur.archlinux.org/packages/jbig2enc-git/>`__ and may be installed
-    using the same series of steps as for the installation of the pdfminer.six
-    and OCRmyPDF AUR packages. Alternatively, it may be built manually from
-    source following the instructions in `Installing the JBIG2 encoder
-    <jbig2>`__.  If JBIG2 is installed, OCRmyPDF 7.0.0 and later will
-    automatically detect it.
+    using the same series of steps as for the installation OCRmyPDF AUR
+    package. Alternatively, it may be built manually from source following the
+    instructions in `Installing the JBIG2 encoder <jbig2>`__.  If JBIG2 is
+    installed, OCRmyPDF 7.0.0 and later will automatically detect it.
 
 Alpine Linux
 ------------
@@ -442,7 +429,10 @@ languages you can optionally install them all:
 Manual installation on macOS
 ----------------------------
 
-These instructions probably work on all macOS supported by Homebrew.
+These instructions probably work on all macOS supported by Homebrew, and are
+for installing a more current version of OCRmyPDF than is available from
+Homebrew. Note that the Homebrew versions usually track the release versions
+fairly closely.
 
 If it's not already present, `install Homebrew <http://brew.sh/>`__.
 
@@ -453,14 +443,8 @@ Update Homebrew:
     brew update
 
 Install or upgrade the required Homebrew packages, if any are missing.
-To do this, download the ``Brewfile`` that lists all of the dependencies
-to the current directory, and run ``brew bundle`` to process them
-(installing or upgrading as needed). ``Brewfile`` is a plain text file.
-
-.. code-block:: bash
-
-    wget https://github.com/jbarlow83/OCRmyPDF/raw/master/.travis/Brewfile
-    brew bundle
+To do this, use ``brew edit ocrmypdf`` to obtain a recent list of Homebrew
+dependencies. You could also check the ``azure-pipelines.yml``.
 
 This will include the English, French, German and Spanish language
 packs. If you need other languages you can optionally install them all:
@@ -574,7 +558,7 @@ Installing on Windows Subsystem for Linux
 
 .. code-block:: powershell
 
-    wsl sudo ln -s  /home/user/.local/bin/ocrmypdf /usr/local/bin/ocrmypdf
+    wsl sudo ln -s  /home/$USER/.local/bin/ocrmypdf /usr/local/bin/ocrmypdf
 
 Then confirm that the expected version from PyPI (|latest|) is installed:
 
@@ -591,6 +575,53 @@ Docker
 You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
+Installing on Cygwin64 under Windows
+====================================
+
+First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::
+
+    python36 (or later)
+    python3?-devel
+    python3?-pip
+    python3?-lxml
+    python3?-imaging
+
+       (where 3? means match the version of python3 you installed)
+
+    gcc-g++
+    ghostscript (<=9.50 or >=9.52-2 see note below)
+    libexempi3
+    libexempi-devel
+    libffi6
+    libffi-devel
+    pngquant
+    qpdf
+    libqpdf-devel
+    tesseract-ocr
+    tesseract-ocr-devel
+
+.. note::
+
+    The Cygwin package for Ghostscript in versions 9.52 and
+    9.52-1 contained a bug that caused an exception to occur when
+    ocrmypdf invoked gs.  Make sure you have either 9.50 (or earlier)
+    or 9.52-2 (or later).
+
+Then open a Cygwin terminal (i.e. ``mintty``), run the following commands. Note
+that if you are using the version of ``pip`` that was installed with the Cygwin
+Python package, the command name will be ``pip3``.  If you have since updated
+``pip`` (with, for instance ``pip3 install --upgrade pip``) the the command is
+likely just ``pip`` instead of ``pip3``:
+
+.. code-block:: bash
+
+    pip3 install wheel
+    pip3 install ocrmypdf
+
+The optional dependency "unpaper" that is currently not available under Cygwin.
+Without it, certain options such as ``--clean`` will produce an error message.
+However, the OCR-to-text-layer functionality is available.
+
 Installing with Python pip
 ==========================
 

docs/performance.rst

@@ -0,0 +1,22 @@
+===========
+Performance
+===========
+
+Some users have noticed that current versions of OCRmyPDF do not run as quickly
+as some older versions (specifically 6.x and older). This is because OCRmyPDF
+added image optimization as a postprocessing step, and it is enabled by default.
+
+Speed
+=====
+
+If running OCRmyPDF quickly is your main goal, you can use settings such as:
+
+* ``--optimize 0`` to disable file size optimization
+* ``--output-type pdf`` to disable PDF/A generation
+* ``--fast-web-view 0`` to disable fast web view optimization
+* ``--skip-big`` to skip large images, if some pages have large images
+
+You can also avoid:
+
+* ``--force-ocr``
+* Image preprocessing

docs/release_notes.rst

@@ -13,6 +13,61 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.8.2
+======
+
+-  Fixed an issue where OCRmyPDF would ignore text inside Form XObject when
+   making certain decisions about whether a document already had text.
+-  Fixed file size increase warning to take overhead of small files into account.
+-  Added instructions for installing on Cygwin.
+
+v9.8.1
+======
+
+-  Fixed an issue where unexpected files in the ``%PROGRAMFILES%\gs`` directory
+   (Windows) caused an exception.
+-  Mark pdfminer.six 20200517 as supported.
+-  If jbig2enc is missing and optimization is requested, a warning is issued
+   instead of an error, which was the intended behavior.
+-  Documentation updates.
+
+v9.8.0
+======
+
+-  Fixed issue where only the first PNG (FlateDecode) image in a file would be
+   considered for optimization. File sizes should be improved from here on.
+-  Fixed a startup crash when the chosen language was Japanese (#543).
+-  Added options to configure polling and log level to watcher.py.
+
+v9.7.2
+======
+
+-  Fixed an issue with ``ocrmypdf.ocr(...language=)`` not accepting a list of
+   languages as documented.
+-  Updated setup.py to confirm that pdfminer.six version 20200402 is supported.
+
+v9.7.1
+======
+
+-  Fixed version check failing when used with qpdf 10.0.0.
+-  Added some missing type annotations.
+-  Updated documentation to warn about need for "ifmain" guard and Windows.
+
+v9.7.0
+======
+
+-  Fixed an error in watcher.py if ``OCR_JSON_SETTINGS`` was not defined.
+-  Ghostscript 9.51 is now blacklisted, due to numerous problems with this version.
+-  Added a workaround for a problem with "txtwrite" in Ghostscript 9.52.
+-  Fixed an issue where the incorrect number of threads used was shown when
+   ``OMP_THREAD_LIMIT`` was manipulated.
+-  Removed a possible performance bottlenecks for files that use hundreds to
+   thousands of images on the same page.
+-  Documentation improvements.
+-  Optimization will now be applied to some monochrome images that have a color
+   profile defined instead of only black and white.
+-  ICC profiles are consulted when determining the simplified colorspace of an
+   image.
 
 v9.6.1
 ======

misc/watcher.py

@@ -14,8 +14,10 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import json
 import logging
 import os
+import sys
 import time
 from datetime import datetime
 from pathlib import Path
@@ -23,6 +25,7 @@ from pathlib import Path
 import pikepdf
 from watchdog.events import PatternMatchingEventHandler
 from watchdog.observers import Observer
+from watchdog.observers.polling import PollingObserver
 
 import ocrmypdf
 
@@ -33,8 +36,10 @@ OUTPUT_DIRECTORY = os.getenv('OCR_OUTPUT_DIRECTORY', '/output')
 OUTPUT_DIRECTORY_YEAR_MONTH = bool(os.getenv('OCR_OUTPUT_DIRECTORY_YEAR_MONTH', False))
 ON_SUCCESS_DELETE = bool(os.getenv('OCR_ON_SUCCESS_DELETE', False))
 DESKEW = bool(os.getenv('OCR_DESKEW', False))
+OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = os.getenv('OCR_POLL_NEW_FILE_SECONDS', 1)
-LOGLEVEL = os.environ.get('OCR_LOGLEVEL', 'INFO').upper()
+USE_POLLING = bool(os.getenv('OCR_USE_POLLING', False))
+LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
 PATTERNS = ['*.pdf']
 
 log = logging.getLogger('ocrmypdf-watcher')
@@ -87,7 +92,10 @@ def execute_ocrmypdf(file_path):
         return
     log.info(f'Attempting to OCRmyPDF to: {output_path}')
     exit_code = ocrmypdf.ocr(
-        input_file=file_path, output_file=output_path, deskew=DESKEW
+        input_file=file_path,
+        output_file=output_path,
+        deskew=DESKEW,
+        **OCR_JSON_SETTINGS,
     )
     if exit_code == 0 and ON_SUCCESS_DELETE:
         log.info(f'OCR is done. Deleting: {file_path}')
@@ -106,6 +114,7 @@ def main():
     ocrmypdf.configure_logging(
         verbosity=ocrmypdf.Verbosity.default, manage_root_logger=True
     )
+    log.setLevel(LOGLEVEL)
     log.info(
         f"Starting OCRmyPDF watcher with config:\n"
         f"Input Directory: {INPUT_DIRECTORY}\n"
@@ -118,12 +127,21 @@ def main():
         f"OUTPUT_DIRECTORY_YEAR_MONTH: {OUTPUT_DIRECTORY_YEAR_MONTH}\n"
         f"ON_SUCCESS_DELETE: {ON_SUCCESS_DELETE}\n"
         f"DESKEW: {DESKEW}\n"
+        f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
+        f"USE_POLLING: {USE_POLLING}\n"
         f"LOGLEVEL: {LOGLEVEL}\n"
     )
 
+    if 'input_file' in OCR_JSON_SETTINGS or 'output_file' in OCR_JSON_SETTINGS:
+        log.error('OCR_JSON_SETTINGS should not specify input file or output file')
+        sys.exit(1)
+
     handler = HandleObserverEvent(patterns=PATTERNS)
-    observer = Observer()
+    if USE_POLLING:
+        observer = PollingObserver()
+    else:
+        observer = Observer()
     observer.schedule(handler, INPUT_DIRECTORY, recursive=True)
     observer.start()
     try:

requirements/dev.txt

@@ -1,2 +0,0 @@
-twine >= 1.8.1
-coverage >= 4.5

requirements/main.txt

@@ -2,9 +2,9 @@
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
 cffi == 1.14.0
-img2pdf == 0.3.3
-pdfminer.six == 20200124
-pikepdf == 1.10.1
-Pillow == 7.0.0
+img2pdf == 0.3.4
+pdfminer.six == 20200402
+pikepdf == 1.11.1
+Pillow == 7.1.1
 reportlab == 3.5.34
-tqdm == 4.42.1
+tqdm == 4.45.0

setup.py

@@ -62,7 +62,7 @@ setup(
     long_description_content_type='text/markdown',
     url='https://github.com/jbarlow83/OCRmyPDF',
     author='James R. Barlow',
-    author_email='jim@purplerock.ca',
+    author_email='james@purplerock.ca',
     packages=find_packages('src', exclude=["tests", "tests.*"]),
     package_dir={'': 'src'},
     keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200124',
+        'pdfminer.six >= 20181108, <= 20200517',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/_sync.py

@@ -245,6 +245,10 @@ def exec_concurrent(context):
     if context.options.tesseract_env is None:
         context.options.tesseract_env = os.environ.copy()
     context.options.tesseract_env.setdefault('OMP_THREAD_LIMIT', str(tess_threads))
+    try:
+        tess_threads = int(context.options.tesseract_env['OMP_THREAD_LIMIT'])
+    except ValueError:  # OMP_THREAD_LIMIT initialized to non-numeric
+        context.log.error("Environment variable OMP_THREAD_LIMIT is not numeric")
     if tess_threads > 1:
         context.log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
 
@@ -357,10 +361,11 @@ def run_pipeline(options, api=False):
         options.jobs = available_cpu_count()
 
     work_folder = mkdtemp(prefix="com.github.ocrmypdf.")
+    debug_log_handler = None
     if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
         'PYTEST_CURRENT_TEST', ''
     ):
-        configure_debug_logging(Path(work_folder) / "debug.log")
+        debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
 
     try:
         check_requested_output_file(options)
@@ -428,6 +433,12 @@ def run_pipeline(options, api=False):
         log.exception("An exception occurred while executing the pipeline")
         return ExitCode.other_error
     finally:
+        if debug_log_handler:
+            try:
+                debug_log_handler.close()
+                log.removeHandler(debug_log_handler)
+            except EnvironmentError as e:
+                print(e, file=sys.stderr)
         cleanup_working_files(work_folder, options)
 
     return ExitCode.ok

src/ocrmypdf/_validation.py

@@ -24,6 +24,7 @@ import sys
 from pathlib import Path
 from shutil import copyfileobj
 
+import pikepdf
 import PIL
 
 from ._unicodefun import verify_python3_env
@@ -410,8 +411,15 @@ def report_output_file_size(options, input_file, output_file):
         input_size = Path(input_file).stat().st_size
     except FileNotFoundError:
         return  # Outputting to stream or something
+    with pikepdf.open(output_file) as p:
+        # Overhead constants obtained by estimating amount of data added by OCR
+        # PDF/A conversion, and possible XMP metadata addition, with compression
+        FILE_OVERHEAD = 4000
+        OCR_PER_PAGE_OVERHEAD = 3000
+        reasonable_overhead = FILE_OVERHEAD + OCR_PER_PAGE_OVERHEAD * len(p.pages)
     ratio = output_size / input_size
-    if ratio < 1.35 or input_size < 25000:
+    reasonable_ratio = output_size / (input_size + reasonable_overhead)
+    if reasonable_ratio < 1.35 or input_size < 25000:
         return  # Seems fine
 
     reasons = []
@@ -466,11 +474,12 @@ def check_dependency_versions(options):
         version_checker=ghostscript.version,
         need_version='9.15',  # limited by Travis CI / Ubuntu 14.04 backports
     )
-    if ghostscript.version() == '9.24':
+    gs_version = ghostscript.version()
+    if gs_version in ('9.24', '9.51'):
         raise MissingDependencyError(
-            "Ghostscript 9.24 contains serious regressions and is not "
-            "supported. Please upgrade to Ghostscript 9.25 or use an older "
-            "version."
+            f"Ghostscript {gs_version} contains serious regressions and is not "
+            "supported. Please upgrade to a newer version, or downgrade to the "
+            "previous version."
         )
     check_external_program(
         program='qpdf',

src/ocrmypdf/api.py

@@ -21,7 +21,7 @@ import sys
 from contextlib import suppress
 from enum import IntEnum
 from pathlib import Path
-from typing import Dict, List
+from typing import Dict, Iterable
 
 from tqdm import tqdm
 
@@ -67,7 +67,11 @@ class Verbosity(IntEnum):
     debug_all = 2  #: More detailed debugging from ocrmypdf and dependent modules
 
 
-def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=False):
+def configure_logging(
+    verbosity: Verbosity,
+    progress_bar_friendly: bool = True,
+    manage_root_logger: bool = False,
+):
     """Set up logging.
 
     Library users may wish to use this function if they want their log output to be
@@ -128,7 +132,7 @@ def configure_logging(verbosity, progress_bar_friendly=True, manage_root_logger=
     return log
 
 
-def create_options(*, input_file, output_file, **kwargs):
+def create_options(*, input_file: os.PathLike, output_file: os.PathLike, **kwargs):
     cmdline = []
     deferred = []
 
@@ -150,6 +154,12 @@ def create_options(*, input_file, output_file, **kwargs):
                 cmdline.append(f"--{cmd_style_arg}")
             continue
 
+        if isinstance(val, Iterable) and not isinstance(val, str):
+            for elem in val:
+                cmdline.append(f"--{cmd_style_arg}")
+                cmdline.append(elem)
+            continue
+
         # We have a parameter
         cmdline.append(f"--{cmd_style_arg}")
         if isinstance(val, (int, float)):
@@ -180,7 +190,7 @@ def ocr(  # pylint: disable=unused-argument
     input_file: os.PathLike,
     output_file: os.PathLike,
     *,
-    language: List[str] = None,
+    language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
     sidecar: os.PathLike = None,
@@ -210,7 +220,7 @@ def ocr(  # pylint: disable=unused-argument
     jbig2_page_group_size: int = None,
     pages: str = None,
     max_image_mpixels: float = None,
-    tesseract_config: List[str] = None,
+    tesseract_config: Iterable[str] = None,
     tesseract_pagesegmode: int = None,
     tesseract_oem: int = None,
     pdf_renderer=None,

src/ocrmypdf/cli.py

@@ -20,6 +20,8 @@ import argparse
 from ._version import PROGRAM_NAME as _PROGRAM_NAME
 from ._version import __version__ as _VERSION
 
+__all__ = ['parser']
+
 
 def numeric(basetype, min_=None, max_=None):
     """Validator for numeric params"""

src/ocrmypdf/exec/__init__.py

@@ -23,7 +23,9 @@ import re
 import shutil
 import sys
 from collections.abc import Mapping
+from distutils.version import LooseVersion
 from functools import lru_cache
+from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError
 from subprocess import run as subprocess_run
 
@@ -137,24 +139,25 @@ def shim_paths_with_program_files(env=None):
     program_files = env.get('PROGRAMFILES', '')
     if not program_files:
         return env.get('PATH', '')
-    paths = []
-    try:
-        for dirname in os.listdir(program_files):
-            if dirname.lower() == 'tesseract-ocr':
-                paths.append(os.path.join(program_files, dirname))
-            elif dirname.lower() == 'gs':
-                try:
-                    latest_gs = max(
-                        os.listdir(os.path.join(program_files, dirname)),
-                        key=lambda d: float(d[2:]),
-                    )
-                except (FileNotFoundError, NotADirectoryError):
-                    continue
-                paths.append(os.path.join(program_files, dirname, latest_gs, 'bin'))
-    except EnvironmentError:
-        pass
-    paths.extend(path for path in os.get_exec_path(env) if path not in set(paths))
-    return os.pathsep.join(paths)
+
+    def path_walker():
+        for path in Path(program_files).iterdir():
+            if not path.is_dir():
+                continue
+            if path.name.lower() == 'tesseract-ocr':
+                yield path
+            elif path.name.lower() == 'gs':
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
+
+    paths = sorted(
+        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
+    )
+    paths.extend(
+        Path(str_path)
+        for str_path in os.get_exec_path(env)
+        if Path(str_path) not in set(paths)
+    )
+    return os.pathsep.join(str(p) for p in paths)
 
 
 missing_program = '''
@@ -232,10 +235,10 @@ def _error_trailer(program, package, **kwargs):
 
 
 def _error_missing_program(program, package, required_for, recommended):
-    if required_for:
+    if recommended:
+        log.warning(missing_recommend_program.format(**locals()))
+    elif required_for:
         log.error(missing_optional_program.format(**locals()))
-    elif recommended:
-        log.info(missing_recommend_program.format(**locals()))
     else:
         log.error(missing_program.format(**locals()))
     _error_trailer(**locals())
@@ -270,7 +273,15 @@ def check_external_program(
             raise MissingDependencyError()
         return
 
-    if found_version < need_version:
+    def remove_leading_v(s):
+        if s.startswith('v'):
+            return s[1:]
+        return s
+
+    found_version = remove_leading_v(found_version)
+    need_version = remove_leading_v(need_version)
+
+    if found_version and LooseVersion(found_version) < LooseVersion(need_version):
         _error_old_version(program, package, need_version, found_version, required_for)
         if not recommended:
             raise MissingDependencyError()

src/ocrmypdf/exec/tesseract.py

@@ -77,20 +77,14 @@ def has_textonly_pdf(tesseract_env=None, langs=None):
     args_tess = tess_base_args(langs, engine_mode=None) + ['--print-parameters', 'pdf']
     params = ''
     try:
-        proc = run(
-            args_tess,
-            check=True,
-            universal_newlines=True,
-            stdout=PIPE,
-            stderr=STDOUT,
-            env=tesseract_env,
-        )
+        # print-parameters can return non-UTF8 if the parameters are so initialized
+        proc = run(args_tess, check=True, stdout=PIPE, stderr=STDOUT, env=tesseract_env)
         params = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(
             "Could not --print-parameters from tesseract"
         ) from e
-    if 'textonly_pdf' in params:
+    if b'textonly_pdf' in params:
         return True
     return False
 

src/ocrmypdf/helpers.py

@@ -28,7 +28,7 @@ from pathlib import Path
 log = logging.getLogger(__name__)
 
 
-def safe_symlink(input_file, soft_link_name, *args, **kwargs):
+def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike, *args, **kwargs):
     """
     Helper function: relinks soft symbolic link if necessary
     """
@@ -76,12 +76,12 @@ def is_iterable_notstr(thing):
     return isinstance(thing, Iterable) and not isinstance(thing, str)
 
 
-def monotonic(L):
+def monotonic(L: Iterable):
     """Does list increase monotonically?"""
     return all(b > a for a, b in zip(L, L[1:]))
 
 
-def page_number(input_file):
+def page_number(input_file: os.PathLike):
     """Get one-based page number implied by filename (000002.pdf -> 2)"""
     return int(os.path.basename(os.fspath(input_file))[0:6])
 
@@ -97,7 +97,7 @@ def available_cpu_count():
     return 1
 
 
-def is_file_writable(test_file):
+def is_file_writable(test_file: os.PathLike):
     """Intentionally racy test if target is writable.
 
     We intend to write to the output file if and only if we succeed and

src/ocrmypdf/lib/compile_leptonica.py

@@ -443,6 +443,12 @@ pixReadBarcodes(PIXA     *pixa,
                 SARRAY  **psaw,
                 l_int32 debugflag);
 
+PIX *
+pixGenHalftoneMask(PIX      *pixs,
+                        PIX     **ppixtext,
+                        l_int32  *phtfound,
+                        PIXA     *pixadb);
+
 l_int32
 l_generateCIDataForPdf(const char *fname,
                        PIX *pix,

src/ocrmypdf/optimize.py

@@ -155,6 +155,17 @@ def extract_image_generic(*, pike, root, log, image, xref, options):
         # generating a PNG from compressed data
         pim.as_pil_image().save(png_name(root, xref))
         return xref, '.png'
+    elif (
+        not pim.indexed
+        and pim.colorspace == Name.ICCBased
+        and pim.bits_per_component == 1
+        and not options.jbig2_lossy
+    ):
+        # We can losslessly optimize 1-bit images to CCITT or JBIG2 without
+        # paying any attention to the ICC profile, provided we're not doing
+        # lossy JBIG2
+        pim.as_pil_image().save(png_name(root, xref))
+        return xref, '.png'
 
     return None
 
@@ -410,9 +421,9 @@ def transcode_pngs(pike, images, image_name_fn, root, log, options):
             )
             continue
         if compdata.type == leptonica.lept.L_FLATE_ENCODE:
-            return rewrite_png(pike, im_obj, compdata, log)
+            rewrite_png(pike, im_obj, compdata, log)
         elif compdata.type == leptonica.lept.L_G4_ENCODE:
-            return rewrite_png_as_g4(pike, im_obj, compdata, log)
+            rewrite_png_as_g4(pike, im_obj, compdata, log)
 
 
 def rewrite_png_as_g4(pike, im_obj, compdata, log):

src/ocrmypdf/pdfinfo/info.py

@@ -18,11 +18,11 @@
 
 import logging
 import re
-from collections import namedtuple
+from collections import defaultdict, namedtuple
 from decimal import Decimal
 from enum import Enum
 from math import hypot, isclose
-from os import fspath
+from os import PathLike, fspath
 from pathlib import Path
 from warnings import warn
 
@@ -30,17 +30,17 @@ import pikepdf
 from pikepdf import PdfMatrix
 from tqdm import tqdm
 
-from ocrmypdf.exceptions import EncryptedPdfError, MissingDependencyError
-
-from . import ghosttext
-from .layout import get_page_analysis, get_text_boxes
+from ocrmypdf.exceptions import EncryptedPdfError
+from ocrmypdf.exec import ghostscript
+from ocrmypdf.pdfinfo import ghosttext
+from ocrmypdf.pdfinfo.layout import get_page_analysis, get_text_boxes
 
 logger = logging.getLogger()
 
 Colorspace = Enum('Colorspace', 'gray rgb cmyk lab icc index sep devn pattern jpeg2000')
 
 Encoding = Enum(
-    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate ' + 'runlength'
+    'Encoding', 'ccitt jpeg jpeg2000 jbig2 asciihex ascii85 lzw flate runlength'
 )
 
 FRIENDLY_COLORSPACE = {
@@ -98,7 +98,7 @@ XobjectSettings = namedtuple('XobjectSettings', ['name', 'shorthand', 'stack_dep
 InlineSettings = namedtuple('InlineSettings', ['iimage', 'shorthand', 'stack_depth'])
 
 ContentsInfo = namedtuple(
-    'ContentsInfo', ['xobject_settings', 'inline_images', 'found_vector']
+    'ContentsInfo', ['xobject_settings', 'inline_images', 'found_vector', 'name_index']
 )
 
 TextboxInfo = namedtuple('TextboxInfo', ['bbox', 'is_visible', 'is_corrupt'])
@@ -151,6 +151,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
     ctm = PdfMatrix(initial_shorthand)
     xobject_settings = []
     inline_images = []
+    name_index = defaultdict(lambda: [])
     found_vector = False
     vector_ops = set('S s f F f* B B* b b*'.split())
     image_ops = set('BI ID EI q Q Do cm'.split())
@@ -185,6 +186,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
                 name=image_name, shorthand=ctm.shorthand, stack_depth=len(stack)
             )
             xobject_settings.append(settings)
+            name_index[image_name].append(settings)
         elif operator == 'INLINE IMAGE':  # BI/ID/EI are grouped into this
             iimage = operands[0]
             inline = InlineSettings(
@@ -198,6 +200,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
         xobject_settings=xobject_settings,
         inline_images=inline_images,
         found_vector=found_vector,
+        name_index=name_index,
     )
 
 
@@ -303,14 +306,22 @@ class ImageInfo:
         if self._enc == Encoding.jpeg2000:
             self._color = Colorspace.jpeg2000
 
-        self._comp = FRIENDLY_COMP.get(self._color, '?')
+        if self._color == Colorspace.icc:
+            # Check the ICC profile to determine actual colorspace
+            pim_icc = pim.icc
+            if pim_icc.profile.xcolor_space == 'GRAY':
+                self._comp = 1
+            elif pim_icc.profile.xcolor_space == 'CMYK':
+                self._comp = 4
+            else:
+                self._comp = 3
+        else:
+            self._comp = FRIENDLY_COMP.get(self._color, '?')
 
-        # Bit of a hack... infer grayscale if component count is uncertain
-        # but encoding must be monochrome. This happens if a monochrome image
-        # has an ICC profile attached. Better solution would be to examine
-        # the ICC profile.
-        if self._comp == '?' and self._enc in (Encoding.ccitt, Encoding.jbig2):
-            self._comp = FRIENDLY_COMP[Colorspace.gray]
+            # Bit of a hack... infer grayscale if component count is uncertain
+            # but encoding only supports monochrome.
+            if self._comp == '?' and self._enc in (Encoding.ccitt, Encoding.jbig2):
+                self._comp = FRIENDLY_COMP[Colorspace.gray]
 
     @property
     def name(self):
@@ -411,13 +422,9 @@ def _find_regular_images(container, contentsinfo):
     """
 
     for pdfimage, xobj in _image_xobjects(container):
-
-        # For each image that is drawn on this, check if we drawing the
-        # current image - yes this is O(n^2), but n == 1 almost always
-        for draw in contentsinfo.xobject_settings:
-            if draw.name != xobj:
-                continue
-
+        if xobj not in contentsinfo.name_index:
+            continue
+        for draw in contentsinfo.name_index[xobj]:
             if draw.stack_depth == 0 and _is_unit_square(draw.shorthand):
                 # At least one PDF in the wild (and test suite) draws an image
                 # when the graphics stack depth is 0, meaning that the image
@@ -551,7 +558,7 @@ def simplify_textboxes(miner, textbox_getter):
         yield TextboxInfo(box.bbox, visible, corrupt)
 
 
-def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
+def _pdf_get_pageinfo(pdf, pageno: int, infile: PathLike, xmltext: str):
     pageinfo = {}
     pageinfo['pageno'] = pageno
     pageinfo['images'] = []
@@ -611,25 +618,28 @@ def _pdf_get_pageinfo(pdf, pageno: int, infile, xmltext):
 
 def _pdf_get_all_pageinfo(infile, detailed_analysis=False, log=None, progbar=False):
     pdf = pikepdf.open(infile)  # Do not close in this function
-    if pdf.is_encrypted:
-        pdf.close()
-        raise EncryptedPdfError()  # Triggered by encryption with empty passwd
-    if detailed_analysis:
-        pages_xml = None
-    else:
-        pages_xml = ghosttext.extract_text_xml(infile, pdf, pageno=None, log=log)
+    try:
+        if pdf.is_encrypted:
+            raise EncryptedPdfError()  # Triggered by encryption with empty passwd
+        if detailed_analysis:
+            pages_xml = None
+        else:
+            pages_xml = ghosttext.extract_text_xml(infile, pdf, pageno=None, log=log)
 
-    pages = []
-    for n, _ in tqdm(
-        enumerate(pdf.pages),
-        total=len(pdf.pages),
-        desc="Scan",
-        unit='page',
-        disable=not progbar,
-    ):
-        page_xml = pages_xml[n] if pages_xml else None
-        page = PageInfo(pdf, n, infile, page_xml, detailed_analysis)
-        pages.append(page)
+        pages = []
+        for n, _ in tqdm(
+            enumerate(pdf.pages),
+            total=len(pdf.pages),
+            desc="Scan",
+            unit='page',
+            disable=not progbar,
+        ):
+            page_xml = pages_xml[n] if pages_xml else None
+            page = PageInfo(pdf, n, infile, page_xml, detailed_analysis)
+            pages.append(page)
+    except Exception:
+        pdf.close()
+        raise
 
     return pages, pdf
 
@@ -750,6 +760,8 @@ class PdfInfo:
 
     def __init__(self, infile, detailed_page_analysis=False, log=logger, progbar=False):
         self._infile = infile
+        if ghostscript.version() in ('9.52',):
+            detailed_page_analysis = True  # txtwrite doesn't work in these versions
         self._pages, pdf = _pdf_get_all_pageinfo(
             infile, detailed_page_analysis, log=log, progbar=progbar
         )
@@ -805,10 +817,14 @@ def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('infile')
     args = parser.parse_args()
-    info = _pdf_get_all_pageinfo(args.infile)
+    pagesinfo, pdfinfo = _pdf_get_all_pageinfo(args.infile)
     from pprint import pprint
 
-    pprint(info)
+    pprint(pdfinfo)
+    for page in pagesinfo:
+        pprint(page)
+        for im in page.images:
+            pprint(im)
 
 
 if __name__ == '__main__':

src/ocrmypdf/pdfinfo/layout.py

@@ -269,7 +269,9 @@ class TextPositionTracker(PDFLayoutAnalyzer):
 
 def get_page_analysis(infile, pageno, pscript5_mode):
     rman = pdfminer.pdfinterp.PDFResourceManager(caching=True)
-    dev = TextPositionTracker(rman, laparams=LAParams())
+    dev = TextPositionTracker(
+        rman, laparams=LAParams(all_texts=True, detect_vertical=True)
+    )
     interp = pdfminer.pdfinterp.PDFPageInterpreter(rman, dev)
 
     if pscript5_mode:

tests/conftest.py

@@ -97,10 +97,48 @@ assert ast.parse(WINDOWS_SHIM_TEMPLATE.format(spoofer=repr(r"C:\\Temp\\file.py")
 def spoof(tmp_path_factory, **kwargs):
     """Modify PATH to override subprocess executables
 
-    spoof(program1='replacement', ...)
+    spoof(tmp_path_factory, program1='replacement', ...)
 
-    Creates temporary directory with symlinks to targets.
+    For the test suite we need a way override executables, so that we can
+    substitute desired results such as errors or just speed up OCR.
 
+    On POSIXish platforms we create a temporary folder with overrides that
+    are symlinks to the executables we want to run. We do not actually override
+    PATH. We also set an environment variable _OCRMYPDF_TEST_PATH, which
+    OCRmyPDF's subprocess wrapper will check before they use regular PATH. The
+    output is a folder full of executables we are overriding. We can override
+    multiple executables. The end result is a folder we can use in a PATH-style
+    lookup to override some executables:
+
+        /tmp/abcxyz/tesseract -> ocrmypdf/tests/resources/spoof/tesseract_crash.py
+        /tmp/abcxyz/gs -> ocrmypdf/tests/resources/spoof/gs_backflip.py
+
+    Windows needs extra help from us because usually, only the Administrator
+    can create symlinks. Instead we create small Python scripts that call
+    the programs we want, implementing the effect of a symlink. This is cleaner
+    than creating Windows executables or trying to use non-Python scripts.
+    The temporary folder generated for Windows could like:
+
+        %TEMP%\abcxyz\tesseract.py:
+            (script that runs ocrmypdf/tests/resources/spoof/tesseract_crash.py)
+        %TEMP%\abcxyz\gswin32c.py:
+            (script that runs ocrmypdf/tests/resources/spoof/gs_backflip.py)
+        %TEMP%\abcxyz\gswin64c.py:
+            (script that runs ocrmypdf/tests/resources/spoof/gs_backflip.py)
+
+    We also address one quirk here, that Ghostscript may be known as gswin32c
+    or gswin64c, depending on what the user installed (regardless of Windows
+    itself). On POSIX, Ghostscript is just 'gs'.  We handle the special case here
+    too.
+
+    All of this is intimately dependent on the machinery in ocrmypdf.exec.run().
+    In particular, for Windows, that code has to know that if there is a .py
+    file, it needs to run it with Python, since Windows does not like being
+    asked to execute files.
+
+    We don't overload PATH directly because we have some tests where we call
+    ocrmypdf as a subprocess (to exercise the command line interface) and some
+    tests where we call it as an API.
     """
     env = os.environ.copy()
     slug = '-'.join(v.replace('.py', '') for v in sorted(kwargs.values()))

tests/spoof/tesseract_badutf8.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-# © 2017 James R. Barlow: github.com/jbarlow83
-#
-# Permission is hereby granted, free of charge, to any person obtaining a
-# copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be included
-# in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-import sys
-
-"""Tesseract bad utf8 spoof
-
-In 'hocr' mode or 'pdf' mode, return error code 1 and some non-Unicode
-text because tesseract seems to do that in some cases related to
-language pack version mismatches
-
-"""
-
-
-VERSION_STRING = '''tesseract 4.0.0
- leptonica-1.77.0
-  libjpeg 9c : libpng 1.6.35 : libtiff 4.0.10 : zlib 1.2.11 : libopenjp2 2.3.0
- Found AVX2
- Found AVX
- Found SSE
-SPOOFED
-'''
-
-# Japanese "Invalid UTF-8" encoded in Shift JIS
-BAD_UTF8 = b'\x96\xb3\x8c\xf8\x82\xc8UTF-8\x0a'
-
-
-def main():
-    if sys.argv[1] == '--version':
-        print(VERSION_STRING, file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[1] == '--list-langs':
-        print('List of available languages (1):\neng', file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[-2] == '--print-parameters':
-        print("Some parameters", file=sys.stderr)
-        print("textonly_pdf\t1\tSome help text")
-        sys.exit(0)
-    elif sys.argv[-2] in ('hocr', 'pdf'):
-        sys.stdout.buffer.write(BAD_UTF8)
-        sys.exit(1)
-    elif sys.argv[-1] == 'stdout':
-        # input file is at sys.argv[-2] but we don't look at it
-        print(
-            """Orientation: 0
-Orientation in degrees: 0
-Orientation confidence: 100.00
-Script: 1
-Script confidence: 100.00""",
-            file=sys.stderr,
-        )
-    else:
-        print("Spoof doesn't understand arguments", file=sys.stderr)
-        print(sys.argv, file=sys.stderr)
-        sys.exit(1)
-
-    sys.exit(0)
-
-
-if __name__ == '__main__':
-    main()

tests/test_api.py

@@ -59,3 +59,10 @@ def test_tqdm_console():
 
     log.info("done")
     assert not before_pbar("done")
+
+
+def test_language_list():
+    with pytest.raises(
+        (ocrmypdf.exceptions.InputFileError, ocrmypdf.exceptions.MissingDependencyError)
+    ):
+        ocrmypdf.ocr('doesnotexist.pdf', '_.pdf', language=['eng', 'deu'])

tests/test_graft.py

@@ -25,25 +25,30 @@ import ocrmypdf
 
 
 def test_no_glyphless_graft(resources, outdir):
-    pdf = pikepdf.open(resources / 'francais.pdf')
-    pdf_aspect = pikepdf.open(resources / 'aspect.pdf')
-    pdf_cmyk = pikepdf.open(resources / 'cmyk.pdf')
-    pdf.pages.extend(pdf_aspect.pages)
-    pdf.pages.extend(pdf_cmyk.pages)
-    pdf.save(outdir / 'test.pdf')
+    with pikepdf.open(resources / 'francais.pdf') as pdf, pikepdf.open(
+        resources / 'aspect.pdf'
+    ) as pdf_aspect, pikepdf.open(resources / 'cmyk.pdf') as pdf_cmyk:
+        pdf.pages.extend(pdf_aspect.pages)
+        pdf.pages.extend(pdf_cmyk.pages)
+        pdf.save(outdir / 'test.pdf')
 
     with patch('ocrmypdf._graft.MAX_REPLACE_PAGES', 2):
         ocrmypdf.ocr(
-            outdir / 'test.pdf', outdir / 'out.pdf', deskew=True, tesseract_timeout=0
+            outdir / 'test.pdf',
+            outdir / 'out.pdf',
+            deskew=True,
+            tesseract_timeout=0,
+            force_ocr=True,
         )
+    # This test needs asserts
 
 
 def test_links(resources, outpdf):
     ocrmypdf.ocr(
         resources / 'link.pdf', outpdf, redo_ocr=True, oversample=200, output_type='pdf'
     )
-    pdf = pikepdf.open(outpdf)
-    p1 = pdf.pages[0]
-    p2 = pdf.pages[1]
-    assert p1.Annots[0].A.D[0].objgen == p2.objgen
-    assert p2.Annots[0].A.D[0].objgen == p1.objgen
+    with pikepdf.open(outpdf) as pdf:
+        p1 = pdf.pages[0]
+        p2 = pdf.pages[1]
+        assert p1.Annots[0].A.D[0].objgen == p2.objgen
+        assert p2.Annots[0].A.D[0].objgen == p1.objgen

tests/test_helpers.py

@@ -17,6 +17,7 @@
 
 import logging
 import multiprocessing
+import os
 from pathlib import Path
 from unittest.mock import MagicMock
 
@@ -95,3 +96,21 @@ class TestFileIsWritable:
         pathmock.exists.return_value = True
         pathmock.is_file.side_effect = PermissionError
         assert not helpers.is_file_writable(pathmock)
+
+
+def test_shim_paths(tmp_path):
+    progfiles = tmp_path / 'Program Files'
+    progfiles.mkdir()
+    (progfiles / 'tesseract-ocr').mkdir()
+    (progfiles / 'gs' / '9.51' / 'bin').mkdir(parents=True)
+    (progfiles / 'gs' / '9.52' / 'bin').mkdir(parents=True)
+    syspath = tmp_path / 'bin'
+    env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
+    from ocrmypdf.exec import shim_paths_with_program_files
+
+    result_str = shim_paths_with_program_files(env=env)
+    results = result_str.split(os.pathsep)
+    assert results[0].endswith('tesseract-ocr')
+    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
+    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
+    assert results[3] == str(syspath)

tests/test_optimize.py

@@ -18,10 +18,12 @@
 import logging
 from os import fspath
 from pathlib import Path
+from unittest.mock import patch
 
+import img2pdf
 import pikepdf
 import pytest
-from PIL import Image
+from PIL import Image, ImageDraw
 
 from ocrmypdf import optimize as opt
 from ocrmypdf.exec import jbig2enc, pngquant
@@ -130,3 +132,41 @@ def test_flate_to_jbig2(resources, outdir, spoof_tesseract_noop):
     pdf = pikepdf.open(outdir / 'out.pdf')
     pim = pikepdf.PdfImage(next(iter(pdf.pages[0].images.values())))
     assert pim.filters[0] == '/JBIG2Decode'
+
+
+def test_multiple_pngs(resources, outdir, spoof_tesseract_noop):
+    with Path.open(outdir / 'in.pdf', 'wb') as inpdf:
+        img2pdf.convert(
+            fspath(resources / 'baiona_colormapped.png'),
+            fspath(resources / 'baiona_gray.png'),
+            with_pdfrw=False,
+            outputstream=inpdf,
+        )
+
+    def mockquant(input_file, output_file, _quality_min, _quality_max):
+        with Image.open(input_file) as im:
+            draw = ImageDraw.Draw(im)
+            draw.rectangle((0, 0, im.width, im.height), fill=128)
+            im.save(output_file)
+
+    with patch('ocrmypdf.optimize.pngquant.quantize', new=mockquant):
+        check_ocrmypdf(
+            outdir / 'in.pdf',
+            outdir / 'out.pdf',
+            '--optimize',
+            '3',
+            '--jobs',
+            '1',
+            '--use-threads',
+            '--output-type',
+            'pdf',
+            env=spoof_tesseract_noop,
+        )
+
+    with pikepdf.open(outdir / 'in.pdf') as inpdf, pikepdf.open(
+        outdir / 'out.pdf'
+    ) as outpdf:
+        for n in range(len(inpdf.pages)):
+            inim = next(iter(inpdf.pages[n].images.values()))
+            outim = next(iter(outpdf.pages[n].images.values()))
+            assert len(outim.read_raw_bytes()) < len(inim.read_raw_bytes()), n

tests/test_pdfinfo.py

@@ -26,6 +26,7 @@ from PIL import Image
 from reportlab.pdfgen.canvas import Canvas
 
 from ocrmypdf import pdfinfo
+from ocrmypdf.exec import ghostscript
 from ocrmypdf.pdfinfo import Colorspace, Encoding
 
 # pylint: disable=protected-access
@@ -183,6 +184,9 @@ def test_ocr_detection(resources):
 @pytest.mark.parametrize(
     'testfile', ('truetype_font_nomapping.pdf', 'type3_font_nomapping.pdf')
 )
+@pytest.mark.xfail(
+    ghostscript.version() in ('9.52',), reason="gs 9.52 txtwrite doesn't work"
+)
 def test_corrupt_font_detection(resources, testfile):
     filename = resources / testfile
     with pytest.raises(NotImplementedError):

tests/test_stdio.py

@@ -33,11 +33,6 @@ run_ocrmypdf_api = pytest.helpers.run_ocrmypdf
 spoof = pytest.helpers.spoof
 
 
-@pytest.fixture
-def spoof_tess_bad_utf8(tmp_path_factory):
-    return spoof(tmp_path_factory, tesseract='tesseract_badutf8.py')
-
-
 def test_stdin(spoof_tesseract_noop, ocrmypdf_exec, resources, outpdf):
     input_file = str(resources / 'francais.pdf')
     output_file = str(outpdf)

tests/test_validation.py

@@ -19,6 +19,7 @@ import logging
 import os
 from unittest.mock import patch
 
+import pikepdf
 import pytest
 
 import ocrmypdf._validation as vd
@@ -111,15 +112,20 @@ def test_output_tty():
 def test_report_file_size(tmp_path, caplog):
     in_ = tmp_path / 'a.pdf'
     out = tmp_path / 'b.pdf'
-    in_.write_bytes(b'123')
-    out.write_bytes(b'')
+    pdf = pikepdf.new()
+    pdf.save(in_)
+    pdf.save(out)
     opts = make_opts()
     vd.report_output_file_size(opts, in_, out)
     assert caplog.text == ''
     caplog.clear()
 
-    os.truncate(in_, 25001)
-    os.truncate(out, 50000)
+    waste_of_space = b'Dummy' * 5000
+    pdf.root.Dummy = waste_of_space
+    pdf.save(in_)
+    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.save(out)
+
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(
         'ocrmypdf._validation.pngquant.available', return_value=True
     ):
@@ -176,3 +182,58 @@ def test_language_warning(caplog):
         vd.check_options_languages(opts)
         assert opts.language == ['eng']
         assert 'assuming --language' in caplog.text
+
+
+def test_version_comparison():
+    vd.check_external_program(
+        program="dummy_basic",
+        package="dummy",
+        version_checker=lambda: '9.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="dummy_doubledigit",
+        package="dummy",
+        version_checker=lambda: '10.0',
+        need_version='8.0.2',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: '4.0.0-beta.1',
+        need_version='4.0.0',
+    )
+    vd.check_external_program(
+        program="tesseract",
+        package="tesseract",
+        version_checker=lambda: 'v5.0.0-alpha.20200201',
+        need_version='4.0.0',
+    )
+    with pytest.raises(MissingDependencyError):
+        vd.check_external_program(
+            program="dummy_fails",
+            package="dummy",
+            version_checker=lambda: '1.0',
+            need_version='2.0',
+        )
+
+
+def test_optional_program_recommended(caplog):
+    caplog.clear()
+
+    def raiser():
+        raise FileNotFoundError('jbig2')
+
+    with caplog.at_level(logging.WARNING):
+        vd.check_external_program(
+            program="jbig2",
+            package="jbig2enc",
+            version_checker=raiser,
+            need_version='42',
+            required_for='this test case',
+            recommended=True,
+        )
+        assert any(
+            (loglevel == logging.WARNING and "recommended" in msg)
+            for _logger_name, loglevel, msg in caplog.record_tuples
+        )