Fix test that failed on Windows

Fixes #565

.github/ISSUE_TEMPLATE/bug_report.md

@@ -20,9 +20,13 @@ ocrmypdf  ...arguments... input.pdf output.pdf
 Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
 
 **Example file**
-Please include an example *input* PDF (or image). The input file is more helpful.
+Include an input PDF or image that demonstrates your issue.
 
-If possible, use an input file with no personal or confidential information. At your option you may GPG-encrypt the file for OCRmyPDF's author only.
+Please provide an input file with no personal or confidential information. At your option you may `GPG-encrypt the file <https://github.com/jbarlow83/OCRmyPDF/wiki>` for OCRmyPDF's author only.
+
+Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
+
+(Exceptions: Issues with installation, command line argument parsing, test suite failures.Issues without example files usually cannot be resolved.)
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.

azure-pipelines.yml

@@ -153,12 +153,13 @@ stages:
           matrix:
             Python37:
               python.version: "3.7"
-            Python38:
+            # Python38:
-              python.version: "3.8"
+            #   python.version: "3.8"
         steps:
-          - task: UsePythonVersion@0
+          # https://github.com/actions/virtual-environments/issues/664
-            inputs:
+          # - task: UsePythonVersion@0
-              versionSpec: "$(python.version)"
+          #   inputs:
+          #     versionSpec: "$(python.version)"
           - bash: |
               brew update
               brew unlink python@2
@@ -169,6 +170,7 @@ stages:
                 leptonica \
                 openjpeg \
                 pngquant \
+                python \
                 tesseract \
                 unpaper
             displayName: "Install system packages"

docs/index.rst

@@ -27,6 +27,7 @@ image processing and OCR to existing PDFs.
    advanced
    batch
    security
+   performance
    errors
 
 .. toctree::

docs/installation.rst

@@ -137,11 +137,12 @@ Installing the latest version on Ubuntu 18.04 LTS
 -------------------------------------------------
 
 Ubuntu 18.04 includes ocrmypdf 6.1.2 - you can install that with ``apt``, but
-it is quite old now. To install a more recent version, first install several
+it is quite old now. To install a more recent version, uninstall the old version
-system dependencies:
+of ocrmypdf, and install the following dependencies:
 
 .. code-block:: bash
 
+    sudo apt-get -y remove ocrmypdf
     sudo apt-get -y update
     sudo apt-get -y install \
         ghostscript \
@@ -326,20 +327,7 @@ standard tooling needed to build packages, such as a compiler and binary tools.
 
    sudo pacman -S base-devel
 
-The OCRmyPDF package depends on `the python-pdfminer.six AUR package
+Now you are ready to install the OCRmyPDF package.
-<https://aur.archlinux.org/packages/python-pdfminer.six/>`__.  Dependencies on
-AUR packages are not automatically resolved, so this package must be manually
-installed first.
-
-.. code-block:: bash
-
-   curl -O https://aur.archlinux.org/cgit/aur.git/snapshot/python-pdfminer.six.tar.gz
-   tar xvzf python-pdfminer.six.tar.gz
-   cd python-pdfminer.six
-   makepkg -sri
-
-With that complete you can then repeat the same series of steps for the
-OCRmyPDF package.
 
 .. code-block:: bash
 
@@ -373,11 +361,10 @@ page.
     fine without it but will produce larger output files. The encoder is
     available from `the jbig2enc-git AUR package
     <https://aur.archlinux.org/packages/jbig2enc-git/>`__ and may be installed
-    using the same series of steps as for the installation of the pdfminer.six
+    using the same series of steps as for the installation OCRmyPDF AUR
-    and OCRmyPDF AUR packages. Alternatively, it may be built manually from
+    package. Alternatively, it may be built manually from source following the
-    source following the instructions in `Installing the JBIG2 encoder
+    instructions in `Installing the JBIG2 encoder <jbig2>`__.  If JBIG2 is
-    <jbig2>`__.  If JBIG2 is installed, OCRmyPDF 7.0.0 and later will
+    installed, OCRmyPDF 7.0.0 and later will automatically detect it.
-    automatically detect it.
 
 Alpine Linux
 ------------
@@ -442,7 +429,10 @@ languages you can optionally install them all:
 Manual installation on macOS
 ----------------------------
 
-These instructions probably work on all macOS supported by Homebrew.
+These instructions probably work on all macOS supported by Homebrew, and are
+for installing a more current version of OCRmyPDF than is available from
+Homebrew. Note that the Homebrew versions usually track the release versions
+fairly closely.
 
 If it's not already present, `install Homebrew <http://brew.sh/>`__.
 
@@ -453,14 +443,8 @@ Update Homebrew:
     brew update
 
 Install or upgrade the required Homebrew packages, if any are missing.
-To do this, download the ``Brewfile`` that lists all of the dependencies
+To do this, use ``brew edit ocrmypdf`` to obtain a recent list of Homebrew
-to the current directory, and run ``brew bundle`` to process them
+dependencies. You could also check the ``azure-pipelines.yml``.
-(installing or upgrading as needed). ``Brewfile`` is a plain text file.
-
-.. code-block:: bash
-
-    wget https://github.com/jbarlow83/OCRmyPDF/raw/master/.travis/Brewfile
-    brew bundle
 
 This will include the English, French, German and Spanish language
 packs. If you need other languages you can optionally install them all:

docs/performance.rst

@@ -0,0 +1,22 @@
+===========
+Performance
+===========
+
+Some users have noticed that current versions of OCRmyPDF do not run as quickly
+as some older versions (specifically 6.x and older). This is because OCRmyPDF
+added image optimization as a postprocessing step, and it is enabled by default.
+
+Speed
+=====
+
+If running OCRmyPDF quickly is your main goal, you can use settings such as:
+
+* ``--optimize 0`` to disable file size optimization
+* ``--output-type pdf`` to disable PDF/A generation
+* ``--fast-web-view 0`` to disable fast web view optimization
+* ``--skip-big`` to skip large images, if some pages have large images
+
+You can also avoid:
+
+* ``--force-ocr``
+* Image preprocessing

docs/release_notes.rst

@@ -13,6 +13,24 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.8.1
+======
+
+-  Fixed an issue where unexpected files in the ``%PROGRAMFILES%\gs`` directory
+   (Windows) caused an exception.
+-  Mark pdfminer.six 20200517 as supported.
+-  If jbig2enc is missing and optimization is requested, a warning is issued
+   instead of an error, which was the intended behavior.
+-  Documentation updates.
+
+v9.8.0
+======
+
+-  Fixed issue where only the first PNG (FlateDecode) image in a file would be
+   considered for optimization. File sizes should be improved from here on.
+-  Fixed a startup crash when the chosen language was Japanese (#543).
+-  Added options to configure polling and log level to watcher.py.
+
 v9.7.2
 ======
 

misc/watcher.py

@@ -25,6 +25,7 @@ from pathlib import Path
 import pikepdf
 from watchdog.events import PatternMatchingEventHandler
 from watchdog.observers import Observer
+from watchdog.observers.polling import PollingObserver
 
 import ocrmypdf
 
@@ -37,7 +38,8 @@ ON_SUCCESS_DELETE = bool(os.getenv('OCR_ON_SUCCESS_DELETE', False))
 DESKEW = bool(os.getenv('OCR_DESKEW', False))
 OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = os.getenv('OCR_POLL_NEW_FILE_SECONDS', 1)
-LOGLEVEL = os.environ.get('OCR_LOGLEVEL', 'INFO').upper()
+USE_POLLING = bool(os.getenv('OCR_USE_POLLING', False))
+LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
 PATTERNS = ['*.pdf']
 
 log = logging.getLogger('ocrmypdf-watcher')
@@ -112,6 +114,7 @@ def main():
     ocrmypdf.configure_logging(
         verbosity=ocrmypdf.Verbosity.default, manage_root_logger=True
     )
+    log.setLevel(LOGLEVEL)
     log.info(
         f"Starting OCRmyPDF watcher with config:\n"
         f"Input Directory: {INPUT_DIRECTORY}\n"
@@ -126,6 +129,7 @@ def main():
         f"DESKEW: {DESKEW}\n"
         f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
+        f"USE_POLLING: {USE_POLLING}\n"
         f"LOGLEVEL: {LOGLEVEL}\n"
     )
 
@@ -134,7 +138,10 @@ def main():
         sys.exit(1)
 
     handler = HandleObserverEvent(patterns=PATTERNS)
-    observer = Observer()
+    if USE_POLLING:
+        observer = PollingObserver()
+    else:
+        observer = Observer()
     observer.schedule(handler, INPUT_DIRECTORY, recursive=True)
     observer.start()
     try:

requirements/dev.txt

@@ -1,2 +0,0 @@
-twine >= 1.8.1
-coverage >= 4.5

requirements/main.txt

@@ -2,9 +2,9 @@
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
 cffi == 1.14.0
-img2pdf == 0.3.3
+img2pdf == 0.3.4
 pdfminer.six == 20200402
-pikepdf == 1.10.2
+pikepdf == 1.11.1
-Pillow == 7.0.0
+Pillow == 7.1.1
 reportlab == 3.5.34
-tqdm == 4.42.1
+tqdm == 4.45.0

setup.py

@@ -62,7 +62,7 @@ setup(
     long_description_content_type='text/markdown',
     url='https://github.com/jbarlow83/OCRmyPDF',
     author='James R. Barlow',
-    author_email='jim@purplerock.ca',
+    author_email='james@purplerock.ca',
     packages=find_packages('src', exclude=["tests", "tests.*"]),
     package_dir={'': 'src'},
     keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200402',
+        'pdfminer.six >= 20181108, <= 20200517',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/exec/__init__.py

@@ -25,6 +25,7 @@ import sys
 from collections.abc import Mapping
 from distutils.version import LooseVersion
 from functools import lru_cache
+from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError
 from subprocess import run as subprocess_run
 
@@ -138,24 +139,25 @@ def shim_paths_with_program_files(env=None):
     program_files = env.get('PROGRAMFILES', '')
     if not program_files:
         return env.get('PATH', '')
-    paths = []
+
-    try:
+    def path_walker():
-        for dirname in os.listdir(program_files):
+        for path in Path(program_files).iterdir():
-            if dirname.lower() == 'tesseract-ocr':
+            if not path.is_dir():
-                paths.append(os.path.join(program_files, dirname))
+                continue
-            elif dirname.lower() == 'gs':
+            if path.name.lower() == 'tesseract-ocr':
-                try:
+                yield path
-                    latest_gs = max(
+            elif path.name.lower() == 'gs':
-                        os.listdir(os.path.join(program_files, dirname)),
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
-                        key=lambda d: float(d[2:]),
+
-                    )
+    paths = sorted(
-                except (FileNotFoundError, NotADirectoryError):
+        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
-                    continue
+    )
-                paths.append(os.path.join(program_files, dirname, latest_gs, 'bin'))
+    paths.extend(
-    except EnvironmentError:
+        Path(str_path)
-        pass
+        for str_path in os.get_exec_path(env)
-    paths.extend(path for path in os.get_exec_path(env) if path not in set(paths))
+        if Path(str_path) not in set(paths)
-    return os.pathsep.join(paths)
+    )
+    return os.pathsep.join(str(p) for p in paths)
 
 
 missing_program = '''
@@ -233,10 +235,10 @@ def _error_trailer(program, package, **kwargs):
 
 
 def _error_missing_program(program, package, required_for, recommended):
-    if required_for:
+    if recommended:
+        log.warning(missing_recommend_program.format(**locals()))
+    elif required_for:
         log.error(missing_optional_program.format(**locals()))
-    elif recommended:
-        log.info(missing_recommend_program.format(**locals()))
     else:
         log.error(missing_program.format(**locals()))
     _error_trailer(**locals())
@@ -279,7 +281,7 @@ def check_external_program(
     found_version = remove_leading_v(found_version)
     need_version = remove_leading_v(need_version)
 
-    if LooseVersion(found_version) < LooseVersion(need_version):
+    if found_version and LooseVersion(found_version) < LooseVersion(need_version):
         _error_old_version(program, package, need_version, found_version, required_for)
         if not recommended:
             raise MissingDependencyError()

src/ocrmypdf/exec/tesseract.py

@@ -77,20 +77,14 @@ def has_textonly_pdf(tesseract_env=None, langs=None):
     args_tess = tess_base_args(langs, engine_mode=None) + ['--print-parameters', 'pdf']
     params = ''
     try:
-        proc = run(
+        # print-parameters can return non-UTF8 if the parameters are so initialized
-            args_tess,
+        proc = run(args_tess, check=True, stdout=PIPE, stderr=STDOUT, env=tesseract_env)
-            check=True,
-            universal_newlines=True,
-            stdout=PIPE,
-            stderr=STDOUT,
-            env=tesseract_env,
-        )
         params = proc.stdout
     except CalledProcessError as e:
         raise MissingDependencyError(
             "Could not --print-parameters from tesseract"
         ) from e
-    if 'textonly_pdf' in params:
+    if b'textonly_pdf' in params:
         return True
     return False
 

src/ocrmypdf/optimize.py

@@ -421,9 +421,9 @@ def transcode_pngs(pike, images, image_name_fn, root, log, options):
             )
             continue
         if compdata.type == leptonica.lept.L_FLATE_ENCODE:
-            return rewrite_png(pike, im_obj, compdata, log)
+            rewrite_png(pike, im_obj, compdata, log)
         elif compdata.type == leptonica.lept.L_G4_ENCODE:
-            return rewrite_png_as_g4(pike, im_obj, compdata, log)
+            rewrite_png_as_g4(pike, im_obj, compdata, log)
 
 
 def rewrite_png_as_g4(pike, im_obj, compdata, log):

tests/spoof/tesseract_badutf8.py

@@ -1,80 +0,0 @@
-#!/usr/bin/env python3
-# © 2017 James R. Barlow: github.com/jbarlow83
-#
-# Permission is hereby granted, free of charge, to any person obtaining a
-# copy of this software and associated documentation files (the
-# "Software"), to deal in the Software without restriction, including
-# without limitation the rights to use, copy, modify, merge, publish,
-# distribute, sublicense, and/or sell copies of the Software, and to
-# permit persons to whom the Software is furnished to do so, subject to
-# the following conditions:
-#
-# The above copyright notice and this permission notice shall be included
-# in all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
-# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-# CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-# TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
-import sys
-
-"""Tesseract bad utf8 spoof
-
-In 'hocr' mode or 'pdf' mode, return error code 1 and some non-Unicode
-text because tesseract seems to do that in some cases related to
-language pack version mismatches
-
-"""
-
-
-VERSION_STRING = '''tesseract 4.0.0
- leptonica-1.77.0
-  libjpeg 9c : libpng 1.6.35 : libtiff 4.0.10 : zlib 1.2.11 : libopenjp2 2.3.0
- Found AVX2
- Found AVX
- Found SSE
-SPOOFED
-'''
-
-# Japanese "Invalid UTF-8" encoded in Shift JIS
-BAD_UTF8 = b'\x96\xb3\x8c\xf8\x82\xc8UTF-8\x0a'
-
-
-def main():
-    if sys.argv[1] == '--version':
-        print(VERSION_STRING, file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[1] == '--list-langs':
-        print('List of available languages (1):\neng', file=sys.stderr)
-        sys.exit(0)
-    elif sys.argv[-2] == '--print-parameters':
-        print("Some parameters", file=sys.stderr)
-        print("textonly_pdf\t1\tSome help text")
-        sys.exit(0)
-    elif sys.argv[-2] in ('hocr', 'pdf'):
-        sys.stdout.buffer.write(BAD_UTF8)
-        sys.exit(1)
-    elif sys.argv[-1] == 'stdout':
-        # input file is at sys.argv[-2] but we don't look at it
-        print(
-            """Orientation: 0
-Orientation in degrees: 0
-Orientation confidence: 100.00
-Script: 1
-Script confidence: 100.00""",
-            file=sys.stderr,
-        )
-    else:
-        print("Spoof doesn't understand arguments", file=sys.stderr)
-        print(sys.argv, file=sys.stderr)
-        sys.exit(1)
-
-    sys.exit(0)
-
-
-if __name__ == '__main__':
-    main()

tests/test_helpers.py

@@ -17,6 +17,7 @@
 
 import logging
 import multiprocessing
+import os
 from pathlib import Path
 from unittest.mock import MagicMock
 
@@ -95,3 +96,21 @@ class TestFileIsWritable:
         pathmock.exists.return_value = True
         pathmock.is_file.side_effect = PermissionError
         assert not helpers.is_file_writable(pathmock)
+
+
+def test_shim_paths(tmp_path):
+    progfiles = tmp_path / 'Program Files'
+    progfiles.mkdir()
+    (progfiles / 'tesseract-ocr').mkdir()
+    (progfiles / 'gs' / '9.51' / 'bin').mkdir(parents=True)
+    (progfiles / 'gs' / '9.52' / 'bin').mkdir(parents=True)
+    syspath = tmp_path / 'bin'
+    env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
+    from ocrmypdf.exec import shim_paths_with_program_files
+
+    result_str = shim_paths_with_program_files(env=env)
+    results = result_str.split(os.pathsep)
+    assert results[0].endswith('tesseract-ocr')
+    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
+    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
+    assert results[3] == str(syspath)

tests/test_optimize.py

@@ -18,10 +18,12 @@
 import logging
 from os import fspath
 from pathlib import Path
+from unittest.mock import patch
 
+import img2pdf
 import pikepdf
 import pytest
-from PIL import Image
+from PIL import Image, ImageDraw
 
 from ocrmypdf import optimize as opt
 from ocrmypdf.exec import jbig2enc, pngquant
@@ -130,3 +132,41 @@ def test_flate_to_jbig2(resources, outdir, spoof_tesseract_noop):
     pdf = pikepdf.open(outdir / 'out.pdf')
     pim = pikepdf.PdfImage(next(iter(pdf.pages[0].images.values())))
     assert pim.filters[0] == '/JBIG2Decode'
+
+
+def test_multiple_pngs(resources, outdir, spoof_tesseract_noop):
+    with Path.open(outdir / 'in.pdf', 'wb') as inpdf:
+        img2pdf.convert(
+            fspath(resources / 'baiona_colormapped.png'),
+            fspath(resources / 'baiona_gray.png'),
+            with_pdfrw=False,
+            outputstream=inpdf,
+        )
+
+    def mockquant(input_file, output_file, _quality_min, _quality_max):
+        with Image.open(input_file) as im:
+            draw = ImageDraw.Draw(im)
+            draw.rectangle((0, 0, im.width, im.height), fill=128)
+            im.save(output_file)
+
+    with patch('ocrmypdf.optimize.pngquant.quantize', new=mockquant):
+        check_ocrmypdf(
+            outdir / 'in.pdf',
+            outdir / 'out.pdf',
+            '--optimize',
+            '3',
+            '--jobs',
+            '1',
+            '--use-threads',
+            '--output-type',
+            'pdf',
+            env=spoof_tesseract_noop,
+        )
+
+    with pikepdf.open(outdir / 'in.pdf') as inpdf, pikepdf.open(
+        outdir / 'out.pdf'
+    ) as outpdf:
+        for n in range(len(inpdf.pages)):
+            inim = next(iter(inpdf.pages[n].images.values()))
+            outim = next(iter(outpdf.pages[n].images.values()))
+            assert len(outim.read_raw_bytes()) < len(inim.read_raw_bytes()), n

tests/test_stdio.py

@@ -33,11 +33,6 @@ run_ocrmypdf_api = pytest.helpers.run_ocrmypdf
 spoof = pytest.helpers.spoof
 
 
-@pytest.fixture
-def spoof_tess_bad_utf8(tmp_path_factory):
-    return spoof(tmp_path_factory, tesseract='tesseract_badutf8.py')
-
-
 def test_stdin(spoof_tesseract_noop, ocrmypdf_exec, resources, outpdf):
     input_file = str(resources / 'francais.pdf')
     output_file = str(outpdf)

tests/test_validation.py

@@ -210,3 +210,24 @@ def test_version_comparison():
             version_checker=lambda: '1.0',
             need_version='2.0',
         )
+
+
+def test_optional_program_recommended(caplog):
+    caplog.clear()
+
+    def raiser():
+        raise FileNotFoundError('jbig2')
+
+    with caplog.at_level(logging.WARNING):
+        vd.check_external_program(
+            program="jbig2",
+            package="jbig2enc",
+            version_checker=raiser,
+            need_version='42',
+            required_for='this test case',
+            recommended=True,
+        )
+        assert any(
+            (loglevel == logging.WARNING and "recommended" in msg)
+            for _logger_name, loglevel, msg in caplog.record_tuples
+        )