v9.8.2 release notes

Use more realistic test data

.github/ISSUE_TEMPLATE/bug_report.md

@@ -20,9 +20,13 @@ ocrmypdf  ...arguments... input.pdf output.pdf
 Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
 
 **Example file**
-Please include an example *input* PDF (or image). The input file is more helpful.
+Include an input PDF or image that demonstrates your issue.
 
-If possible, use an input file with no personal or confidential information. At your option you may GPG-encrypt the file for OCRmyPDF's author only.
+Please provide an input file with no personal or confidential information. At your option you may `GPG-encrypt the file <https://github.com/jbarlow83/OCRmyPDF/wiki>` for OCRmyPDF's author only.
+
+Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
+
+(Exceptions: Issues with installation, command line argument parsing, test suite failures.Issues without example files usually cannot be resolved.)
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.

docs/index.rst

@@ -27,6 +27,7 @@ image processing and OCR to existing PDFs.
    advanced
    batch
    security
+   performance
    errors
 
 .. toctree::

docs/installation.rst

@@ -327,20 +327,7 @@ standard tooling needed to build packages, such as a compiler and binary tools.
 
    sudo pacman -S base-devel
 
-The OCRmyPDF package depends on `the python-pdfminer.six AUR package
+Now you are ready to install the OCRmyPDF package.
-<https://aur.archlinux.org/packages/python-pdfminer.six/>`__.  Dependencies on
-AUR packages are not automatically resolved, so this package must be manually
-installed first.
-
-.. code-block:: bash
-
-   curl -O https://aur.archlinux.org/cgit/aur.git/snapshot/python-pdfminer.six.tar.gz
-   tar xvzf python-pdfminer.six.tar.gz
-   cd python-pdfminer.six
-   makepkg -sri
-
-With that complete you can then repeat the same series of steps for the
-OCRmyPDF package.
 
 .. code-block:: bash
 
@@ -374,11 +361,10 @@ page.
     fine without it but will produce larger output files. The encoder is
     available from `the jbig2enc-git AUR package
     <https://aur.archlinux.org/packages/jbig2enc-git/>`__ and may be installed
-    using the same series of steps as for the installation of the pdfminer.six
+    using the same series of steps as for the installation OCRmyPDF AUR
-    and OCRmyPDF AUR packages. Alternatively, it may be built manually from
+    package. Alternatively, it may be built manually from source following the
-    source following the instructions in `Installing the JBIG2 encoder
+    instructions in `Installing the JBIG2 encoder <jbig2>`__.  If JBIG2 is
-    <jbig2>`__.  If JBIG2 is installed, OCRmyPDF 7.0.0 and later will
+    installed, OCRmyPDF 7.0.0 and later will automatically detect it.
-    automatically detect it.
 
 Alpine Linux
 ------------
@@ -443,7 +429,10 @@ languages you can optionally install them all:
 Manual installation on macOS
 ----------------------------
 
-These instructions probably work on all macOS supported by Homebrew.
+These instructions probably work on all macOS supported by Homebrew, and are
+for installing a more current version of OCRmyPDF than is available from
+Homebrew. Note that the Homebrew versions usually track the release versions
+fairly closely.
 
 If it's not already present, `install Homebrew <http://brew.sh/>`__.
 
@@ -454,14 +443,8 @@ Update Homebrew:
     brew update
 
 Install or upgrade the required Homebrew packages, if any are missing.
-To do this, download the ``Brewfile`` that lists all of the dependencies
+To do this, use ``brew edit ocrmypdf`` to obtain a recent list of Homebrew
-to the current directory, and run ``brew bundle`` to process them
+dependencies. You could also check the ``azure-pipelines.yml``.
-(installing or upgrading as needed). ``Brewfile`` is a plain text file.
-
-.. code-block:: bash
-
-    wget https://github.com/jbarlow83/OCRmyPDF/raw/master/.travis/Brewfile
-    brew bundle
 
 This will include the English, French, German and Spanish language
 packs. If you need other languages you can optionally install them all:
@@ -592,6 +575,53 @@ Docker
 You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
+Installing on Cygwin64 under Windows
+====================================
+
+First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::
+
+    python36 (or later)
+    python3?-devel
+    python3?-pip
+    python3?-lxml
+    python3?-imaging
+
+       (where 3? means match the version of python3 you installed)
+
+    gcc-g++
+    ghostscript (<=9.50 or >=9.52-2 see note below)
+    libexempi3
+    libexempi-devel
+    libffi6
+    libffi-devel
+    pngquant
+    qpdf
+    libqpdf-devel
+    tesseract-ocr
+    tesseract-ocr-devel
+
+.. note::
+
+    The Cygwin package for Ghostscript in versions 9.52 and
+    9.52-1 contained a bug that caused an exception to occur when
+    ocrmypdf invoked gs.  Make sure you have either 9.50 (or earlier)
+    or 9.52-2 (or later).
+
+Then open a Cygwin terminal (i.e. ``mintty``), run the following commands. Note
+that if you are using the version of ``pip`` that was installed with the Cygwin
+Python package, the command name will be ``pip3``.  If you have since updated
+``pip`` (with, for instance ``pip3 install --upgrade pip``) the the command is
+likely just ``pip`` instead of ``pip3``:
+
+.. code-block:: bash
+
+    pip3 install wheel
+    pip3 install ocrmypdf
+
+The optional dependency "unpaper" that is currently not available under Cygwin.
+Without it, certain options such as ``--clean`` will produce an error message.
+However, the OCR-to-text-layer functionality is available.
+
 Installing with Python pip
 ==========================
 

docs/performance.rst

@@ -0,0 +1,22 @@
+===========
+Performance
+===========
+
+Some users have noticed that current versions of OCRmyPDF do not run as quickly
+as some older versions (specifically 6.x and older). This is because OCRmyPDF
+added image optimization as a postprocessing step, and it is enabled by default.
+
+Speed
+=====
+
+If running OCRmyPDF quickly is your main goal, you can use settings such as:
+
+* ``--optimize 0`` to disable file size optimization
+* ``--output-type pdf`` to disable PDF/A generation
+* ``--fast-web-view 0`` to disable fast web view optimization
+* ``--skip-big`` to skip large images, if some pages have large images
+
+You can also avoid:
+
+* ``--force-ocr``
+* Image preprocessing

docs/release_notes.rst

@@ -13,6 +13,24 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.8.2
+======
+
+-  Fixed an issue where OCRmyPDF would ignore text inside Form XObject when
+   making certain decisions about whether a document already had text.
+-  Fixed file size increase warning to take overhead of small files into account.
+-  Added instructions for installing on Cygwin.
+
+v9.8.1
+======
+
+-  Fixed an issue where unexpected files in the ``%PROGRAMFILES%\gs`` directory
+   (Windows) caused an exception.
+-  Mark pdfminer.six 20200517 as supported.
+-  If jbig2enc is missing and optimization is requested, a warning is issued
+   instead of an error, which was the intended behavior.
+-  Documentation updates.
+
 v9.8.0
 ======
 

setup.py

@@ -62,7 +62,7 @@ setup(
     long_description_content_type='text/markdown',
     url='https://github.com/jbarlow83/OCRmyPDF',
     author='James R. Barlow',
-    author_email='jim@purplerock.ca',
+    author_email='james@purplerock.ca',
     packages=find_packages('src', exclude=["tests", "tests.*"]),
     package_dir={'': 'src'},
     keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
@@ -98,7 +98,7 @@ setup(
         'chardet >= 3.0.4, < 4',  # unlisted requirement of pdfminer.six 20181108
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20181108, <= 20200402',
+        'pdfminer.six >= 20181108, <= 20200517',
         'pikepdf >= 1.8.1, < 2',
         'Pillow >= 6.2.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling

src/ocrmypdf/_validation.py

@@ -24,6 +24,7 @@ import sys
 from pathlib import Path
 from shutil import copyfileobj
 
+import pikepdf
 import PIL
 
 from ._unicodefun import verify_python3_env
@@ -410,8 +411,15 @@ def report_output_file_size(options, input_file, output_file):
         input_size = Path(input_file).stat().st_size
     except FileNotFoundError:
         return  # Outputting to stream or something
+    with pikepdf.open(output_file) as p:
+        # Overhead constants obtained by estimating amount of data added by OCR
+        # PDF/A conversion, and possible XMP metadata addition, with compression
+        FILE_OVERHEAD = 4000
+        OCR_PER_PAGE_OVERHEAD = 3000
+        reasonable_overhead = FILE_OVERHEAD + OCR_PER_PAGE_OVERHEAD * len(p.pages)
     ratio = output_size / input_size
-    if ratio < 1.35 or input_size < 25000:
+    reasonable_ratio = output_size / (input_size + reasonable_overhead)
+    if reasonable_ratio < 1.35 or input_size < 25000:
         return  # Seems fine
 
     reasons = []

src/ocrmypdf/exec/__init__.py

@@ -25,6 +25,7 @@ import sys
 from collections.abc import Mapping
 from distutils.version import LooseVersion
 from functools import lru_cache
+from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError
 from subprocess import run as subprocess_run
 
@@ -138,24 +139,25 @@ def shim_paths_with_program_files(env=None):
     program_files = env.get('PROGRAMFILES', '')
     if not program_files:
         return env.get('PATH', '')
-    paths = []
+
-    try:
+    def path_walker():
-        for dirname in os.listdir(program_files):
+        for path in Path(program_files).iterdir():
-            if dirname.lower() == 'tesseract-ocr':
+            if not path.is_dir():
-                paths.append(os.path.join(program_files, dirname))
+                continue
-            elif dirname.lower() == 'gs':
+            if path.name.lower() == 'tesseract-ocr':
-                try:
+                yield path
-                    latest_gs = max(
+            elif path.name.lower() == 'gs':
-                        os.listdir(os.path.join(program_files, dirname)),
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
-                        key=lambda d: float(d[2:]),
+
-                    )
+    paths = sorted(
-                except (FileNotFoundError, NotADirectoryError):
+        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
-                    continue
+    )
-                paths.append(os.path.join(program_files, dirname, latest_gs, 'bin'))
+    paths.extend(
-    except EnvironmentError:
+        Path(str_path)
-        pass
+        for str_path in os.get_exec_path(env)
-    paths.extend(path for path in os.get_exec_path(env) if path not in set(paths))
+        if Path(str_path) not in set(paths)
-    return os.pathsep.join(paths)
+    )
+    return os.pathsep.join(str(p) for p in paths)
 
 
 missing_program = '''
@@ -233,10 +235,10 @@ def _error_trailer(program, package, **kwargs):
 
 
 def _error_missing_program(program, package, required_for, recommended):
-    if required_for:
+    if recommended:
+        log.warning(missing_recommend_program.format(**locals()))
+    elif required_for:
         log.error(missing_optional_program.format(**locals()))
-    elif recommended:
-        log.info(missing_recommend_program.format(**locals()))
     else:
         log.error(missing_program.format(**locals()))
     _error_trailer(**locals())
@@ -279,7 +281,7 @@ def check_external_program(
     found_version = remove_leading_v(found_version)
     need_version = remove_leading_v(need_version)
 
-    if LooseVersion(found_version) < LooseVersion(need_version):
+    if found_version and LooseVersion(found_version) < LooseVersion(need_version):
         _error_old_version(program, package, need_version, found_version, required_for)
         if not recommended:
             raise MissingDependencyError()

src/ocrmypdf/pdfinfo/layout.py

@@ -269,7 +269,9 @@ class TextPositionTracker(PDFLayoutAnalyzer):
 
 def get_page_analysis(infile, pageno, pscript5_mode):
     rman = pdfminer.pdfinterp.PDFResourceManager(caching=True)
-    dev = TextPositionTracker(rman, laparams=LAParams())
+    dev = TextPositionTracker(
+        rman, laparams=LAParams(all_texts=True, detect_vertical=True)
+    )
     interp = pdfminer.pdfinterp.PDFPageInterpreter(rman, dev)
 
     if pscript5_mode:

tests/test_helpers.py

@@ -17,6 +17,7 @@
 
 import logging
 import multiprocessing
+import os
 from pathlib import Path
 from unittest.mock import MagicMock
 
@@ -95,3 +96,21 @@ class TestFileIsWritable:
         pathmock.exists.return_value = True
         pathmock.is_file.side_effect = PermissionError
         assert not helpers.is_file_writable(pathmock)
+
+
+def test_shim_paths(tmp_path):
+    progfiles = tmp_path / 'Program Files'
+    progfiles.mkdir()
+    (progfiles / 'tesseract-ocr').mkdir()
+    (progfiles / 'gs' / '9.51' / 'bin').mkdir(parents=True)
+    (progfiles / 'gs' / '9.52' / 'bin').mkdir(parents=True)
+    syspath = tmp_path / 'bin'
+    env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
+    from ocrmypdf.exec import shim_paths_with_program_files
+
+    result_str = shim_paths_with_program_files(env=env)
+    results = result_str.split(os.pathsep)
+    assert results[0].endswith('tesseract-ocr')
+    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
+    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
+    assert results[3] == str(syspath)

tests/test_validation.py

@@ -19,6 +19,7 @@ import logging
 import os
 from unittest.mock import patch
 
+import pikepdf
 import pytest
 
 import ocrmypdf._validation as vd
@@ -111,15 +112,20 @@ def test_output_tty():
 def test_report_file_size(tmp_path, caplog):
     in_ = tmp_path / 'a.pdf'
     out = tmp_path / 'b.pdf'
-    in_.write_bytes(b'123')
+    pdf = pikepdf.new()
-    out.write_bytes(b'')
+    pdf.save(in_)
+    pdf.save(out)
     opts = make_opts()
     vd.report_output_file_size(opts, in_, out)
     assert caplog.text == ''
     caplog.clear()
 
-    os.truncate(in_, 25001)
+    waste_of_space = b'Dummy' * 5000
-    os.truncate(out, 50000)
+    pdf.root.Dummy = waste_of_space
+    pdf.save(in_)
+    pdf.root.Dummy2 = waste_of_space + waste_of_space
+    pdf.save(out)
+
     with patch('ocrmypdf._validation.jbig2enc.available', return_value=True), patch(
         'ocrmypdf._validation.pngquant.available', return_value=True
     ):
@@ -210,3 +216,24 @@ def test_version_comparison():
             version_checker=lambda: '1.0',
             need_version='2.0',
         )
+
+
+def test_optional_program_recommended(caplog):
+    caplog.clear()
+
+    def raiser():
+        raise FileNotFoundError('jbig2')
+
+    with caplog.at_level(logging.WARNING):
+        vd.check_external_program(
+            program="jbig2",
+            package="jbig2enc",
+            version_checker=raiser,
+            need_version='42',
+            required_for='this test case',
+            recommended=True,
+        )
+        assert any(
+            (loglevel == logging.WARNING and "recommended" in msg)
+            for _logger_name, loglevel, msg in caplog.record_tuples
+        )