Mention that v9.0.4 requires a source install for Py3.8 for now, due to lack of CI availability

Closes #337

.travis.yml

@@ -88,8 +88,31 @@ matrix:
             - tesseract-ocr-eng
             - tesseract-ocr-fra
             - unpaper
+    # - os: linux
+    #   dist: xenial
+    #   sudo: required
+    #   language: python
+    #   python: "3.8"
+    #   env:
+    #     - DIST=xenial
+    #   addons:
+    #     apt:
+    #       update: true
+    #       sources:
+    #         - sourceline: "ppa:alex-p/tesseract-ocr"
+    #       packages:
+    #         - ghostscript
+    #         - libexempi3
+    #         - libffi-dev
+    #         - pngquant
+    #         - poppler-utils
+    #         - qpdf
+    #         - tesseract-ocr
+    #         - tesseract-ocr-deu
+    #         - tesseract-ocr-eng
+    #         - tesseract-ocr-fra
+    #         - unpaper
     - os: osx
-      osx_image: xcode9.2
       language: generic
       addons:
         homebrew:

README.md

@@ -1,6 +1,6 @@
 <img src="docs/images/logo.svg" width="240" alt="OCRmyPDF">
 
-[![Travis build status][travis]](https://travis-ci.org/jbarlow83/OCRmyPDF) [![PyPI version][pypi]](https://pypi.org/project/ocrmypdf/) ![Homebrew version][homebrew] ![ReadTheDocs][docs]
+[![Travis build status][travis]](https://travis-ci.org/jbarlow83/OCRmyPDF) [![PyPI version][pypi]](https://pypi.org/project/ocrmypdf/) ![Homebrew version][homebrew] ![ReadTheDocs][docs] ![Python versions][pyversions]
 
 [travis]: https://travis-ci.org/jbarlow83/OCRmyPDF.svg?branch=master "Travis build status"
 
@@ -10,6 +10,8 @@
 
 [docs]: https://readthedocs.org/projects/ocrmypdf/badge/?version=latest "RTD"
 
+[pyversions]: https://img.shields.io/pypi/pyversions/ocrmypdf "Supported Python versions"
+
 OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched or copy-pasted.
 
 ```bash
@@ -120,7 +122,7 @@ If you detect an issue, please:
 Requirements
 ------------
 
-Runs on CPython 3.5, 3.6 and 3.7. Requires external program installations of Ghostscript, Tesseract OCR, QPDF, and Leptonica. ocrmypdf is pure Python, but uses CFFI to portably generate library bindings.
+In addition to the required Python version (3.6+), OCRmyPDF requires external program installations of Ghostscript, Tesseract OCR, QPDF, and Leptonica. ocrmypdf is pure Python, but uses CFFI to portably generate library bindings.
 
 Press & Media
 -------------

docs/advanced.rst

@@ -322,15 +322,23 @@ working files on a per page basis have the page number as a prefix
 (starting with page 1), an infix indicates the processing stage, and a
 suffix indicates the file type. Some important files include:
 
--  ``.page.png`` - what the input page looks like
+-  ``_rasterize.png`` - what the input page looks like
--  ``.image`` - the image we will show the user if we are in a mode that
+-  ``_ocr.png`` - the file that is sent to Tesseract for OCR; depending
-   changes the final appearance; may be in one of several image formats
--  ``.text.pdf`` - the OCR file; this will load as a blank page but
-   should have visible text if checked with a tool like pdftotext or
-   pdfminder.six
--  ``.ocr.png`` - the file that is sent to Tesseract for OCR; depending
    on arguments this may differ from the presentation image
--  ``layers.rendered.pdf`` - the composite PDF, before metadata repair
+-  ``_pp_deskew.png`` - the image, after deskewing
-   and optimization
+-  ``_pp_clean.png`` - the image, after cleaning with unpaper
+-  ``_ocr_tess.pdf`` - the OCR file; appears as a blank page with invisible
+   text embedded
+-  ``_ocr_tess.txt`` - the OCR text (not necessarily all text on the page,
+   if the page is mixed format)
+-  ``fix_docinfo.pdf`` - a temporary file created to fix the PDF DocumentInfo
+   data structure
+-  ``graft_layers.pdf`` - the rendered PDF with OCR layers grafted on
+-  ``pdfa.pdf`` - ``graft_layers.pdf`` after conversion to PDF/A
+-  ``pdfa.ps`` - a PostScript file used by Ghostscript for PDF/A conversion
+-  ``optimize.pdf`` - the PDF generated before optimization
+-  ``optimize.out.pdf`` - the PDF generated by optimization
+-  ``origin`` - the input file
+-  ``origin.pdf`` - the input file or the input image converted to PDF
 -  ``images/*`` - images extracted during the optimization process; here
    the prefix indicates a PDF object ID not a page number

docs/contributing.rst

@@ -0,0 +1,38 @@
+=======================
+Contributing guidelines
+=======================
+
+Contributions are welcome!
+
+Big changes
+===========
+
+Please open a new issue to discuss or propose a major change. Not only is it fun
+to discuss big ideas, but we might save each other's time too. Perhaps some of the
+work you're contemplating is already half-done in a development branch.
+
+Code style
+==========
+
+We use PEP8, ``black`` for code formatting and ``isort`` for import sorting. The
+settings for programs are in ``pyproject.toml`` and ``setup.cfg``.
+
+Tests
+=====
+
+New features should come with tests that confirm their correctness.
+
+New Python dependencies
+=======================
+
+If you are proposing a change that will require a new Python dependency, we
+prefer dependencies that are already packaged by Debian or Red Hat. This makes
+life much easier for our downstream package maintainers.
+
+Python dependencies must also be GPLv3 compatible.
+
+New non-Python dependencies
+===========================
+
+OCRmyPDF uses several external programs (Tesseract, Ghostscript and others) for
+its functionality. In general we prefer to avoid adding new external programs.

docs/index.rst

@@ -22,11 +22,16 @@ PDF is the best format for storing and exchanging scanned documents.  Unfortunat
    cookbook
    docker
    advanced
-   api
    batch
    security
    errors
 
+.. toctree::
+   :caption: Developers
+   :maxdepth: 2
+
+   api
+   contributing
 
 Indices and tables
 ==================

docs/installation.rst

@@ -517,7 +517,7 @@ manager. ``pip`` cannot provide them.
 
 As of ocrmypdf 7.2.1, the following versions are recommended:
 
--  Python 3.7
+-  Python 3.7 or 3.8
 -  Ghostscript 9.23 or newer
 -  qpdf 8.2.1
 -  Tesseract 4.0.0 or newer

docs/release_notes.rst

@@ -13,6 +13,28 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.
 
+v9.0.4
+======
+
+-  Fixed compatibility with Python 3.8 (but requires source install for the moment).
+-  Fixed Tesseract settings for ``--user-words`` and ``--user-patterns``.
+-  Changed to pikepdf 1.6.5 (for Python 3.8).
+-  Changed to Pillow 6.2.0 (to mitigate a security vulnerability in earlier Pillow).
+-  A debug message now mentions when English is automatically selected if the locale
+   is not English.
+
+v9.0.3
+======
+
+-  Embed an encoded version of the sRGB ICC profile in the intermediate
+   Postscript file (used for PDF/A conversion). Previously we included the
+   filename, which required Postscript to run with file access enabled. For
+   security, Ghostscript 9.28 enables ``-dSAFER`` and as such, no longer
+   permits access to any file by default. This fix is necessary for
+   compatibility with Ghostscript 9.28.
+-  Exclude a test that sometimes times out and fails in continuous integration
+   from the standard test suite.
+
 v9.0.2
 ======
 
@@ -23,7 +45,7 @@ v9.0.2
 -  Fixed an issue that caused inversion of black and white in monochrome images.
    We are not certain but the problem seems to be linked to Leptonica 1.76.0 and
    older.
--  Fixed some cases where the test suite failed or produced unexpected if
+-  Fixed some cases where the test suite failed if
    English or German Tesseract language packs were not installed.
 -  Fixed a runtime error if the Tesseract English language is not installed.
 -  Improved explicit closing of Pillow images after use.

pyproject.toml

@@ -10,7 +10,7 @@ build-backend = "setuptools.build_meta"
 
 [tool.black]
 line-length = 88
-py36 = true
+target-version = ["py36", "py37", "py38"]
 skip-string-normalization = true
 include = '\.pyi?$'
 exclude = '''

requirements/main.txt

@@ -5,8 +5,8 @@ chardet == 3.0.4
 cffi == 1.12.2
 img2pdf == 0.3.3
 pdfminer.six == 20181108
-pikepdf == 1.6.1
+pikepdf == 1.6.5
-Pillow >= 5.0.0, != 5.1.0 ; sys_platform == "darwin"
+Pillow >= 6.2.0
 pycparser == 2.19
 python-xmp-toolkit == 2.0.1
 reportlab == 3.5.13

setup.py

@@ -68,6 +68,7 @@ setup(
     classifiers=[
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
         "Development Status :: 5 - Production/Stable",
         "Environment :: Console",
         "Intended Audience :: End Users/Desktop",
@@ -96,10 +97,8 @@ setup(
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
         'pdfminer.six == 20181108',
-        'pikepdf >= 1.6.0, < 2',
+        'pikepdf >= 1.6.5, < 2',
-        'Pillow >= 4.0.0, != 5.1.0 ; sys_platform == "darwin"',
+        'Pillow >= 6.2.0',
-        # Pillow < 4 has BytesIO/TIFF bug w/img2pdf 0.2.3
-        # block 5.1.0, broken wheels
         'reportlab >= 3.3.0',  # oldest released version with sane image handling
         'tqdm >= 4',
     ],

src/ocrmypdf/_sync.py

@@ -189,7 +189,7 @@ def worker_init(queue):
     root.addHandler(h)
 
 
-def worker_thread_init(queue):
+def worker_thread_init(_queue):
     pass
 
 
@@ -225,15 +225,15 @@ def exec_concurrent(context):
     if max_workers > 1:
         context.log.info("Start processing %d pages concurrent", max_workers)
 
-    # Tesseract 4.0 is multithreaded, and we also run multiple workers. We want to
+    # Tesseract 4.x can be multithreaded, and we also run multiple workers. We want
-    # avoid the situation where we end up trying to run NxN jobs on N CPU cores,
+    # to manage how many threads it uses to avoid creating total threads than cores.
-    # as that gives poor performance. Performance testing shows we're better off
+    # Performance testing shows we're better off
     # parallelizing ocrmypdf and forcing Tesseract to be single threaded, which we
     # get by setting the envvar OMP_THREAD_LIMIT to 1. But if the page count of the
     # input file is small, then we allow Tesseract to use threads, subject to the
-    # constraint: (ocrmypdf workers) * (tesseract threads) <= max_workers and limiting
+    # constraint: (ocrmypdf workers) * (tesseract threads) <= max_workers.
-    # Tesseract to 4 threads.
+    # As of Tesseract 4.1, 3 threads is the most effective on a 4 core/8 thread system.
-    tess_threads = min(4, context.options.jobs // max_workers)
+    tess_threads = min(3, context.options.jobs // max_workers)
     if context.options.tesseract_env is None:
         context.options.tesseract_env = os.environ.copy()
     context.options.tesseract_env.setdefault('OMP_THREAD_LIMIT', str(tess_threads))
@@ -301,7 +301,7 @@ def exec_concurrent(context):
 class NeverRaise(Exception):
     """An exception that is never raised"""
 
-    pass
+    pass  # pylint: disable=unnecessary-pass
 
 
 def run_pipeline(options, api=False):

src/ocrmypdf/_validation.py

@@ -17,6 +17,7 @@
 # along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
 
 
+import locale
 import logging
 import os
 import sys
@@ -47,6 +48,7 @@ from .helpers import is_file_writable, is_iterable_notstr, monotonic, re_symlink
 # External dependencies
 
 HOCR_OK_LANGS = frozenset(['eng', 'deu', 'spa', 'ita', 'por'])
+DEFAULT_LANGUAGE = 'eng'  # Enforce English hegemony
 
 log = logging.getLogger(__name__)
 
@@ -58,7 +60,10 @@ verify_python3_env()
 
 def check_options_languages(options):
     if not options.language:
-        options.language = ['eng']  # Enforce English hegemony
+        options.language = [DEFAULT_LANGUAGE]
+        system_lang = locale.getlocale()[0]
+        if system_lang and not system_lang.startswith('en'):
+            log.debug("No language specified; assuming --language %s", DEFAULT_LANGUAGE)
 
     # Support v2.x "eng+deu" language syntax
     if '+' in options.language[0]:

src/ocrmypdf/api.py

@@ -33,7 +33,7 @@ class TqdmConsole:
 
     def __init__(self, file):
         self.file = file
-        self.py36 = sys.version_info >= (3, 6)
+        self.py36 = sys.version_info[0:2] == (3, 6)
 
     def write(self, msg):
         # When no progress bar is active, tqdm.write() routes to print()
@@ -122,13 +122,23 @@ def create_options(*, input_file, output_file, **kwargs):
     for arg, val in kwargs.items():
         if val is None:
             continue
-        if arg == 'tesseract_env':
+
+        # These arguments with special handling for which we bypass
+        # argparse
+        if arg in {'tesseract_env', 'progress_bar'}:
             deferred.append((arg, val))
             continue
+
         cmd_style_arg = arg.replace('_', '-')
-        cmdline.append(f"--{cmd_style_arg}")
+
+        # Booleans are special: add only if True, omit for False
         if isinstance(val, bool):
+            if val:
+                cmdline.append(f"--{cmd_style_arg}")
             continue
+
+        # We have a parameter
+        cmdline.append(f"--{cmd_style_arg}")
         if isinstance(val, (int, float)):
             cmdline.append(str(val))
         elif isinstance(val, str):

src/ocrmypdf/exec/ghostscript.py

@@ -259,6 +259,7 @@ def generate_pdfa(
                 "-dQUIET",
                 "-dBATCH",
                 "-dNOPAUSE",
+                "-dSAFER",
                 "-dCompatibilityLevel=" + str(pdf_version),
                 "-sDEVICE=pdfwrite",
                 "-dAutoRotatePages=/None",

src/ocrmypdf/exec/tesseract.py

@@ -17,10 +17,8 @@
 
 import os
 import shutil
-import sys
 from collections import namedtuple
 from contextlib import suppress
-from functools import lru_cache
 from os import fspath
 from subprocess import PIPE, STDOUT, CalledProcessError, TimeoutExpired, run
 
@@ -271,6 +269,9 @@ def generate_hocr(
     if user_patterns:
         args_tesseract.extend(['--user-patterns', user_patterns])
 
+    if user_words or user_patterns:
+        args_tesseract.extend(['-c', 'lstm_use_matrix=1'])
+
     # Reminder: test suite tesseract spoofers will break after any changes
     # to the number of order parameters here
     args_tesseract.extend([input_file, prefix, 'hocr', 'txt'] + tessconfig)

src/ocrmypdf/optimize.py

@@ -111,6 +111,11 @@ def extract_image_generic(*, pike, root, log, image, xref, options):
     if pim.bits_per_component == 1:
         return None
 
+    try:
+        pim.indexed  # pikepdf 1.6.3 can't handle [/Indexed [/Array...]]
+    except NotImplementedError:
+        return None
+
     if filtdp[0] == Name.DCTDecode and options.optimize >= 2:
         # This is a simple heuristic derived from some training data, that has
         # about a 70% chance of guessing whether the JPEG is high quality,
@@ -544,11 +549,11 @@ def main(infile, outfile, level, jobs=1):
         """Emulate ocrmypdf's options"""
 
         def __init__(
-            self, input_file, jobs, optimize, jpeg_quality, png_quality, jb2lossy
+            self, input_file, jobs, optimize_, jpeg_quality, png_quality, jb2lossy
         ):
             self.input_file = input_file
             self.jobs = jobs
-            self.optimize = optimize
+            self.optimize = optimize_
             self.jpeg_quality = jpeg_quality
             self.png_quality = png_quality
             self.jbig2_page_group_size = 0
@@ -559,7 +564,7 @@ def main(infile, outfile, level, jobs=1):
     options = OptimizeOptions(
         input_file=infile,
         jobs=jobs,
-        optimize=int(level),
+        optimize_=int(level),
         jpeg_quality=0,  # Use default
         png_quality=0,
         jb2lossy=False,

src/ocrmypdf/pdfa.py

@@ -31,6 +31,7 @@ Ghostscript's handling of pdfmark.
 
 """
 
+import base64
 import os
 from binascii import hexlify
 from pathlib import Path
@@ -48,26 +49,13 @@ SRGB_ICC_PROFILE = pkg_resources.resource_filename('ocrmypdf', ICC_PROFILE_RELPA
 # files, from the Ghostscript documentation. Lines beginning with % are
 # comments. Python substitution variables have a '$' prefix.
 pdfa_def_template = u"""%!
-% Define entries in the document Info dictionary :
+% Define an ICC profile :
 /ICCProfile $icc_profile
 def
 
-% Define an ICC profile :
-
 [/_objdef {icc_PDFA} /type /stream /OBJ pdfmark
-[{icc_PDFA}
+[{icc_PDFA} << /N 3 >> /PUT pdfmark
-<<
+[{icc_PDFA} ICCProfile /PUT pdfmark
-  /N currentpagedevice /ProcessColorModel known {
-    currentpagedevice /ProcessColorModel get dup /DeviceGray eq
-    {pop 1} {
-      /DeviceRGB eq
-      {3}{4} ifelse
-    } ifelse
-  } {
-    (ERROR, unable to determine ProcessColorModel) == flush
-  } ifelse
->> /PUT pdfmark
-[{icc_PDFA} ICCProfile (r) file /PUT pdfmark
 
 % Define the output intent dictionary :
 
@@ -104,16 +92,10 @@ def generate_pdfa_ps(target_filename, icc='sRGB'):
     else:
         raise NotImplementedError("Only supporting sRGB")
 
-    # pdfmark must contain the full path to the ICC profile, and pdfmark must be
+    # Read the ICC profile, encode as ASCII85 and convert to a string which we
-    # also encoded in ASCII. ocrmypdf can be installed anywhere, including to
+    # will insert in the .ps file
-    # paths that have a non-ASCII character in the filename. Ghostscript
+    bytes_icc_profile = Path(icc_profile).read_bytes()
-    # accepts hex-encoded strings and converts them to byte strings, so
+    icc_profile = base64.a85encode(bytes_icc_profile, adobe=True).decode('ascii')
-    # we encode the path with fsencode() and use the hex representation.
-    # UTF-16 not accepted here. (Even though ASCII encodable is the usual case,
-    # do this always to avoid making it a rare conditional.)
-    bytes_icc_profile = os.fsencode(icc_profile)
-    hex_icc_profile = hexlify(bytes_icc_profile)
-    icc_profile = '<' + hex_icc_profile.decode('ascii') + '>'
 
     t = Template(pdfa_def_template)
     ps = t.substitute(icc_profile=icc_profile, icc_identifier=icc)

tests/test_main.py

@@ -217,7 +217,7 @@ def test_skip_ocr(spoof_tesseract_cache, resources, outpdf):
     assert pdfinfo[0].has_text
 
 
-def test_redo_ocr(spoof_tesseract_cache, resources, outpdf):
+def test_redo_ocr(resources, outpdf):
     in_ = resources / 'graph_ocred.pdf'
     before = PdfInfo(in_, detailed_page_analysis=True)
     out = outpdf
@@ -613,6 +613,7 @@ language_model_penalty_non_freq_dict_word 0
     )
 
 
+@pytest.mark.slow  # This test sometimes times out in CI
 @pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_config_notfound(renderer, resources, outdir):
     cfg_file = outdir / 'nofile.cfg'
@@ -629,6 +630,7 @@ def test_tesseract_config_notfound(renderer, resources, outdir):
     assert p.returncode == ExitCode.ok, err
 
 
+@pytest.mark.slow  # This test sometimes times out in CI
 @pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_config_invalid(renderer, resources, outdir):
     cfg_file = outdir / 'test.cfg'

tests/test_validation.py

@@ -15,20 +15,23 @@
 # You should have received a copy of the GNU General Public License
 # along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
 
+import locale
+import logging
 import os
-from unittest.mock import MagicMock, patch
+from unittest.mock import patch
 
 import pytest
 
 import ocrmypdf._validation as vd
 from ocrmypdf.api import create_options
 from ocrmypdf.exceptions import MissingDependencyError, BadArgsError
+from ocrmypdf.pdfinfo import PdfInfo
 
 
 def make_opts(input_file='a.pdf', output_file='b.pdf', language='eng', **kwargs):
-    return create_options(
+    if language is not None:
-        input_file=input_file, output_file=output_file, language=language, **kwargs
+        kwargs['language'] = language
-    )
+    return create_options(input_file=input_file, output_file=output_file, **kwargs)
 
 
 def test_hocr_notlatin_warning(caplog):
@@ -119,3 +122,41 @@ def test_report_file_size(tmp_path, caplog):
     os.truncate(out, 50000)
     vd.report_output_file_size(opts, in_, out)
     assert 'No reason' in caplog.text
+
+
+def test_false_action_store_true():
+    opts = make_opts(keep_temporary_files=True)
+    assert opts.keep_temporary_files == True
+    opts = make_opts(keep_temporary_files=False)
+    assert opts.keep_temporary_files == False
+
+
+@pytest.mark.parametrize('progress_bar', [True, False])
+def test_no_progress_bar(progress_bar, resources):
+    opts = make_opts(progress_bar=progress_bar, input_file=(resources / 'trivial.pdf'))
+    with patch('ocrmypdf.pdfinfo.info.tqdm', autospec=True) as tqdmpatch:
+        vd.check_options(opts)
+        pdfinfo = PdfInfo(opts.input_file, progbar=opts.progress_bar)
+        assert pdfinfo is not None
+        assert tqdmpatch.called
+        _args, kwargs = tqdmpatch.call_args
+        assert kwargs['disable'] != progress_bar
+
+
+def test_language_warning(caplog):
+    opts = make_opts(language=None)
+    caplog.set_level(logging.DEBUG)
+    with patch(
+        'ocrmypdf._validation.locale.getlocale', return_value=('en_US', 'UTF-8')
+    ):
+        vd.check_options_languages(opts)
+        assert opts.language == ['eng']
+        assert '' in caplog.text
+
+    opts = make_opts(language=None)
+    with patch(
+        'ocrmypdf._validation.locale.getlocale', return_value=('fr_FR', 'UTF-8')
+    ):
+        vd.check_options_languages(opts)
+        assert opts.language == ['eng']
+        assert 'assuming --language' in caplog.text