v5.3 release notes

Omitted French because the rare 'oe' and 'ÿ' glyphs are not in Latin-1.
Basically steer people away from HOCR renderer but avoid a potential
disruptive behavior change.

.travis.yml

@@ -84,11 +84,10 @@ deploy:
   script: /usr/bin/true
   on:
     branch: master
-    tags: false
     condition: $TRAVIS_OS_NAME == "osx"
 
 after_deploy: |
-  if [[ "$TRAVIS_OS_NAME" == "osx" && "$TRAVIS_BRANCH" == "master" ]]; then
+  if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
     bash .travis/osx_brew.sh
   elif [[ "$TRAVIS_PYTHON_VERSION" == "3.6" && "$TRAVIS_OS_NAME" == "linux" ]]; then
     curl -H "Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/jbarlow83/ocrmypdf/trigger/$DOCKERHUB_OCRMYPDF_TOKEN/

.travis/autobrew.py

@@ -15,6 +15,7 @@ class Ocrmypdf < Formula
   ${ocrmypdf_sha256}
 
   depends_on :python3
+  depends_on :x11  # Pillow needs XQuartz
   depends_on "pkg-config" => :build
   depends_on "libffi"
   depends_on "tesseract"

docs/advanced.rst

@@ -33,20 +33,27 @@ If you want to adjust the amount of time spent on OCR, change ``--tesseract-time
 Overriding default tesseract
 """"""""""""""""""""""""""""
 
-OCRmyPDF checks the environment variable ``OCRMYPDF_TESSERACT`` for the full path *to the tesseract executable* first. 
+OCRmyPDF checks the environment variable ``OCRMYPDF_TESSERACT`` for the full path *to the tesseract binary* first. 
 
-For example, if you are testing tesseract 4.00 and don't wish to disturb your tesseract 3.04 installation, you can launch OCRmyPDF as follows:
+.. envvar:: OCRMYPDF_TESSERACT
+
+	Specify the location of the Tesseract binary.
+
+.. envvar:: TESSDATA_PREFIX
+
+	A Tesseract environment variable that overrides the path to Tesseract's data files.
+
+For example, if you are testing tesseract 4.00 and don't wish to disturb use an existing tesseract 3.04 installation, you can launch OCRmyPDF as follows:
 
 .. code-block:: bash
 
 	env \
 		OCRMYPDF_TESSERACT=/home/user/src/tesseract4/api/tesseract \
 		TESSDATA_PREFIX=/home/user/src/tesseract4 \
-		ocrmypdf --pdf-renderer tess4 --tesseract-oem 2 input.pdf output.pdf
+		ocrmypdf --tesseract-oem 2 input.pdf output.pdf
+
+In this example ``TESSDATA_PREFIX`` directs tesseract 4.0 to use LSTM training data. ``--tesseract-oem 1`` requests tesseract 4.0's new LSTM engine. (Tesseract 4.0 only.)
 
-* ``TESSDATA_PREFIX`` directs tesseract 4.0 to use LSTM training data. This is a tesseract environment variable.
-* ``--pdf-renderer tess4`` takes advantage of new tesseract 4.0 PDF renderer in OCRmyPDF. (Tesseract 4.0 only.)
-* ``--tesseract-oem 1`` requests tesseract 4.0's new LSTM engine. (Tesseract 4.0 only.)
 
 Overriding other support programs
 """""""""""""""""""""""""""""""""
@@ -57,7 +64,20 @@ In addition to tesseract, OCRmyPDF uses the following external binaries:
 * ``unpaper``
 * ``qpdf``
 
-In each case OCRmyPDF will check the environment variable ``OCRMYPDF_{program}`` before asking the system to find ``{program}`` on the PATH. For example, you could redirect OCRmyPDF to ``OCRMYPDF_GS`` to override Ghostscript.
+In each case OCRmyPDF will check the environment variable ``OCRMYPDF_{program}`` before asking the system to find ``{program}`` on the PATH. For example, you could redirect OCRmyPDF to ``OCRMYPDF_GS`` to override Ghostscript. The full list is below:
+
+.. envvar:: OCRMYPDF_GS
+
+	Specify the location of the Ghostscript binary.
+
+.. envvar:: OCRMYPDF_UNPAPER
+
+	Specify the location of the unpaper binary.
+
+.. envvar:: OCRMYPDF_QPDF
+
+	Specify the location of the qpdf binary.
+
 
 Changing tesseract configuration variables
 """"""""""""""""""""""""""""""""""""""""""
@@ -95,33 +115,29 @@ rendering
   Creating a new PDF from other data (such as an existing PDF).
 
 
-OCRmyPDF has three PDF renderers: ``hocr``, ``tesseract`` and ``tess4``. The renderer may be selected using ``--pdf-renderer``. The default is ``auto`` which lets OCRmyPDF select the renderer to use. Currently, ``auto`` always selects ``hocr``. 
+OCRmyPDF has three PDF renderers: ``sandwich``, ``hocr``, ``tesseract``. The renderer may be selected using ``--pdf-renderer``. The default is ``auto`` which lets OCRmyPDF select the renderer to use. Currently, ``auto`` selects ``sandwich`` for Tesseract 3.05.01, and newer, ``hocr`` for older versions of Tesseract.
+
+The ``sandwich`` renderer
+"""""""""""""""""""""""""
+
+The ``sandwich`` renderer uses Tesseract's new text-only PDF feature, which produces a PDF page that lays out the OCR in invisible text. This page is then "sandwiched" onto the original PDF page, allowing lossless application of OCR even to PDF pages that contain other vector objects.
+
+When image preprocessing features like ``--deskew`` are used, the original PDF will be rendered as a full page and the OCR layer will be placed on top.
+
+This renderer requires Tesseract 3.05.01 or newer.
 
 The ``hocr`` renderer
 """""""""""""""""""""
 
-The ``hocr`` renderer is the default because it works in most cases. In this mode the whole PDF is rasterized, the raster image is run through OCR to generate a .hocr file, which is an HTML-like file that specifies the location of all identified words.
-
-The .hocr file is then rendered as a PDF and merged with the image layer.
-
-The image layer is copied from the original PDF page if possible, avoiding potentially lossy transcoding or loss of other PDF information. If preprocessing is specified, then the image layer is a new PDF.
-
-This is the only option for tesseract 3.02 and older.
+The ``hocr`` renderer works with older versions of Tesseract. The image layer is copied from the original PDF page if possible, avoiding potentially lossy transcoding or loss of other PDF information. If preprocessing is specified, then the image layer is a new PDF.
 
+This works in all versions of Tesseract.
 
 The ``tesseract`` renderer
 """"""""""""""""""""""""""
 
-The tesseract renderer uses tesseract's capability to produce a PDF directly. In version 3, tesseract automatically combined the image layer and text, meaning that this mode *always* transcodes and loses potentially loses image quality and other PDF information.
+The ``tesseract`` renderer creates a PDF with the image and text layers precomposed, meaning that it always transcodes, loses image quality and rasterizes and vector objects. It does a better job on non-Latin text and document structure than ``hocr``.
 
-It does a much better job on non-Latin text.
+If a PDF created with this renderer using Tesseract versions older than 3.05.00 is then passed through Ghostscript's pdfwrite feature, the OCR text *may* be corrupted. The ``--output-type=pdfa`` argument will produce a warning in this situation.
 
-In a future release this will become the "tess3" renderer and ultimately will be dropped.
-
-
-The ``tess4`` renderer
-""""""""""""""""""""""
-
-The tess4 renderer uses tesseract 4.00 alpha's text-only PDF feature added in January 2017. This combines the advantages of the tesseract and hocr renderers, transcoding the image layer only if required by preprocessing options.
-
-Ghostscript PDF/A still sometimes inserts spaces between words when the tess4 renderer is used, affecting search quality.  ``--output-pdf pdf`` may be used to avoid this issue.
+*This renderer is deprecated and will be removed whenever support for older versions of Tesseract is dropped.*

docs/cookbook.rst

@@ -72,19 +72,11 @@ Language packs must be installed for all languages specified. See :ref:`Installi
 Produce PDF and text file containing OCR text
 """""""""""""""""""""""""""""""""""""""""""""
 
-This produces a file named "output.pdf" and a companion text file named "output.txt". The ``pdftotext`` program from 
-`Poppler <https://poppler.freedesktop.org/>`_ is used to extract text from the finished PDF.
-
+This produces a file named "output.pdf" and a companion text file named "output.txt".
 
 .. code-block:: bash
 
-	ocrmypdf input.pdf - | tee output.pdf | pdftotext - output.txt
-
-.. note::
-
-	To get pdftotext, Debian/Ubuntu users may ``apt-get install poppler-utils`` 
-	and macOS users may ``brew install poppler`` respectively.
-
+	ocrmypdf --sidecar output.txt input.pdf output.pdf
 
 OCR images, not PDFs
 --------------------

docs/release_notes.rst

@@ -1,7 +1,27 @@
 Release notes
 =============
 
-OCRmyPDF uses `semantic versioning <http://semver.org/>`_.
+OCRmyPDF uses `semantic versioning <http://semver.org/>`_ for its command line interface.
+
+The OCRmyPDF package itself does not contain a public API, although it is fairly stable and breaking changes are usually timed with a major release. A future release will clearly define the stable public API.
+
+
+v5.3
+----
+
+-   Added ``--user-words`` and ``--user-patterns`` arguments which are forwarded to Tesseract OCR as words and regular expressions respective to use to guide OCR. Supplying a list of subject-domain words should assist Tesseract with resolving words. (#165)
+-   Using a non Latin-1 language with the "hocr" renderer now warns about possible OCR quality and recommends workarounds (#176)
+-   Output file path added to error message when that location is not writable (#175)
+-   Otherwise valid PDFs with leading whitespace at the beginning of the file are now accepted
+
+
+v5.2
+----
+
+-   When using Tesseract 3.05.01 or newer, OCRmyPDF will select the "sandwich" PDF renderer by default, unless another PDF renderer is specified with the ``--pdf-renderer`` argument. The previous behavior was to select ``--pdf-renderer=hocr``.
+-   The "tesseract" PDF renderer is now deprecated, since it can cause problems with Ghostscript on Tesseract 3.05.00
+-   The "tess4" PDF renderer has been renamed to "sandwich". "tess4" is now a deprecated alias for "sandwich".
+
 
 v5.1
 ----
@@ -12,7 +32,7 @@ v5.1
 v5.0.1
 ------
 
--   Fixed issue #169, exception due to failure to create sidecar text files on some versions of Tesseract 3.04, including the jbarlow83/ocrmypdf image
+-   Fixed issue #169, exception due to failure to create sidecar text files on some versions of Tesseract 3.04, including the jbarlow83/ocrmypdf Docker image
 
 
 v5.0

ocrmypdf/__main__.py

@@ -41,6 +41,9 @@ warnings.simplefilter('ignore', pypdf.utils.PdfReadWarning)
 
 MINIMUM_TESS_VERSION = '3.04'
 
+HOCR_OK_LANGS = frozenset([
+    'eng', 'deu', 'spa', 'ita', 'por'
+])
 
 def complain(message):
     print(*textwrap.wrap(message), file=sys.stderr)
@@ -250,15 +253,18 @@ advanced.add_argument(
          "3 - default.")
     )
 advanced.add_argument(
-    '--pdf-renderer', choices=['auto', 'tesseract', 'hocr', 'tess4'], default='auto',
+    '--pdf-renderer',
+    choices=['auto', 'tesseract', 'hocr', 'tess4', 'sandwich'], default='auto',
     help="Choose OCR PDF renderer - the default option is to let OCRmyPDF "
-         "choose.  The 'tesseract' PDF renderer is more accurate and does a "
-         "better job and document structure such as recognizing columns. It "
-         "also does a better job on non-Latin languages. However, it does "
-         "not work as well when older versions of Tesseract or Ghostscript "
-         "are installed, and some combinations of arguments to do not work "
-         "with --pdf-renderer tesseract.  The 'tess4' PDF renderer is similar "
-         "to 'tesseract', requires tesseract 4, and gives superior results.")
+         "choose."
+         "auto - let OCRmyPDF choose; "
+         "sandwich - default renderer for Tesseract 3.05.01 and newer; "
+         "hocr - default renderer for older versions of Tesseract; "
+         "tesseract - gives better results for non-Latin languages and "
+         "Tesseract older than 3.05.01 but has problems with some versions "
+         " of Ghostscript; deprecated"
+         "tess4 - deprecated alias for 'sandwich'"
+    )
 advanced.add_argument(
     '--tesseract-timeout', default=180.0, type=float, metavar='SECONDS',
     help='Give up on OCR after the timeout, but copy the preprocessed page '
@@ -278,6 +284,15 @@ advanced.add_argument(
          "are applied to all pages, including those for which OCR was "
          "skipped.  Not supported for --output-type=pdf ; that setting "
          "preserves the original compression of all images.")
+advanced.add_argument(
+    '--user-words', metavar='FILE',
+    help="Specify the location of the Tesseract user words file. This is a "
+         "list of words Tesseract should consider while performing OCR in "
+         "addition to its standard language dictionaries. This can improve "
+         "OCR quality especially for specialized and technical documents.")
+advanced.add_argument(
+    '--user-patterns', metavar='FILE',
+    help="Specify the location of the Tesseract user patterns file.")
 
 debugging = parser.add_argument_group(
     "Debugging",
@@ -312,28 +327,31 @@ def check_options_languages(options, _log):
 
 def check_options_output(options, log):
     if options.pdf_renderer == 'auto':
-        options.pdf_renderer = 'hocr'
+        if tesseract.has_textonly_pdf():
+            options.pdf_renderer = 'sandwich'
+        else:
+            options.pdf_renderer = 'hocr'
 
-    if options.pdf_renderer in ('tesseract', 'tess4'):
-        if tesseract.version() < '3.05':
-            log.warning(
-                "The setting --pdf-renderer=tesseract is not recommend for "
-                " use with tesseract versions less than 3.05, because it "
-                " produces OCR text that is incompatible with Ghostscript and "
-                " some other software.")
-        elif tesseract.version() == '4.00.00alpha':
-            log.warning(
-                "The setting --pdf-renderer={tesseract,tess4} is not"
-                " recommended for builds of tesseract 4.00.00alpha older than"
-                " February 2017. Make sure you are using a recent build.")
+    if options.pdf_renderer == 'tess4':
+        log.warning("The 'tess4' PDF renderer has been renamed to 'sandwich'. "
+                    "Please use --pdf-renderer=sandwich.")
+        options.pdf_renderer = 'sandwich'
 
-    if options.debug_rendering and options.pdf_renderer == 'tesseract':
+    if options.pdf_renderer == 'tesseract':
+        log.warning("The 'tesseract' PDF renderer is deprecated.")
+        if tesseract.version() < '3.05' and options.output_type == 'pdfa':
+            log.warning(
+                "For best results use --pdf-renderer=tesseract "
+                "--output-type=pdf to disable PDF/A generation via "
+                "Ghostscript, which is known to corrupt the OCR text of "
+                "some PDFs produced your version of Tesseract.")
+
+    if options.debug_rendering and options.pdf_renderer != 'hocr':
         log.info(
-            "Ignoring --debug-rendering because it is not supported with"
-            "--pdf-renderer=tesseract.")
+            "Ignoring --debug-rendering because it requires --pdf-renderer=hocr")
 
     lossless_reconstruction = False
-    if options.pdf_renderer in ('hocr', 'tess4'):
+    if options.pdf_renderer in ('hocr', 'sandwich'):
         if not any((options.deskew, options.clean_final, options.force_ocr,
                    options.remove_background)):
             lossless_reconstruction = True
@@ -380,12 +398,20 @@ def check_options_ocr_behavior(options, log):
     #     raise argparse.ArgumentError(
     #         "Error: --redo-ocr and other OCR options are incompatible.")
 
-    if set(options.language) & {'chi_sim', 'chi_tra'} and \
-            (options.pdf_renderer == 'hocr' or options.output_type == 'pdfa'):
-        log.warning(
-            "Your settings are known to cause problems with OCR of Chinese text. "
-            "Try adding these arguments: "
-            "    ocrmypdf --pdf-renderer tesseract --output-type pdf")
+    if options.pdf_renderer == 'hocr' and \
+            not set(options.language).issubset(HOCR_OK_LANGS):
+        msg = (
+            "The 'hocr' PDF renderer is known to cause problems with one "
+            "or more of the languages in your document. ")
+
+        if tesseract.has_textonly_pdf():
+            msg += (
+                "Use --pdf-renderer auto (the default) to avoid this issue.")
+        else:
+            msg += (
+                "Use --pdf-renderer tesseract --output-type pdf to avoid "
+                "this issue")
+        log.warning(msg)
 
 
 def check_options_advanced(options, log):
@@ -660,8 +686,9 @@ def run_pipeline():
                     file."""))
                 return ExitCode.bad_args
         elif not is_file_writable(options.output_file):
-            _log.error(textwrap.dedent("""\
-                Output file location is not writable."""))
+            _log.error(
+                "Output file location (" + options.output_file + ") " +
+                "is not writable.")
             return ExitCode.file_access_error
 
         manager = JobContextManager()

ocrmypdf/exec/ghostscript.py

@@ -10,6 +10,7 @@ import sys
 from . import get_program
 from ..exceptions import SubprocessOutputError
 from PIL import Image
+from ..helpers import fspath
 
 
 @lru_cache(maxsize=1)
@@ -43,8 +44,8 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
     (xres, yres) even if those numbers are noninteger. The image's DPI will
      be overridden with the values in page_dpi.
     
-    :param input_file: 
-    :param output_file: 
+    :param input_file: pathlike
+    :param output_file: pathlike
     :param xres: resolution at which to rasterize page
     :param yres: 
     :param raster_device: 
@@ -69,7 +70,7 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
             '-dLastPage=%i' % pageno,
             '-o', tmp.name,
             '-r{0}x{1}'.format(str(int_res[0]), str(int_res[1])),
-            input_file
+            fspath(input_file)
         ]
 
         p = run(args_gs, stdout=PIPE, stderr=STDOUT,
@@ -95,9 +96,10 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
                 log.debug(
                     "Ghostscript: resize output image {} -> {}".format(
                         im.size, expected_size))
-                im.resize(expected_size).save(output_file, dpi=page_dpi)
+                im.resize(expected_size).save(
+                    fspath(output_file), dpi=page_dpi)
             else:
-                copy(tmp.name, output_file)
+                copy(tmp.name, fspath(output_file))
 
 
 def generate_pdfa(pdf_pages, output_file, compression, log,

ocrmypdf/exec/tesseract.py

@@ -63,26 +63,28 @@ def v4():
     return (version() >= '4')
 
 
+@lru_cache(maxsize=1)
 def has_textonly_pdf():
-    if version() == '4.00.00alpha':
-        # textonly_pdf added during the 4.00.00alpha cycle, so we must test
-        # more carefully to see if it is present
-        args_tess = [
-            get_program('tesseract'),
-            '--print-parameters'
-        ]
-        try:
-            params = check_output(
-                    args_tess, close_fds=True, universal_newlines=True,
-                    stderr=STDOUT)
-        except CalledProcessError as e:
-            print("Could not --print-parameters from tesseract",
-                  file=sys.stderr)
-            raise MissingDependencyError from e
-        if 'textonly_pdf' in params:
-            return True
-    else:
-        return v4()
+    """Does Tesseract have textonly_pdf capability?
+    
+    Available in 3.05.01, and v4.00.00alpha since January 2017. Best to 
+    parse the parameter list
+    """
+    args_tess = [
+        get_program('tesseract'),
+        '--print-parameters'
+    ]
+    try:
+        params = check_output(
+                args_tess, close_fds=True, universal_newlines=True,
+                stderr=STDOUT)
+    except CalledProcessError as e:
+        print("Could not --print-parameters from tesseract",
+              file=sys.stderr)
+        raise MissingDependencyError from e
+    if 'textonly_pdf' in params:
+        return True
+    return False
 
 
 def psm():
@@ -211,7 +213,8 @@ def _generate_null_hocr(output_hocr, output_sidecar, image):
 
 def generate_hocr(input_file, output_files, language: list, engine_mode,
                   tessconfig: list,
-                  timeout: float, pagesegmode: int, log):
+                  timeout: float, pagesegmode: int, user_words, user_patterns,
+                  log):
 
     output_hocr = next(o for o in output_files if o.endswith('.hocr'))
     output_sidecar = next(o for o in output_files if o.endswith('.txt'))
@@ -222,10 +225,17 @@ def generate_hocr(input_file, output_files, language: list, engine_mode,
     if pagesegmode is not None:
         args_tesseract.extend([psm(), str(pagesegmode)])
 
+    if user_words:
+        args_tesseract.extend(['--user-words', user_words])
+
+    if user_patterns:
+        args_tesseract.extend(['--user-patterns', user_patterns])
+
     # Reminder: test suite tesseract spoofers will break after any changes
     # to the number of order parameters here
     # Tesseract 3.04 requires the order here to be "hocr txt" and will fail
     # on "txt hocr"
+
     args_tesseract.extend([
         input_file,
         prefix,
@@ -283,7 +293,8 @@ def use_skip_page(text_only, skip_pdf, output_pdf, output_text):
 
 def generate_pdf(*, input_image, skip_pdf, output_pdf, output_text,
                  language: list, engine_mode, text_only: bool,
-                 tessconfig: list, timeout: float, pagesegmode: int, log):
+                 tessconfig: list, timeout: float, pagesegmode: int,
+                 user_words, user_patterns, log):
     '''Use Tesseract to render a PDF.
 
     input_image -- image to analyze
@@ -306,10 +317,17 @@ def generate_pdf(*, input_image, skip_pdf, output_pdf, output_text,
     if text_only:
         args_tesseract.extend(['-c', 'textonly_pdf=1'])
 
+    if user_words:
+        args_tesseract.extend(['--user-words', user_words])
+
+    if user_patterns:
+        args_tesseract.extend(['--user-patterns', user_patterns])
+
     prefix = os.path.splitext(output_pdf)[0]  # Tesseract appends suffixes
 
-    # Reminder: test suite tesseract spoofers will break after any changes
+    # Reminder: test suite tesseract spoofers might break after any changes
     # to the number of order parameters here
+
     args_tesseract.extend([
         input_image,
         prefix,

ocrmypdf/helpers.py

@@ -79,10 +79,43 @@ def is_file_writable(test_file):
         return True
 
 
-@contextmanager
-def universal_open(p, *args, **kwargs):
-    "Work around Python 3.5's inability to open(pathlib.Path())"
-    try:
-        yield p.open(*args, **kwargs)
-    except AttributeError:
-        yield open(p, *args, **kwargs)
+if sys.version_info[0:2] <= (3, 5):
+    def universal_open(p, *args, **kwargs):
+        "Work around Python 3.5's inability to open(pathlib.Path())"
+        try:
+            return p.open(*args, **kwargs)
+        except AttributeError:
+            return open(p, *args, **kwargs)
+
+
+    def fspath(path):
+        import pathlib
+        '''https://www.python.org/dev/peps/pep-0519/#os'''
+        if isinstance(path, (str, bytes)):
+            return path
+
+        # Work from the object's type to match method resolution of other magic
+        # methods.
+        path_type = type(path)
+        try:
+            path = path_type.__fspath__(path)
+        except AttributeError:
+            # Added for Python 3.5 support.
+            if isinstance(path, pathlib.Path):
+                return str(path)
+            elif hasattr(path_type, '__fspath__'):
+                raise
+        else:
+            if isinstance(path, (str, bytes)):
+                return path
+            else:
+                raise TypeError("expected __fspath__() to return str or bytes, "
+                                "not " + type(path).__name__)
+
+        raise TypeError(
+            "expected str, bytes, pathlib.Path or os.PathLike object, not "
+            + path_type.__name__)
+
+else:
+    universal_open = open
+    fspath = os.fspath

ocrmypdf/leptonica.py

@@ -14,6 +14,7 @@ from ctypes.util import find_library
 from .lib._leptonica import ffi
 from functools import lru_cache
 from enum import Enum
+from .helpers import fspath
 
 lept = ffi.dlopen(find_library('lept'))
 
@@ -208,26 +209,27 @@ class Pix:
             return 'P'
 
     @classmethod
-    def read(cls, filename):
+    def read(cls, path):
         """Load an image file into a PIX object.
 
         Leptonica can load TIFF, PNM (PBM, PGM, PPM), PNG, and JPEG.  If
         loading fails then the object will wrap a C null pointer.
         """
+        filename = fspath(path)
         with LeptonicaErrorTrap():
-            return cls(lept.pixRead(
-                filename.encode(sys.getfilesystemencoding())))
+            return cls(lept.pixRead(os.fsencode(filename)))
 
     def write_implied_format(
-            self, filename, jpeg_quality=0, jpeg_progressive=0):
+            self, path, jpeg_quality=0, jpeg_progressive=0):
         """Write pix to the filename, with the extension indicating format.
 
         jpeg_quality -- quality (iff JPEG; 1 - 100, 0 for default)
         jpeg_progressive -- (iff JPEG; 0 for baseline seq., 1 for progressive)
         """
+        filename = fspath(path)
         with LeptonicaErrorTrap():
             lept.pixWriteImpliedFormat(
-                filename.encode(sys.getfilesystemencoding()),
+                os.fsencode(filename),
                 self._pix, jpeg_quality, jpeg_progressive)
 
     def topil(self):

ocrmypdf/pipeline.py

@@ -165,8 +165,8 @@ def triage(
     options = context.get_options()
     try:
         with open(input_file, 'rb') as f:
-            signature = f.read(4)
-            if signature == b'%PDF':
+            signature = f.read(1024)
+            if b'%PDF' in signature:
                 if options.image_dpi:
                     log.warning("Argument --image-dpi ignored because the "
                                 "input file is a PDF, not an image.")
@@ -202,7 +202,7 @@ def repair_pdf(
 
 
 def get_pageinfo(input_file, context):
-    pageno = int(os.path.basename(input_file)[0:6]) - 1
+    pageno = page_number(input_file) - 1
     pageinfo = context.get_pdfinfo()[pageno]
     return pageinfo
 
@@ -406,7 +406,7 @@ def orient_page(
         with open(output_file, 'wb') as out:
             writer.write(out)
 
-        pageno = int(os.path.basename(page_pdf)[0:6]) - 1
+        pageno = page_number(page_pdf) - 1
         pdfinfo = context.get_pdfinfo()
         pdfinfo[pageno].rotation = orient_conf.angle
         context.set_pdfinfo(pdfinfo)
@@ -525,6 +525,8 @@ def ocr_tesseract_hocr(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log
         )
 
@@ -595,7 +597,7 @@ def select_image_layer(
         # as it accurately describes the image. It would be possible to
         # resample the image at this stage back to non-square DPI to more
         # closely resemble the input, except that the hocr renderer does not
-        # understand non-square DPI. The tess4 renderer would be fine.
+        # understand non-square DPI. The sandwich renderer would be fine.
         dpi = get_page_square_dpi(pageinfo, options)
         layout_fun = img2pdf.get_fixed_dpi_layout_fun((dpi, dpi))
 
@@ -755,6 +757,8 @@ def ocr_tesseract_and_render_pdf(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log)
 
 
@@ -783,6 +787,8 @@ def ocr_tesseract_textonly_pdf(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log)
 
 
@@ -812,9 +818,16 @@ def get_pdfmark(base_pdf, options):
     if options.subject:
         pdfmark['/Subject'] = options.subject
 
-    pdfmark['/Creator'] = '{0} {1} / Tesseract OCR{2} {3}'.format(
+    if options.pdf_renderer == 'tesseract':
+        renderer_tag = 'OCR+PDF'
+    elif options.pdf_renderer == 'sandwich':
+        renderer_tag = 'OCR-PDF'
+    else:
+        renderer_tag = 'OCR'
+
+    pdfmark['/Creator'] = '{0} {1} / Tesseract {2} {3}'.format(
             PROGRAM_NAME, VERSION,
-            '+PDF' if options.pdf_renderer == 'tesseract' else '',
+            renderer_tag,
             tesseract.version())
     return pdfmark
 
@@ -856,7 +869,7 @@ def merge_pages_ghostscript(
         right that way.'''
         if s.endswith('.ps'):
             return 99999999
-        key = int(os.path.basename(s)[0:6]) * 10
+        key = page_number(s) * 10
         if 'debug' in os.path.basename(s):
             key += 1
         return key
@@ -891,7 +904,7 @@ def merge_pages_qpdf(
     def input_file_order(s):
         '''Sort order: All rendered pages followed
         by their debug page.'''
-        key = int(os.path.basename(s)[0:6]) * 10
+        key = page_number(s) * 10
         if 'debug' in os.path.basename(s):
             key += 1
         return key
@@ -1083,7 +1096,7 @@ def build_pipeline(options, work_folder, log, context):
     task_select_image_layer.graphviz(
         fillcolor='"#00cc66"', shape='diamond')
     task_select_image_layer.active_if(
-        options.pdf_renderer == 'hocr' or options.pdf_renderer == 'tess4')
+        options.pdf_renderer == 'hocr' or options.pdf_renderer == 'sandwich')
 
     task_render_hocr_page = main_pipeline.transform(
         task_func=render_hocr_page,
@@ -1113,7 +1126,7 @@ def build_pipeline(options, work_folder, log, context):
                 os.path.join(work_folder, r'\1.text.txt')],
         extras=[log, context])
     task_ocr_tesseract_textonly_pdf.graphviz(fillcolor='"#ff69b4"')
-    task_ocr_tesseract_textonly_pdf.active_if(options.pdf_renderer == 'tess4')
+    task_ocr_tesseract_textonly_pdf.active_if(options.pdf_renderer == 'sandwich')
     if tesseract.v4():
         task_ocr_tesseract_textonly_pdf.jobs_limit(2)
 
@@ -1126,7 +1139,7 @@ def build_pipeline(options, work_folder, log, context):
         output=os.path.join(work_folder, r'\1.rendered.pdf'),
         extras=[log, context])
     task_combine_layers.graphviz(fillcolor='"#00cc66"')
-    task_combine_layers.active_if(options.pdf_renderer == 'hocr' or options.pdf_renderer == 'tess4')
+    task_combine_layers.active_if(options.pdf_renderer == 'hocr' or options.pdf_renderer == 'sandwich')
 
     # Tesseract OCR+PDF
     task_ocr_tesseract_and_render_pdf = main_pipeline.collate(

tests/conftest.py

@@ -129,7 +129,8 @@ def run_ocrmypdf(input_file, output_file, *args, env=None):
     if env is None:
         env = os.environ
 
-    p_args = OCRMYPDF + list(args) + [str(input_file), str(output_file)]
+    p_args = OCRMYPDF + [str(arg) for arg in args] + \
+             [str(input_file), str(output_file)]
     p = Popen(
         p_args, close_fds=True, stdout=PIPE, stderr=PIPE,
         universal_newlines=True, env=env)

tests/spoof/tesseract_big_image_error.py

@@ -22,6 +22,9 @@ def main():
     elif sys.argv[1] == '--list-langs':
         print('List of available languages (1):\neng', file=sys.stderr)
         sys.exit(0)
+    elif sys.argv[1] == '--print-parameters':
+        print('A parameter list would go here', file=sys.stderr)
+        sys.exit(0)
     elif sys.argv[-2] == 'hocr':
         print("Image too large: (33830, 14959)\n"
               "Error during processing.", file=sys.stderr)

tests/spoof/tesseract_cache.py

@@ -85,6 +85,12 @@ def main():
     except ValueError:
         m.update(b'default-lang')
 
+    try:
+        textonly = sys.argv[sys.argv.index('-c') + 1]
+        m.update(textonly.encode())
+    except ValueError:
+        m.update(b'textonly_pdf=0')
+
     psm_arg = ''
     if '--psm' in sys.argv:
         psm_arg = '--psm'

tests/spoof/tesseract_crash.py

@@ -29,6 +29,9 @@ def main():
     elif sys.argv[1] == '--list-langs':
         print('List of available languages (1):\neng', file=sys.stderr)
         sys.exit(0)
+    elif sys.argv[1] == '--print-parameters':
+        print('List of parameters would go here', file=sys.stderr)
+        sys.exit(0)
     elif sys.argv[-2] == 'hocr':
         print("KABOOM! Tesseract failed for some reason", file=sys.stderr)
         sys.exit(128 + signal.SIGSEGV)

tests/spoof/tesseract_noop.py

@@ -2,6 +2,7 @@
 # © 2016 James R. Barlow: github.com/jbarlow83
 import sys
 import img2pdf
+import PyPDF2 as pypdf
 from PIL import Image
 
 
@@ -17,7 +18,7 @@ In orientation check mode, report the orientation is upright.
 """
 
 
-VERSION_STRING = '''tesseract 3.04.00
+VERSION_STRING = '''tesseract 3.05.01
  leptonica-1.72
   libjpeg 8d : libpng 1.6.19 : libtiff 4.0.6 : zlib 1.2.5
 SPOOFED
@@ -53,6 +54,10 @@ def main():
     elif sys.argv[1] == '--list-langs':
         print('List of available languages (1):\neng', file=sys.stderr)
         sys.exit(0)
+    elif sys.argv[1] == '--print-parameters':
+        print("Some parameters", file=sys.stderr)
+        print("textonly_pdf\t1\tSome help text")
+        sys.exit(0)
     elif sys.argv[-2] == 'hocr':
         inputf = sys.argv[-4]
         output = sys.argv[-3]
@@ -63,13 +68,27 @@ def main():
         with open(output + '.txt', 'w') as f:
             f.write('')
     elif sys.argv[-2] == 'pdf':
-        inputf = sys.argv[-4]
-        output = sys.argv[-3]
-        pdf_bytes = img2pdf.convert([inputf], dpi=300)
-        with open(output + '.pdf', 'wb') as f:
-            f.write(pdf_bytes)
-        with open(output + '.txt', 'w') as f:
-            f.write('')
+        if 'textonly_pdf=1' in sys.argv:
+            inputf = sys.argv[-4]
+            output = sys.argv[-3]
+            with Image.open(inputf) as im:
+                dpi = im.info['dpi']
+                imsize = im.size[0] * dpi[0] / 72, im.size[1] * dpi[1] / 72
+
+            pdf_out = pypdf.PdfFileWriter()
+            pdf_out.addBlankPage(imsize[0], imsize[1])
+            with open(output + '.pdf', 'wb') as f:
+                pdf_out.write(f)
+            with open(output + '.txt', 'w') as f:
+                f.write('')
+        else:
+            inputf = sys.argv[-4]
+            output = sys.argv[-3]
+            pdf_bytes = img2pdf.convert([inputf], dpi=300)
+            with open(output + '.pdf', 'wb') as f:
+                f.write(pdf_bytes)
+            with open(output + '.txt', 'w') as f:
+                f.write('')
     elif sys.argv[-1] == 'stdout':
         inputf = sys.argv[-2]
         print("""Orientation: 0

tests/test_main.py

@@ -10,7 +10,7 @@ import PyPDF2 as pypdf
 from ocrmypdf.exceptions import ExitCode
 from ocrmypdf import leptonica
 from ocrmypdf.pdfa import file_claims_pdfa
-from ocrmypdf.exec import ghostscript
+from ocrmypdf.exec import ghostscript, tesseract
 import logging
 from math import isclose
 
@@ -20,6 +20,11 @@ run_ocrmypdf = pytest.helpers.run_ocrmypdf
 spoof = pytest.helpers.spoof
 
 
+RENDERERS = ['hocr', 'tesseract']
+if tesseract.has_textonly_pdf():
+    RENDERERS.append('sandwich')
+
+
 @pytest.fixture
 def spoof_tesseract_crash():
     return spoof(tesseract='tesseract_crash.py')
@@ -72,8 +77,8 @@ def test_deskew(spoof_tesseract_noop, resources, outdir):
     deskewed_png = outdir / 'deskewed.png'
 
     ghostscript.rasterize_pdf(
-        str(deskewed_pdf),
-        str(deskewed_png),
+        deskewed_pdf,
+        deskewed_png,
         xres=150,
         yres=150,
         raster_device='pngmono',
@@ -111,8 +116,8 @@ def test_remove_background(spoof_tesseract_noop, resources, outdir):
     output_png = outdir / 'remove_bg.png'
 
     ghostscript.rasterize_pdf(
-        str(output_pdf),
-        str(output_png),
+        output_pdf,
+        output_png,
         xres=100,
         yres=100,
         raster_device='png16m',
@@ -128,7 +133,7 @@ def test_remove_background(spoof_tesseract_noop, resources, outdir):
 @pytest.mark.parametrize(
     "pdf",
     ['palette.pdf', 'cmyk.pdf', 'ccitt.pdf', 'jbig2.pdf', 'lichtenstein.pdf'])
-@pytest.mark.parametrize("renderer", ['hocr', 'tesseract'])
+@pytest.mark.parametrize("renderer", ['auto', 'tesseract'])
 @pytest.mark.parametrize("output_type", ['pdf', 'pdfa'])
 def test_exotic_image(spoof_tesseract_cache, pdf, renderer, output_type,
                       resources, outdir):
@@ -210,10 +215,7 @@ def test_high_unicode(spoof_tesseract_noop, resources, no_outpdf):
     assert p.returncode == ExitCode.bad_args, err
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_oversample(spoof_tesseract_cache, renderer, resources, outpdf):
     oversampled_pdf = check_ocrmypdf(
         resources / 'skew.pdf', outpdf, '--oversample', '350',
@@ -268,9 +270,7 @@ def check_monochrome_correlation(
             print(png)
             return
         ghostscript.rasterize_pdf(
-            str(pdf),
-            str(png),
-            xres=100, yres=100,
+            pdf, png, xres=100, yres=100,
             raster_device='pngmono', log=gslog, pageno=pageno)
 
     rasterize(reference_pdf, reference_pageno, reference_png)
@@ -303,10 +303,7 @@ def test_monochrome_correlation(resources, outdir):
     assert corr > 0.90
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_autorotate(spoof_tesseract_cache, renderer, resources, outdir):
     # cardinal.pdf contains four copies of an image rotated in each cardinal
     # direction - these ones are "burned in" not tagged with /Rotate
@@ -341,10 +338,7 @@ def test_autorotate_threshold(
     assert eval(correlation_test)
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer',RENDERERS)
 def test_ocr_timeout(renderer, resources, outpdf):
     out = check_ocrmypdf(resources / 'skew.pdf', outpdf,
                          '--tesseract-timeout', '1.0')
@@ -359,7 +353,7 @@ def test_skip_big(spoof_tesseract_cache, resources, outpdf):
     assert not pdfinfo[0].has_text
 
 
-@pytest.mark.parametrize('renderer', ['hocr', 'tesseract'])
+@pytest.mark.parametrize('renderer', RENDERERS)
 @pytest.mark.parametrize('output_type', ['pdf', 'pdfa'])
 def test_maximum_options(spoof_tesseract_cache, renderer, output_type,
                          resources, outpdf):
@@ -489,10 +483,7 @@ def test_pagesegmode(renderer, spoof_tesseract_cache, resources, outpdf):
         '--pdf-renderer', renderer, env=spoof_tesseract_cache)
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_crash(renderer, spoof_tesseract_crash,
                          resources, no_outpdf):
     p, out, err = run_ocrmypdf(
@@ -515,10 +506,7 @@ def test_tesseract_crash_autorotate(spoof_tesseract_crash,
     print(err)
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_image_too_big(renderer, spoof_tesseract_big_image_error,
                                  resources, outpdf):
     check_ocrmypdf(
@@ -566,10 +554,7 @@ def test_non_square_resolution(renderer, spoof_tesseract_cache,
     assert in_pageinfo[0].yres == out_pageinfo[0].yres
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract'
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_convert_to_square_resolution(renderer, spoof_tesseract_cache,
                                       resources, outpdf):
     from math import isclose
@@ -770,28 +755,22 @@ language_model_penalty_non_freq_dict_word 0
 
     check_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_config_notfound(renderer, resources, outdir):
     cfg_file = outdir / 'nofile.cfg'
 
     p, out, err = run_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
         '--pdf-renderer', renderer,
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
     assert "Can't open" in err, "No error message about missing config file"
     assert p.returncode == ExitCode.ok
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-    ])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_tesseract_config_invalid(renderer, resources, outdir):
     cfg_file = outdir / 'test.cfg'
     with cfg_file.open('w') as f:
@@ -802,21 +781,50 @@ THIS FILE IS INVALID
     p, out, err = run_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
         '--pdf-renderer', renderer,
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
     assert "parameter not found" in err, "No error message"
     assert p.returncode == ExitCode.invalid_config
 
 
+def test_user_words(resources, outdir):
+    word_list = outdir / 'wordlist.txt'
+    sidecar_before = outdir / 'sidecar_before.txt'
+    sidecar_after = outdir / 'sidecar_after.txt'
+
+    # Don't know how to make this test pass on various versions and platforms
+    # so weaken to merely testing that the argument is accepted
+    consistent = False
+
+    if consistent:
+        check_ocrmypdf(
+            resources / 'crom.png', outdir / 'out.pdf',
+            '--image-dpi', 150,
+            '--sidecar', sidecar_before
+        )
+
+        assert 'cromulent' not in sidecar_before.open().read()
+
+    with word_list.open('w') as f:
+        f.write('cromulent\n')  # a perfectly cromulent word
+
+    check_ocrmypdf(
+        resources / 'crom.png', outdir / 'out.pdf',
+        '--image-dpi', 150,
+        '--sidecar', sidecar_after,
+        '--user-words', word_list
+    )
+
+    if consistent:
+        assert 'cromulent' in sidecar_after.open().read()
+
+
 def test_form_xobject(spoof_tesseract_noop, resources, outpdf):
     check_ocrmypdf(resources / 'formxobject.pdf', outpdf,
                    '--force-ocr',
                    env=spoof_tesseract_noop)
 
 
-@pytest.mark.parametrize('renderer', [
-    'hocr',
-    'tesseract',
-])
+@pytest.mark.parametrize('renderer', RENDERERS)
 def test_pagesize_consistency(renderer, resources, outpdf):
     from math import isclose
 

tests/test_tess3.py

@@ -11,14 +11,26 @@ pytestmark = pytest.mark.skipif(tesseract.v4(),
                                 reason="tesseract 3.x required")
 
 
-def test_textonly_pdf_on_tess3(resources, no_outpdf):
+@pytest.mark.skipif(tesseract.has_textonly_pdf(),
+                    reason="check that missing dep is reported on old tess3")
+def test_textonly_pdf_on_older_tess3(resources, no_outpdf):
     p, _, _ = pytest.helpers.run_ocrmypdf(
         resources / 'linn.pdf',
-        no_outpdf, '--pdf-renderer', 'tess4')
+        no_outpdf, '--pdf-renderer', 'sandwich')
 
     assert p.returncode == ExitCode.missing_dependency
 
 
+@pytest.mark.skipif(not tesseract.has_textonly_pdf(),
+                    reason="check that feature is exercised on new test3")
+def test_textonly_pdf_on_newer_tess3(resources, no_outpdf):
+    p, _, _ = pytest.helpers.run_ocrmypdf(
+        resources / 'linn.pdf',
+        no_outpdf, '--pdf-renderer', 'sandwich')
+
+    assert p.returncode == ExitCode.ok
+
+
 def test_oem_on_tess3(resources, no_outpdf):
     p, _, err = pytest.helpers.run_ocrmypdf(
         resources / 'aspect.pdf',

tests/test_tess4.py

@@ -8,6 +8,7 @@ from ocrmypdf import pdfinfo
 import sys
 import os
 import PyPDF2 as pypdf
+from contextlib import contextmanager
 
 
 spoof = pytest.helpers.spoof
@@ -32,14 +33,25 @@ def ensure_tess4():
     raise EnvironmentError("Can't find Tesseract 4")
 
 
+@contextmanager
+def modified_os_environ(env):
+    old_env = os.environ.copy()
+    os.environ = env
+    yield
+    os.environ = old_env
+
+
 def tess4_available():
     """Check if a tesseract 4 binary is available, even if it's not the
     official "tesseract" on PATH
 
     """
     try:
-        ensure_tess4()
-        return True
+        # ensure_tess4 locates the tess4 binary we are going to check
+        env = ensure_tess4()
+        with modified_os_environ(env):
+            # Now jump into this environment and make sure it really is Tess4
+            return tesseract.v4() and tesseract.has_textonly_pdf()
     except EnvironmentError:
         pass
 
@@ -63,7 +75,6 @@ def test_textonly_pdf(ensure_tess4, resources, outdir):
         env=ensure_tess4)
 
 
-@pytest.mark.skipif(sys.version_info < (3, 5), reason="needs math.isclose")
 def test_pagesize_consistency_tess4(ensure_tess4, resources, outpdf):
     from math import isclose