v5.3 release notes

Omitted French because the rare 'oe' and 'ÿ' glyphs are not in Latin-1.
Basically steer people away from HOCR renderer but avoid a potential
disruptive behavior change.

docs/advanced.rst

@@ -33,9 +33,17 @@ If you want to adjust the amount of time spent on OCR, change ``--tesseract-time
 Overriding default tesseract
 """"""""""""""""""""""""""""
 
-OCRmyPDF checks the environment variable ``OCRMYPDF_TESSERACT`` for the full path *to the tesseract executable* first. 
+OCRmyPDF checks the environment variable ``OCRMYPDF_TESSERACT`` for the full path *to the tesseract binary* first. 
 
-For example, if you are testing tesseract 4.00 and don't wish to disturb your tesseract 3.04 installation, you can launch OCRmyPDF as follows:
+.. envvar:: OCRMYPDF_TESSERACT
+
+	Specify the location of the Tesseract binary.
+
+.. envvar:: TESSDATA_PREFIX
+
+	A Tesseract environment variable that overrides the path to Tesseract's data files.
+
+For example, if you are testing tesseract 4.00 and don't wish to disturb use an existing tesseract 3.04 installation, you can launch OCRmyPDF as follows:
 
 .. code-block:: bash
 
@@ -44,8 +52,8 @@ For example, if you are testing tesseract 4.00 and don't wish to disturb your te
 		TESSDATA_PREFIX=/home/user/src/tesseract4 \
 		ocrmypdf --tesseract-oem 2 input.pdf output.pdf
 
-* ``TESSDATA_PREFIX`` directs tesseract 4.0 to use LSTM training data. This is a tesseract environment variable.
+In this example ``TESSDATA_PREFIX`` directs tesseract 4.0 to use LSTM training data. ``--tesseract-oem 1`` requests tesseract 4.0's new LSTM engine. (Tesseract 4.0 only.)
-* ``--tesseract-oem 1`` requests tesseract 4.0's new LSTM engine. (Tesseract 4.0 only.)
+
 
 Overriding other support programs
 """""""""""""""""""""""""""""""""
@@ -56,7 +64,20 @@ In addition to tesseract, OCRmyPDF uses the following external binaries:
 * ``unpaper``
 * ``qpdf``
 
-In each case OCRmyPDF will check the environment variable ``OCRMYPDF_{program}`` before asking the system to find ``{program}`` on the PATH. For example, you could redirect OCRmyPDF to ``OCRMYPDF_GS`` to override Ghostscript.
+In each case OCRmyPDF will check the environment variable ``OCRMYPDF_{program}`` before asking the system to find ``{program}`` on the PATH. For example, you could redirect OCRmyPDF to ``OCRMYPDF_GS`` to override Ghostscript. The full list is below:
+
+.. envvar:: OCRMYPDF_GS
+
+	Specify the location of the Ghostscript binary.
+
+.. envvar:: OCRMYPDF_UNPAPER
+
+	Specify the location of the unpaper binary.
+
+.. envvar:: OCRMYPDF_QPDF
+
+	Specify the location of the qpdf binary.
+
 
 Changing tesseract configuration variables
 """"""""""""""""""""""""""""""""""""""""""

docs/cookbook.rst

@@ -72,19 +72,11 @@ Language packs must be installed for all languages specified. See :ref:`Installi
 Produce PDF and text file containing OCR text
 """""""""""""""""""""""""""""""""""""""""""""
 
-This produces a file named "output.pdf" and a companion text file named "output.txt". The ``pdftotext`` program from 
+This produces a file named "output.pdf" and a companion text file named "output.txt".
-`Poppler <https://poppler.freedesktop.org/>`_ is used to extract text from the finished PDF.
-
 
 .. code-block:: bash
 
-	ocrmypdf input.pdf - | tee output.pdf | pdftotext - output.txt
+	ocrmypdf --sidecar output.txt input.pdf output.pdf
-
-.. note::
-
-	To get pdftotext, Debian/Ubuntu users may ``apt-get install poppler-utils`` 
-	and macOS users may ``brew install poppler`` respectively.
-
 
 OCR images, not PDFs
 --------------------

docs/release_notes.rst

@@ -6,6 +6,15 @@ OCRmyPDF uses `semantic versioning <http://semver.org/>`_ for its command line i
 The OCRmyPDF package itself does not contain a public API, although it is fairly stable and breaking changes are usually timed with a major release. A future release will clearly define the stable public API.
 
 
+v5.3
+----
+
+-   Added ``--user-words`` and ``--user-patterns`` arguments which are forwarded to Tesseract OCR as words and regular expressions respective to use to guide OCR. Supplying a list of subject-domain words should assist Tesseract with resolving words. (#165)
+-   Using a non Latin-1 language with the "hocr" renderer now warns about possible OCR quality and recommends workarounds (#176)
+-   Output file path added to error message when that location is not writable (#175)
+-   Otherwise valid PDFs with leading whitespace at the beginning of the file are now accepted
+
+
 v5.2
 ----
 

ocrmypdf/__main__.py

@@ -41,6 +41,9 @@ warnings.simplefilter('ignore', pypdf.utils.PdfReadWarning)
 
 MINIMUM_TESS_VERSION = '3.04'
 
+HOCR_OK_LANGS = frozenset([
+    'eng', 'deu', 'spa', 'ita', 'por'
+])
 
 def complain(message):
     print(*textwrap.wrap(message), file=sys.stderr)
@@ -281,6 +284,15 @@ advanced.add_argument(
          "are applied to all pages, including those for which OCR was "
          "skipped.  Not supported for --output-type=pdf ; that setting "
          "preserves the original compression of all images.")
+advanced.add_argument(
+    '--user-words', metavar='FILE',
+    help="Specify the location of the Tesseract user words file. This is a "
+         "list of words Tesseract should consider while performing OCR in "
+         "addition to its standard language dictionaries. This can improve "
+         "OCR quality especially for specialized and technical documents.")
+advanced.add_argument(
+    '--user-patterns', metavar='FILE',
+    help="Specify the location of the Tesseract user patterns file.")
 
 debugging = parser.add_argument_group(
     "Debugging",
@@ -386,12 +398,20 @@ def check_options_ocr_behavior(options, log):
     #     raise argparse.ArgumentError(
     #         "Error: --redo-ocr and other OCR options are incompatible.")
 
-    if set(options.language) & {'chi_sim', 'chi_tra'} and \
+    if options.pdf_renderer == 'hocr' and \
-            (options.pdf_renderer == 'hocr' or options.output_type == 'pdfa'):
+            not set(options.language).issubset(HOCR_OK_LANGS):
-        log.warning(
+        msg = (
-            "Your settings are known to cause problems with OCR of Chinese text. "
+            "The 'hocr' PDF renderer is known to cause problems with one "
-            "Try adding these arguments: "
+            "or more of the languages in your document. ")
-            "    ocrmypdf --pdf-renderer tesseract --output-type pdf")
+
+        if tesseract.has_textonly_pdf():
+            msg += (
+                "Use --pdf-renderer auto (the default) to avoid this issue.")
+        else:
+            msg += (
+                "Use --pdf-renderer tesseract --output-type pdf to avoid "
+                "this issue")
+        log.warning(msg)
 
 
 def check_options_advanced(options, log):
@@ -666,8 +686,9 @@ def run_pipeline():
                     file."""))
                 return ExitCode.bad_args
         elif not is_file_writable(options.output_file):
-            _log.error(textwrap.dedent("""\
+            _log.error(
-                Output file location is not writable."""))
+                "Output file location (" + options.output_file + ") " +
+                "is not writable.")
             return ExitCode.file_access_error
 
         manager = JobContextManager()

ocrmypdf/exec/ghostscript.py

@@ -10,6 +10,7 @@ import sys
 from . import get_program
 from ..exceptions import SubprocessOutputError
 from PIL import Image
+from ..helpers import fspath
 
 
 @lru_cache(maxsize=1)
@@ -43,8 +44,8 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
     (xres, yres) even if those numbers are noninteger. The image's DPI will
      be overridden with the values in page_dpi.
     
-    :param input_file: 
+    :param input_file: pathlike
-    :param output_file: 
+    :param output_file: pathlike
     :param xres: resolution at which to rasterize page
     :param yres: 
     :param raster_device: 
@@ -69,7 +70,7 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
             '-dLastPage=%i' % pageno,
             '-o', tmp.name,
             '-r{0}x{1}'.format(str(int_res[0]), str(int_res[1])),
-            input_file
+            fspath(input_file)
         ]
 
         p = run(args_gs, stdout=PIPE, stderr=STDOUT,
@@ -95,9 +96,10 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
                 log.debug(
                     "Ghostscript: resize output image {} -> {}".format(
                         im.size, expected_size))
-                im.resize(expected_size).save(output_file, dpi=page_dpi)
+                im.resize(expected_size).save(
+                    fspath(output_file), dpi=page_dpi)
             else:
-                copy(tmp.name, output_file)
+                copy(tmp.name, fspath(output_file))
 
 
 def generate_pdfa(pdf_pages, output_file, compression, log,

ocrmypdf/exec/tesseract.py

@@ -213,7 +213,8 @@ def _generate_null_hocr(output_hocr, output_sidecar, image):
 
 def generate_hocr(input_file, output_files, language: list, engine_mode,
                   tessconfig: list,
-                  timeout: float, pagesegmode: int, log):
+                  timeout: float, pagesegmode: int, user_words, user_patterns,
+                  log):
 
     output_hocr = next(o for o in output_files if o.endswith('.hocr'))
     output_sidecar = next(o for o in output_files if o.endswith('.txt'))
@@ -224,10 +225,17 @@ def generate_hocr(input_file, output_files, language: list, engine_mode,
     if pagesegmode is not None:
         args_tesseract.extend([psm(), str(pagesegmode)])
 
+    if user_words:
+        args_tesseract.extend(['--user-words', user_words])
+
+    if user_patterns:
+        args_tesseract.extend(['--user-patterns', user_patterns])
+
     # Reminder: test suite tesseract spoofers will break after any changes
     # to the number of order parameters here
     # Tesseract 3.04 requires the order here to be "hocr txt" and will fail
     # on "txt hocr"
+
     args_tesseract.extend([
         input_file,
         prefix,
@@ -285,7 +293,8 @@ def use_skip_page(text_only, skip_pdf, output_pdf, output_text):
 
 def generate_pdf(*, input_image, skip_pdf, output_pdf, output_text,
                  language: list, engine_mode, text_only: bool,
-                 tessconfig: list, timeout: float, pagesegmode: int, log):
+                 tessconfig: list, timeout: float, pagesegmode: int,
+                 user_words, user_patterns, log):
     '''Use Tesseract to render a PDF.
 
     input_image -- image to analyze
@@ -308,10 +317,17 @@ def generate_pdf(*, input_image, skip_pdf, output_pdf, output_text,
     if text_only:
         args_tesseract.extend(['-c', 'textonly_pdf=1'])
 
+    if user_words:
+        args_tesseract.extend(['--user-words', user_words])
+
+    if user_patterns:
+        args_tesseract.extend(['--user-patterns', user_patterns])
+
     prefix = os.path.splitext(output_pdf)[0]  # Tesseract appends suffixes
 
-    # Reminder: test suite tesseract spoofers will break after any changes
+    # Reminder: test suite tesseract spoofers might break after any changes
     # to the number of order parameters here
+
     args_tesseract.extend([
         input_image,
         prefix,

ocrmypdf/helpers.py

@@ -79,10 +79,43 @@ def is_file_writable(test_file):
         return True
 
 
-@contextmanager
+if sys.version_info[0:2] <= (3, 5):
-def universal_open(p, *args, **kwargs):
+    def universal_open(p, *args, **kwargs):
-    "Work around Python 3.5's inability to open(pathlib.Path())"
+        "Work around Python 3.5's inability to open(pathlib.Path())"
-    try:
+        try:
-        yield p.open(*args, **kwargs)
+            return p.open(*args, **kwargs)
-    except AttributeError:
+        except AttributeError:
-        yield open(p, *args, **kwargs)
+            return open(p, *args, **kwargs)
+
+
+    def fspath(path):
+        import pathlib
+        '''https://www.python.org/dev/peps/pep-0519/#os'''
+        if isinstance(path, (str, bytes)):
+            return path
+
+        # Work from the object's type to match method resolution of other magic
+        # methods.
+        path_type = type(path)
+        try:
+            path = path_type.__fspath__(path)
+        except AttributeError:
+            # Added for Python 3.5 support.
+            if isinstance(path, pathlib.Path):
+                return str(path)
+            elif hasattr(path_type, '__fspath__'):
+                raise
+        else:
+            if isinstance(path, (str, bytes)):
+                return path
+            else:
+                raise TypeError("expected __fspath__() to return str or bytes, "
+                                "not " + type(path).__name__)
+
+        raise TypeError(
+            "expected str, bytes, pathlib.Path or os.PathLike object, not "
+            + path_type.__name__)
+
+else:
+    universal_open = open
+    fspath = os.fspath

ocrmypdf/leptonica.py

@@ -14,6 +14,7 @@ from ctypes.util import find_library
 from .lib._leptonica import ffi
 from functools import lru_cache
 from enum import Enum
+from .helpers import fspath
 
 lept = ffi.dlopen(find_library('lept'))
 
@@ -208,26 +209,27 @@ class Pix:
             return 'P'
 
     @classmethod
-    def read(cls, filename):
+    def read(cls, path):
         """Load an image file into a PIX object.
 
         Leptonica can load TIFF, PNM (PBM, PGM, PPM), PNG, and JPEG.  If
         loading fails then the object will wrap a C null pointer.
         """
+        filename = fspath(path)
         with LeptonicaErrorTrap():
-            return cls(lept.pixRead(
+            return cls(lept.pixRead(os.fsencode(filename)))
-                filename.encode(sys.getfilesystemencoding())))
 
     def write_implied_format(
-            self, filename, jpeg_quality=0, jpeg_progressive=0):
+            self, path, jpeg_quality=0, jpeg_progressive=0):
         """Write pix to the filename, with the extension indicating format.
 
         jpeg_quality -- quality (iff JPEG; 1 - 100, 0 for default)
         jpeg_progressive -- (iff JPEG; 0 for baseline seq., 1 for progressive)
         """
+        filename = fspath(path)
         with LeptonicaErrorTrap():
             lept.pixWriteImpliedFormat(
-                filename.encode(sys.getfilesystemencoding()),
+                os.fsencode(filename),
                 self._pix, jpeg_quality, jpeg_progressive)
 
     def topil(self):

ocrmypdf/pipeline.py

@@ -165,8 +165,8 @@ def triage(
     options = context.get_options()
     try:
         with open(input_file, 'rb') as f:
-            signature = f.read(4)
+            signature = f.read(1024)
-            if signature == b'%PDF':
+            if b'%PDF' in signature:
                 if options.image_dpi:
                     log.warning("Argument --image-dpi ignored because the "
                                 "input file is a PDF, not an image.")
@@ -202,7 +202,7 @@ def repair_pdf(
 
 
 def get_pageinfo(input_file, context):
-    pageno = int(os.path.basename(input_file)[0:6]) - 1
+    pageno = page_number(input_file) - 1
     pageinfo = context.get_pdfinfo()[pageno]
     return pageinfo
 
@@ -406,7 +406,7 @@ def orient_page(
         with open(output_file, 'wb') as out:
             writer.write(out)
 
-        pageno = int(os.path.basename(page_pdf)[0:6]) - 1
+        pageno = page_number(page_pdf) - 1
         pdfinfo = context.get_pdfinfo()
         pdfinfo[pageno].rotation = orient_conf.angle
         context.set_pdfinfo(pdfinfo)
@@ -525,6 +525,8 @@ def ocr_tesseract_hocr(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log
         )
 
@@ -755,6 +757,8 @@ def ocr_tesseract_and_render_pdf(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log)
 
 
@@ -783,6 +787,8 @@ def ocr_tesseract_textonly_pdf(
         tessconfig=options.tesseract_config,
         timeout=options.tesseract_timeout,
         pagesegmode=options.tesseract_pagesegmode,
+        user_words=options.user_words,
+        user_patterns=options.user_patterns,
         log=log)
 
 
@@ -863,7 +869,7 @@ def merge_pages_ghostscript(
         right that way.'''
         if s.endswith('.ps'):
             return 99999999
-        key = int(os.path.basename(s)[0:6]) * 10
+        key = page_number(s) * 10
         if 'debug' in os.path.basename(s):
             key += 1
         return key
@@ -898,7 +904,7 @@ def merge_pages_qpdf(
     def input_file_order(s):
         '''Sort order: All rendered pages followed
         by their debug page.'''
-        key = int(os.path.basename(s)[0:6]) * 10
+        key = page_number(s) * 10
         if 'debug' in os.path.basename(s):
             key += 1
         return key

tests/conftest.py

@@ -129,7 +129,8 @@ def run_ocrmypdf(input_file, output_file, *args, env=None):
     if env is None:
         env = os.environ
 
-    p_args = OCRMYPDF + list(args) + [str(input_file), str(output_file)]
+    p_args = OCRMYPDF + [str(arg) for arg in args] + \
+             [str(input_file), str(output_file)]
     p = Popen(
         p_args, close_fds=True, stdout=PIPE, stderr=PIPE,
         universal_newlines=True, env=env)

tests/test_main.py

@@ -77,8 +77,8 @@ def test_deskew(spoof_tesseract_noop, resources, outdir):
     deskewed_png = outdir / 'deskewed.png'
 
     ghostscript.rasterize_pdf(
-        str(deskewed_pdf),
+        deskewed_pdf,
-        str(deskewed_png),
+        deskewed_png,
         xres=150,
         yres=150,
         raster_device='pngmono',
@@ -116,8 +116,8 @@ def test_remove_background(spoof_tesseract_noop, resources, outdir):
     output_png = outdir / 'remove_bg.png'
 
     ghostscript.rasterize_pdf(
-        str(output_pdf),
+        output_pdf,
-        str(output_png),
+        output_png,
         xres=100,
         yres=100,
         raster_device='png16m',
@@ -270,9 +270,7 @@ def check_monochrome_correlation(
             print(png)
             return
         ghostscript.rasterize_pdf(
-            str(pdf),
+            pdf, png, xres=100, yres=100,
-            str(png),
-            xres=100, yres=100,
             raster_device='pngmono', log=gslog, pageno=pageno)
 
     rasterize(reference_pdf, reference_pageno, reference_png)
@@ -485,10 +483,7 @@ def test_pagesegmode(renderer, spoof_tesseract_cache, resources, outpdf):
         '--pdf-renderer', renderer, env=spoof_tesseract_cache)
 
 
-@pytest.mark.parametrize('renderer', [
+@pytest.mark.parametrize('renderer', RENDERERS)
-    'hocr',
-    'tesseract',
-    ])
 def test_tesseract_crash(renderer, spoof_tesseract_crash,
                          resources, no_outpdf):
     p, out, err = run_ocrmypdf(
@@ -760,7 +755,7 @@ language_model_penalty_non_freq_dict_word 0
 
     check_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
 
 
 @pytest.mark.parametrize('renderer', RENDERERS)
@@ -770,7 +765,7 @@ def test_tesseract_config_notfound(renderer, resources, outdir):
     p, out, err = run_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
         '--pdf-renderer', renderer,
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
     assert "Can't open" in err, "No error message about missing config file"
     assert p.returncode == ExitCode.ok
 
@@ -786,11 +781,43 @@ THIS FILE IS INVALID
     p, out, err = run_ocrmypdf(
         resources / 'ccitt.pdf', outdir / 'out.pdf',
         '--pdf-renderer', renderer,
-        '--tesseract-config', str(cfg_file))
+        '--tesseract-config', cfg_file)
     assert "parameter not found" in err, "No error message"
     assert p.returncode == ExitCode.invalid_config
 
 
+def test_user_words(resources, outdir):
+    word_list = outdir / 'wordlist.txt'
+    sidecar_before = outdir / 'sidecar_before.txt'
+    sidecar_after = outdir / 'sidecar_after.txt'
+
+    # Don't know how to make this test pass on various versions and platforms
+    # so weaken to merely testing that the argument is accepted
+    consistent = False
+
+    if consistent:
+        check_ocrmypdf(
+            resources / 'crom.png', outdir / 'out.pdf',
+            '--image-dpi', 150,
+            '--sidecar', sidecar_before
+        )
+
+        assert 'cromulent' not in sidecar_before.open().read()
+
+    with word_list.open('w') as f:
+        f.write('cromulent\n')  # a perfectly cromulent word
+
+    check_ocrmypdf(
+        resources / 'crom.png', outdir / 'out.pdf',
+        '--image-dpi', 150,
+        '--sidecar', sidecar_after,
+        '--user-words', word_list
+    )
+
+    if consistent:
+        assert 'cromulent' in sidecar_after.open().read()
+
+
 def test_form_xobject(spoof_tesseract_noop, resources, outpdf):
     check_ocrmypdf(resources / 'formxobject.pdf', outpdf,
                    '--force-ocr',