v10.2.1 release notes

More typing improvements
Typing fixes bugs.
2020-07-01 03:26:57 -07:00 · 2020-06-30 15:02:30 -07:00 · 2020-06-30 04:20:58 -07:00 · 2020-06-30 04:20:14 -07:00 · 2020-06-30 04:17:30 -07:00 · 2020-06-29 02:17:14 -07:00
25 changed files with 388 additions and 270 deletions
@@ -3,15 +3,10 @@
 [![Build Status][azure]](https://dev.azure.com/jim0585/ocrmypdf/_build/latest?definitionId=2&branchName=master) [![PyPI version][pypi]](https://pypi.org/project/ocrmypdf/) ![Homebrew version][homebrew] ![ReadTheDocs][docs] ![Python versions][pyversions]

 [azure]: https://dev.azure.com/jim0585/ocrmypdf/_apis/build/status/jbarlow83.OCRmyPDF?branchName=master
-
 [travis]: https://travis-ci.org/jbarlow83/OCRmyPDF.svg?branch=master "Travis build status"
-
 [pypi]: https://img.shields.io/pypi/v/ocrmypdf.svg "PyPI version"
-
 [homebrew]: https://img.shields.io/homebrew/v/ocrmypdf.svg "Homebrew version"
-
 [docs]: https://readthedocs.org/projects/ocrmypdf/badge/?version=latest "RTD"
-
 [pyversions]: https://img.shields.io/pypi/pyversions/ocrmypdf "Supported Python versions"

 OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched or copy-pasted.
@@ -30,8 +25,7 @@ ocrmypdf                      # it's a scriptable command line program

 [See the release notes for details on the latest changes](https://ocrmypdf.readthedocs.io/en/latest/release_notes.html).

-Main features
-------------
+## Main features

 - Generates a searchable [PDF/A](https://en.wikipedia.org/?title=PDF/A) file from a regular PDF
 - Places OCR text accurately below the image to ease copy / paste
@@ -47,8 +41,7 @@ Main features

 For details: please consult the [documentation](https://ocrmypdf.readthedocs.io/en/latest/).

-Motivation
----------
+## Motivation

 I searched the web for a free command line tool to OCR PDF files: I found many, but none of them were really satisfying:

@@ -62,8 +55,7 @@ I searched the web for a free command line tool to OCR PDF files: I found many,

 ...so I decided to develop my own tool.

-Installation
------------
+## Installation

 Linux, Windows, macOS and FreeBSD are supported. Docker images are also available.

@@ -87,8 +79,7 @@ brew install ocrmypdf

 For everyone else, [see our documentation](https://ocrmypdf.readthedocs.io/en/latest/installation.html) for installation steps.

-Languages
---------
+## Languages

 OCRmyPDF uses Tesseract for OCR, and relies on its language packs. For Linux users, you can often find packages that provide language packs:

@@ -105,8 +96,7 @@ pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English a

 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.

-Documentation and support
-------------------------
+## Documentation and support

 Once OCRmyPDF is installed, the built-in help which explains the command syntax and options can be accessed via:

@@ -118,27 +108,24 @@ Our [documentation is served on Read the Docs](https://ocrmypdf.readthedocs.io/e

 Please report issues on our [GitHub issues](https://github.com/jbarlow83/OCRmyPDF/issues) page, and follow the issue template for quick response.

-Requirements
------------
+## Requirements

 In addition to the required Python version (3.6+), OCRmyPDF requires external program installations of Ghostscript, Tesseract OCR, QPDF, and Leptonica. OCRmyPDF is pure Python, but uses CFFI to portably generate library bindings. OCRmyPDF works on pretty much everything: Linux, macOS, Windows and FreeBSD.

-Press & Media
-------------
+## Press & Media

 - [Going paperless with OCRmyPDF](https://medium.com/@ikirichenko/going-paperless-with-ocrmypdf-e2f36143f46a)
 - [Converting a scanned document into a compressed searchable PDF with redactions](https://medium.com/@treyharris/converting-a-scanned-document-into-a-compressed-searchable-pdf-with-redactions-63f61c34fe4c)
 - [c't 1-2014, page 59](https://heise.de/-2279695): Detailed presentation of OCRmyPDF v1.0 in the leading German IT magazine c't
 - [heise Open Source, 09/2014: Texterkennung mit OCRmyPDF](https://heise.de/-2356670)
 - [heise Durchsuchbare PDF-Dokumente mit OCRmyPDF erstellen](https://www.heise.de/ratgeber/Durchsuchbare-PDF-Dokumente-mit-OCRmyPDF-erstellen-4607592.html)
+- [Excellent Utilities: OCRmyPDF](https://www.linuxlinks.com/excellent-utilities-ocrmypdf-add-ocr-text-layer-scanned-pdfs/)

-Business enquiries
------------------
+## Business enquiries

 OCRmyPDF would not be the software that it is today without companies and users choosing to provide support for feature development and consulting enquiries. We are happy to discuss all enquiries, whether for extending the existing feature set, or integrating OCRmyPDF into a larger system.

-License
-------
+## License

 The OCRmyPDF software is licensed under the GNU GPLv3. Certain files are covered by other licenses, as noted in their source files.

@@ -146,7 +133,6 @@ The license for each test file varies, and is noted in tests/resources/README.rs

 OCRmyPDF versions prior to 6.0 were distributed under the MIT License.

-Disclaimer
----------
+## Disclaimer

 The software is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@@ -106,7 +106,3 @@ Reference
    :undoc-members:

 .. autofunction:: ocrmypdf.configure_logging
-
-.. automodule:: ocrmypdf.exceptions
-    :members:
-    :undoc-members:
@@ -0,0 +1,43 @@
+=============
+API Reference
+=============
+
+This page summarizes the rest of the public API. Generally speaking this
+should mainly of interest to plugin developers.
+
+ocrmypdf.exceptions
+===================
+
+.. automodule:: ocrmypdf.exceptions
+    :members:
+    :undoc-members:
+
+ocrmypdf.helpers
+================
+
+.. automodule:: ocrmypdf.helpers
+    :members:
+
+ocrmypdf.hocrtransform
+======================
+
+.. automodule:: ocrmypdf.hocrtransform
+    :members:
+
+ocrmypdf.pdfa
+=============
+
+.. automodule:: ocrmypdf.pdfa
+    :members:
+
+ocrmypdf.quality
+================
+
+.. automodule:: ocrmypdf.quality
+    :members:
+
+ocrmypdf.subprocess
+===================
+
+.. automodule:: ocrmypdf.subprocess
+    :members:
@@ -1,5 +1,4 @@
 #!/usr/bin/env python3
-# -*- coding: utf-8 -*-
 #
 # ocrmypdf documentation build configuration file, created by
 # sphinx-quickstart on Sun Sep  4 14:29:43 2016.
@@ -21,6 +20,8 @@
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))

+"""isort:skip_file"""
+
 # -- General configuration ------------------------------------------------

 # If your documentation needs a minimal Sphinx version, state it here.
@@ -32,6 +33,8 @@
 # ones.
 extensions = ['sphinx.ext.napoleon']

+napoleon_use_rtype = False
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']

@@ -51,7 +54,7 @@ master_doc = 'index'
 # General information about the project.
 project = 'ocrmypdf'
 copyright = (
-    '2019, James R. Barlow. Licensed under Creative Commons Attribution-ShareAlike 4.0.'
+    '2020, James R. Barlow. Licensed under Creative Commons Attribution-ShareAlike 4.0.'
 )
 author = 'James R. Barlow'

@@ -90,6 +93,7 @@ from pkg_resources import get_distribution, DistributionNotFound
 release = get_distribution('ocrmypdf').version
 version = '.'.join(release.split('.')[:2])

+
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
 #
@@ -174,7 +178,7 @@ html_theme_options = {'display_version': False}
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
 #
-# html_logo = None
+# html_logo = "images/logo.svg"  # looks bad

 # The name of an image file (relative to this directory) to use as a favicon of
 # the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -36,6 +36,7 @@ image processing and OCR to existing PDFs.

   api
   plugins
+   apiref
   contributing

 Indices and tables
@@ -8,20 +8,26 @@ Installing OCRmyPDF
 |latest|

 The easiest way to install OCRmyPDF is to follow the steps for your operating
-system/platform, although sometimes this version may be out of date. This
-installation guide provides information allowing you to compare the current
-version to the one provided by your platform.
+system/platform. This version may be out of date, however.

-If you want to use the latest version of OCRmyPDF and all of its optional
-dependencies, the easiest way to get that is install the Homebrew package. Homebrew
-is best known as a macOS package manger, but also works for
-`Linux and Windows Subsystem for Linux <https://docs.brew.sh/Homebrew-on-Linux>`__.
-After Homebrew is installed, simply run ``brew install ocrmypdf``.
+These platforms have one-liner installs:

-You can also use the more detailed procedures here to manually install OCRmyPDF
-from source or with the ``pip`` package manager for binary wheels. The reason
-for these varied steps is that OCRmyPDF requires third-party executables that are
-not part of Python.
+-----------------------------+-------------------------------+
+| Debian, Ubuntu              | ``apt install ocrmypdf``      |
+-----------------------------+-------------------------------+
+| Windows Subsystem for Linux | ``apt install ocrmypdf``      |
+-----------------------------+-------------------------------+
+| Fedora                      | ``dnf install ocrmypdf``      |
+-----------------------------+-------------------------------+
+| macOS                       | ``brew install ocrmypdf``     |
+-----------------------------+-------------------------------+
+| LinuxBrew                   | ``brew install ocrmypdf``     |
+-----------------------------+-------------------------------+
+| FreeBSD                     | ``pkg install py37-ocrmypdf`` |
+-----------------------------+-------------------------------+
+
+More detailed procedures are outlined below. If you want to do a manual
+install, or install a more recent version than your platform provides, read on.

 .. contents:: Platform-specific steps
    :depth: 2
@@ -58,7 +64,7 @@ Debian and Ubuntu 18.04 or newer
 | |ubu-1804| |ubu-2004|                         |
 +-----------------------------------------------+

-Users of Debian 9 ("stretch") or later or Ubuntu 18.04 or later, including users
+Users of Debian 9 ("stretch") or later, or Ubuntu 18.04 or later, including users
 of Windows Subsystem for Linux, may simply

 .. code-block:: bash
@@ -87,11 +93,11 @@ For full details on version availability for your platform, check the
 Fedora 29 or newer
 ------------------

-.. |fedora-29| image:: https://repology.org/badge/version-for-repo/fedora_29/ocrmypdf.svg
-    :alt: Fedora 29
+.. |fedora-31| image:: https://repology.org/badge/version-for-repo/fedora_31/ocrmypdf.svg
+    :alt: Fedora 31

-.. |fedora-30| image:: https://repology.org/badge/version-for-repo/fedora_30/ocrmypdf.svg
-    :alt: Fedora 30
+.. |fedora-32| image:: https://repology.org/badge/version-for-repo/fedora_32/ocrmypdf.svg
+    :alt: Fedora 32

 .. |fedora-rawhide| image:: https://repology.org/badge/version-for-repo/fedora_rawhide/ocrmypdf.svg
    :alt: Fedore Rawhide
@@ -101,7 +107,7 @@ Fedora 29 or newer
 +-----------------------------------------------+
 | |latest|                                      |
 +-----------------------------------------------+
-| |fedora-29| |fedora-30| |fedora-rawhide|      |
+| |fedora-31| |fedora-32| |fedora-rawhide|      |
 +-----------------------------------------------+

 Users of Fedora 29 or later may simply
@@ -162,7 +168,7 @@ To install for the current user only:
    pip3 install --user ocrmypdf

 Ubuntu 18.04 LTS
-------------------------------------------------
+----------------

 Ubuntu 18.04 includes ocrmypdf 6.1.2 - you can install that with ``apt``, but
 it is quite old now. To install a more recent version, uninstall the old version
@@ -265,68 +271,6 @@ environment variable contains ``$HOME/.local/bin``.

 To add JBIG2 encoding, see :ref:`jbig2`.

-Ubuntu 14.04 LTS
----------------
-
-Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some
-other options, because of its age. Several backports are required. For
-explanations of some steps of this procedure, see the similar steps for
-Ubuntu 16.04.
-
-Install system dependencies:
-
-.. code-block:: bash
-
-    sudo apt-get update
-    sudo apt-get install \
-        software-properties-common python-software-properties \
-        zlib1g-dev \
-        libexempi3 \
-        libjpeg-dev \
-        libffi-dev \
-        pngquant \
-        qpdf
-
-We will need backports of Ghostscript 9.16, libav-11 (for unpaper 6.1),
-Tesseract 4.00 (alpha), and Python 3.6. This will replace Ghostscript
-and Tesseract 3.x on your system. Python 3.6 will be installed alongside
-the system Python 3.4.
-
-If you prefer to not modify your system in this matter, consider using a
-Docker container.
-
-.. code-block:: bash
-
-    sudo add-apt-repository ppa:vshn/ghostscript -y
-    sudo add-apt-repository ppa:heyarje/libav-11 -y
-    sudo add-apt-repository ppa:alex-p/tesseract-ocr -y
-    sudo add-apt-repository ppa:jonathonf/python-3.6 -y
-
-    sudo apt-get update
-
-    sudo apt-get install \
-        python3.6-dev \
-        ghostscript \
-        tesseract-ocr \
-        tesseract-ocr-eng \
-        libavformat56 libavcodec56 libavutil54 \
-        wget
-
-Now we need to install ``pip`` and let it install ocrmypdf:
-
-.. code-block:: bash
-
-    curl https://bootstrap.pypa.io/ez_setup.py -o - | python3.6 && python3.6 -m easy_install pip
-    pip3.6 install ocrmypdf
-
-The optional dependency ``unpaper`` is only available at 0.4.2 in Ubuntu 14.04,
-and no backports are available. Previously the author maintained a backported
-.deb package for unpaper 6.1, but since Ubuntu 14.04 is now end of life, this is
-not supported. As such, ``unpaper`` is not available on Ubuntu 14.04 or must by
-compiled by hand.
-
-To add JBIG2 encoding, see :ref:`jbig2`.
-
 Arch Linux (AUR)
 ----------------

@@ -407,6 +351,43 @@ To install OCRmyPDF for Alpine Linux:

    apk add ocrmypdf

+Mageia 7
+--------
+
+Install the following dependencies:
+
+.. code-block:: bash
+
+    # As root user
+    urpmi.update -a
+    urpmi \
+        ghostscript \
+        icc-profiles-openicc \
+        jbig2dec \
+        lib64leptonica5 \
+        pngquant \
+        python3-pip \
+        python3-cffi \
+        python3-distutils-extra \
+        python3-pkg-resources \
+        python3-reportlab \
+        qpdf \
+        tesseract \
+        tesseract-osd \
+        tesseract-eng \
+        tesseract-fra
+
+To install ocrmypdf for the system:
+
+    # As root user
+    pip3 install ocrmypdf
+    ldconfig
+
+Or, to install for the current user only:
+
+    export PATH=$HOME/.local/bin:$PATH
+    pip3 install --user ocrmypdf
+
 Other Linux packages
 --------------------

@@ -507,40 +488,15 @@ The command line program should now be available:

    ocrmypdf --help

-Installing on FreeBSD
-=====================
-
-.. image:: https://repology.org/badge/version-for-repo/freebsd/python:ocrmypdf.svg
-    :alt: FreeBSD
-    :target: https://repology.org/project/python:ocrmypdf/versions
-
-FreeBSD 11.3, 12.0, 12.1-RELEASE and 13.0-CURRENT are supported. Other
-versions likely work but have not been tested.
-
-.. code-block:: bash
-
-    pkg install py37-ocrmypdf
-
-To install a more recent version, you could attempt to first install the system
-version with ``pkg``, then use ``pip install --user ocrmypdf``.
-
-Installing the Docker image
-===========================
-
-For some users, installing the Docker image will be easier than
-installing all of OCRmyPDF's dependencies.
-
-See `OCRmyPDF Docker Image <docker>`__ for more information.
-
 Installing on Windows
 =====================

-.. warning::
+Native Windows
+--------------

-    Native Windows support is new. Consider it "beta" software. Some
-    functionality is missing or may be more difficult to enable. If you need a
-    production-ready solution, use Windows Subsystem for Linux or a Docker
-    image.
+.. note::
+
+    It is easier to install OCRmyPDF on Windows Subsystem for Linux.

 .. note::

@@ -548,7 +504,7 @@ Installing on Windows

 You must install the following for Windows:

-* Python 3.7 (64-bit)
+* Python 3.7 (64-bit) or later
 * Tesseract 4.0 or later
 * Ghostscript 9.50 or later

@@ -577,8 +533,8 @@ You may then use pip to install ocrmypdf:

 * ``pip install ocrmypdf``

-Installing on Windows Subsystem for Linux
-=========================================
+Windows Subsystem for Linux
+---------------------------

 #. Install Ubuntu 18.04 for Windows Subsystem for Linux, if not already installed.
 #. Follow the procedure to install :ref:`OCRmyPDF on Ubuntu 18.04 <ubuntu-lts-latest>`.
@@ -597,14 +553,8 @@ Then confirm that the expected version from PyPI (|latest|) is installed:
 You can then run OCRmyPDF in the Windows command prompt or Powershell, prefixing
 ``wsl``, and call it from Windows programs or batch files.

-Docker
-^^^^^^
-
-You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
-your command prompt can run the docker "hello world" container.
-
-Installing on Cygwin64 under Windows
-====================================
+Cygwin64
+--------

 First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::

@@ -650,6 +600,37 @@ The optional dependency "unpaper" that is currently not available under Cygwin.
 Without it, certain options such as ``--clean`` will produce an error message.
 However, the OCR-to-text-layer functionality is available.

+Docker
+------
+
+You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
+your command prompt can run the docker "hello world" container.
+
+Installing on FreeBSD
+=====================
+
+.. image:: https://repology.org/badge/version-for-repo/freebsd/python:ocrmypdf.svg
+    :alt: FreeBSD
+    :target: https://repology.org/project/python:ocrmypdf/versions
+
+FreeBSD 11.3, 12.0, 12.1-RELEASE and 13.0-CURRENT are supported. Other
+versions likely work but have not been tested.
+
+.. code-block:: bash
+
+    pkg install py37-ocrmypdf
+
+To install a more recent version, you could attempt to first install the system
+version with ``pkg``, then use ``pip install --user ocrmypdf``.
+
+Installing the Docker image
+===========================
+
+For some users, installing the Docker image will be easier than
+installing all of OCRmyPDF's dependencies.
+
+See `OCRmyPDF Docker Image <docker>`__ for more information.
+
 Installing with Python pip
 ==========================

@@ -76,5 +76,43 @@ A plugin may provide the following hooks. Hooks should be decorated with
 The following is a complete list of hooks that may be installed and when
 they are called.

-.. automodule:: ocrmypdf.pluginspec
+Custom command line arguments
+-----------------------------
+
+.. autofunction:: ocrmypdf.pluginspec.add_options
+
+.. autofunction:: ocrmypdf.pluginspec.check_options
+
+Applying special behavior before processing
+-------------------------------------------
+
+.. autofunction:: ocrmypdf.pluginspec.validate
+
+PDF page to image
+-----------------
+
+.. autofunction:: ocrmypdf.pluginspec.rasterize_pdf_page
+
+Modifying intermediate images
+-----------------------------
+
+.. autofunction:: ocrmypdf.pluginspec.filter_ocr_image
+
+.. autofunction:: ocrmypdf.pluginspec.filter_page_image
+
+OCR engine
+----------
+
+.. autofunction:: ocrmypdf.pluginspec.get_ocr_engine
+
+.. autoclass:: ocrmypdf.pluginspec.OcrEngine
    :members:
+
+    .. automethod:: __str__
+
+.. autoclass:: ocrmypdf.pluginspec.OrientationConfidence
+
+PDF/A production
+----------------
+
+.. autofunction:: ocrmypdf.pluginspec.generate_pdfa
@@ -16,6 +16,15 @@ Note that it is licensed under GPLv3, so scripts that
 ``import ocrmypdf`` and are released publicly should probably also be
 licensed under GPLv3.

+v10.2.1
+=======
+
+-  Disabled calculation of text box order with pdfminer. We never needed this result
+   and it is expensive to calculate on files with complex pre-existing text.
+-  Fixed plugin manager to accept ``Path(plugin)`` as a path to a plugin.
+-  Fixed some typing errors.
+-  Documentation improvements.
+
 v10.2.0
 =======

@@ -10,8 +10,7 @@ build-backend = "setuptools.build_meta"

 [tool.black]
 line-length = 88
-target-version = ["py36",
-"py37", "py38"]
+target-version = ["py36", "py37", "py38"]
 skip-string-normalization = true
 include = '\.pyi?$'
 exclude = '''
@@ -5,7 +5,7 @@ cffi == 1.14.0
 coloredlogs == 14.0  # technically optional
 img2pdf == 0.3.6
 pdfminer.six == 20200517
-pikepdf == 1.15.1
+pikepdf == 1.16.1
 pluggy == 0.13.1
 Pillow == 7.1.2
 reportlab == 3.5.42
@@ -54,6 +54,7 @@ setup(
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: 3.7",
        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
        "Development Status :: 5 - Production/Stable",
        "Environment :: Console",
        "Intended Audience :: End Users/Desktop",
@@ -84,7 +84,7 @@ def exec_progress_pool(
    task_arguments: Optional[Iterable] = None,
    task_finished: Optional[Callable] = None,
 ):
-    log_queue = multiprocessing.Queue(-1)
+    log_queue: multiprocessing.Queue = multiprocessing.Queue(-1)
    listener = threading.Thread(target=log_listener, args=(log_queue,))

    if use_threads:
@@ -25,6 +25,7 @@ from os import fspath
 from pathlib import Path
 from shutil import which
 from subprocess import PIPE, CalledProcessError
+from typing import Optional, cast

 from PIL import Image

@@ -34,12 +35,12 @@ from ocrmypdf.subprocess import get_version, run

 log = logging.getLogger(__name__)

-GS = 'gs'
+_gswin = None
 if os.name == 'nt':
-    GS = which('gswin64c')
-    if not GS:
-        GS = which('gswin32c')
-        if not GS:
+    _gswin = which('gswin64c')
+    if not _gswin:
+        _gswin = which('gswin32c')
+        if not _gswin:
            raise MissingDependencyError(
                """
                ---------------------------------------------------------------------
@@ -52,7 +53,10 @@ if os.name == 'nt':
                ---------------------------------------------------------------------
                """
            )
-    GS = Path(GS).stem
+    _gswin = Path(_gswin).stem
+
+GS = _gswin if _gswin else 'gs'
+del _gswin


 def version():
@@ -77,7 +81,7 @@ def jpeg_passthrough_available() -> bool:


 def _gs_error_reported(stream) -> bool:
-    return re.search(r'error', stream, flags=re.IGNORECASE)
+    return True if re.search(r'error', stream, flags=re.IGNORECASE) else False


 def rasterize_pdf(
@@ -123,7 +123,7 @@ def get_languages():
    return set(lang.strip() for lang in rest)


-def tess_base_args(langs: List[str], engine_mode) -> List[str]:
+def tess_base_args(langs: List[str], engine_mode: int) -> List[str]:
    args = ['tesseract']
    if langs:
        args.extend(['-l', '+'.join(langs)])
@@ -132,7 +132,7 @@ def tess_base_args(langs: List[str], engine_mode) -> List[str]:
    return args


-def get_orientation(input_file: Path, engine_mode, timeout: float):
+def get_orientation(input_file: Path, engine_mode: int, timeout: float):
    args_tesseract = tess_base_args(['osd'], engine_mode) + [
        '--psm',
        '0',
@@ -229,9 +229,9 @@ def generate_hocr(
    input_file: Path,
    output_hocr: Path,
    output_text: Path,
-    languages: list,
-    engine_mode,
-    tessconfig: list,
+    languages: List[str],
+    engine_mode: int,
+    tessconfig: List[str],
    timeout: float,
    pagesegmode: int,
    user_words,
@@ -290,7 +290,7 @@ def generate_pdf(
    output_pdf: Path,
    output_text: Path,
    languages: List[str],
-    engine_mode,
+    engine_mode: int,
    tessconfig: List[str],
    timeout: float,
    pagesegmode: int,
@@ -21,7 +21,7 @@ import importlib.util
 import sys
 from functools import partial
 from pathlib import Path
-from typing import Callable, List, Tuple
+from typing import Callable, List, Tuple, Union

 import pluggy

@@ -64,18 +64,22 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
        )


-def _setup_plugins(pm: pluggy.PluginManager, plugins: List[str], builtins: bool = True):
+def _setup_plugins(
+    pm: pluggy.PluginManager, plugins: List[Union[str, Path]], builtins: bool = True
+):
    pm.add_hookspecs(pluginspec)

+    all_plugins: List[Union[str, Path]] = []
    if builtins:
-        all_plugins = [
-            'ocrmypdf.builtin_plugins.ghostscript',
-            'ocrmypdf.builtin_plugins.tesseract_ocr',
-        ] + plugins
-    else:
-        all_plugins = plugins
+        all_plugins.extend(
+            [
+                'ocrmypdf.builtin_plugins.ghostscript',
+                'ocrmypdf.builtin_plugins.tesseract_ocr',
+            ]
+        )
+    all_plugins.extend(plugins)
    for name in all_plugins:
-        if name.endswith('.py'):
+        if isinstance(name, Path) or name.endswith('.py'):
            # Import by filename
            module_name = Path(name).stem
            spec = importlib.util.spec_from_file_location(module_name, name)
@@ -18,7 +18,6 @@
 import logging
 import os
 import sys
-from argparse import ArgumentParser
 from enum import IntEnum
 from pathlib import Path
 from typing import BinaryIO, Iterable, Union
@@ -27,7 +26,8 @@ from ocrmypdf._logging import PageNumberFilter, TqdmConsole
 from ocrmypdf._plugin_manager import get_plugin_manager
 from ocrmypdf._sync import run_pipeline
 from ocrmypdf._validation import check_options
-from ocrmypdf.cli import get_parser
+from ocrmypdf.cli import ArgumentParser, get_parser
+from ocrmypdf.helpers import is_iterable_notstr

 try:
    import coloredlogs
@@ -153,7 +153,7 @@ def create_options(
                cmdline.append(f"--{cmd_style_arg}")
            continue

-        if isinstance(val, Iterable) and not isinstance(val, str):
+        if is_iterable_notstr(val):
            for elem in val:
                cmdline.append(f"--{cmd_style_arg}")
                cmdline.append(elem)
@@ -35,6 +35,8 @@ log = logging.getLogger(__name__)


 class Resolution(namedtuple('Resolution', ('x', 'y'))):
+    """The number of pixels per inch in each 2D direction."""
+
    __slots__ = ()

    def round(self, ndigits: int):
@@ -113,6 +115,7 @@ def samefile(f1: os.PathLike, f2: os.PathLike):


 def is_iterable_notstr(thing: Any) -> bool:
+    """Is this is an iterable type, other than a string?"""
    return isinstance(thing, Iterable) and not isinstance(thing, str)


@@ -127,6 +130,7 @@ def page_number(input_file: os.PathLike) -> int:


 def available_cpu_count() -> int:
+    """Returns number of CPUs in the system."""
    try:
        return multiprocessing.cpu_count()
    except NotImplementedError:
@@ -173,6 +177,10 @@ def is_file_writable(test_file: os.PathLike) -> bool:


 def check_pdf(input_file: Path) -> bool:
+    """Check if a PDF complies with the PDF specification.
+
+    Checks for proper formatting and proper linearization.
+    """
    pdf = None
    try:
        pdf = pikepdf.open(input_file)
@@ -210,6 +218,7 @@ T = TypeVar('T')


 def clamp(n: T, smallest: T, largest: T) -> T:
+    """Clamps the value of n to between smallest and largest."""
    return max(smallest, min(n, largest))


@@ -35,7 +35,7 @@ from collections import namedtuple
 from itertools import chain
 from math import atan, cos, sin
 from pathlib import Path
-from typing import Union
+from typing import Optional, Tuple, Union
 from xml.etree import ElementTree

 from reportlab.lib.colors import black, cyan, magenta, red
@@ -133,7 +133,7 @@ class HocrTransform:
        return out

    @classmethod
-    def baseline(cls, element):
+    def baseline(cls, element) -> Tuple[float, float]:
        """
        Returns a tuple containing the baseline slope and intercept.
        """
@@ -143,7 +143,7 @@ class HocrTransform:
                return float(matches.group(1)), int(matches.group(2))
        return (0.0, 0.0)

-    def pt_from_pixel(self, pxl):
+    def pt_from_pixel(self, pxl) -> Rect:
        """
        Returns the quantity in PDF units (pt) given quantity in pixels
        """
@@ -156,11 +156,11 @@ class HocrTransform:
        return xpath

    @classmethod
-    def replace_unsupported_chars(cls, s: str):
+    def replace_unsupported_chars(cls, s: str) -> str:
        """
        Given an input string, returns the corresponding string that:
-        - is available in the helvetica facetype
-        - does not contain any ligature (to allow easy search in the PDF file)
+        * is available in the Helvetica facetype
+        * does not contain any ligature (to allow easy search in the PDF file)
        """
        return s.translate(cls.ligatures)

@@ -172,12 +172,12 @@ class HocrTransform:
    def to_pdf(
        self,
        out_filename: Path,
-        image_filename: Path = None,
+        image_filename: Optional[Path] = None,
        show_bounding_boxes: bool = False,
        fontname: str = "Helvetica",
        invisible_text: bool = False,
        interword_spaces: bool = False,
-    ):
+    ) -> None:
        """
        Creates a PDF file with an image superimposed on top of the text.
        Text is positioned according to the bounding box of the lines in
@@ -185,6 +185,20 @@ class HocrTransform:
        The image need not be identical to the image used to create the hOCR
        file.
        It can have a lower resolution, different color mode, etc.
+
+        Arguments:
+            out_filename: Path of PDF to write.
+            image_filename: Image to use for this file. If omitted, the OCR text
+                is shown.
+            show_bounding_boxes: Show bounding boxes around various text regions,
+                for debugging.
+            fontname: Name of font to use.
+            invisible_text: If True, text is rendered invisible so that is
+                selectable but never drawn. If False, text is visible and may
+                be seen if the image is skipped or deleted in Acrobat.
+            interword_spaces: If True, insert spaces between words rather than
+                drawing each word without spaces. Generally this improves text
+                extraction.
        """
        # create the PDF file
        # page size in points (1/72 in.)
@@ -16,24 +16,13 @@
 # along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.

 """
-Generate a PDFMARK file for Ghostscript >= 9.14, for PDF/A conversion
-
-pdfmark is an extension to the Postscript language that describes some PDF
-features like bookmarks and annotations. It was originally specified Adobe
-Distiller, for Postscript to PDF conversion:
-https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/pdfmark_reference.pdf
-
-Ghostscript uses pdfmark for PDF to PDF/A conversion as well. To use Ghostscript
-to create a PDF/A, we need to create a pdfmark file with the necessary metadata.
-
-This takes care of the many version-specific bugs and pecularities in
-Ghostscript's handling of pdfmark.
-
+Utilities for PDF/A production and confirmation with Ghostspcript.
 """

 import base64
 from pathlib import Path
 from string import Template
+from typing import Dict, Union

 import pikepdf
 import pkg_resources
@@ -68,20 +57,28 @@ def
 """


-def generate_pdfa_ps(target_filename, icc='sRGB'):
-    """Create a Postscript pdfmark file for Ghostscript PDF/A conversion
+def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
+    """Create a Postscript PDFMARK file for Ghostscript PDF/A conversion

-    A pdfmark file is a small Postscript program that provides some information
-    Ghostscript needs to perform PDF/A conversion. The only information we put
-    in specifies that we want the file to be a PDF/A, and we want to Ghostscript
-    to convert objects to the sRGB colorspace if it runs into any object that
-    it decides must be converted.
+    pdfmark is an extension to the Postscript language that describes some PDF
+    features like bookmarks and annotations. It was originally specified Adobe
+    Distiller, for Postscript to PDF conversion.

-    See the Adobe pdfmark Reference for details:
-    https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/pdfmark_reference.pdf
+    Ghostscript uses pdfmark for PDF to PDF/A conversion as well. To use Ghostscript
+    to create a PDF/A, we need to create a pdfmark file with the necessary metadata.

-    :param target_filename: filename to save
-    :param icc: ICC identifier such as 'sRGB'
+    This function takes care of the many version-specific bugs and pecularities in
+    Ghostscript's handling of pdfmark.
+
+    The only information we put in specifies that we want the file to be a
+    PDF/A, and we want to Ghostscript to convert objects to the sRGB colorspace
+    if it runs into any object that it decides must be converted.
+
+    Arguments:
+        target_filename: filename to save
+        icc: ICC identifier such as 'sRGB'
+    References:
+        Adobe PDFMARK Reference: https://www.adobe.com/content/dam/acom/en/devnet/acrobat/pdfs/pdfmark_reference.pdf
    """
    if icc == 'sRGB':
        icc_profile = SRGB_ICC_PROFILE
@@ -102,7 +99,7 @@ def generate_pdfa_ps(target_filename, icc='sRGB'):
    return target_filename


-def file_claims_pdfa(filename):
+def file_claims_pdfa(filename: Path):
    """Determines if the file claims to be PDF/A compliant

    This only checks if the XMP metadata contains a PDF/A marker. It does not
@@ -119,7 +116,7 @@ def file_claims_pdfa(filename):
            }
        valid_part_conforms = {'1A', '1B', '2A', '2B', '2U', '3A', '3B', '3U'}
        conformance = f'PDF/A-{pdfmeta.pdfa_status}'
-        pdfa_dict = {}
+        pdfa_dict: Dict[str, Union[str, bool]] = {}
        if pdfmeta.pdfa_status in valid_part_conforms:
            pdfa_dict['pass'] = True
            pdfa_dict['output'] = 'pdfa'
@@ -25,7 +25,7 @@ from functools import partial
 from math import hypot, isclose
 from os import PathLike
 from pathlib import Path
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Optional, Union
 from warnings import warn

 import pikepdf
@@ -237,7 +237,7 @@ def _get_dpi(ctm_shorthand, image_size):
    width-axis vector v0 (1, 0), height-axis vector vh (0, 1) with the matrix,
    which gives the dimensions of the image in PDF units. From there we can
    compare to actual image dimensions. PDF uses
-    row vector * matrix_tranposed unlike the traditional
+    row vector * matrix_transposed unlike the traditional
    matrix * column vector.

    The offset, width and height vectors can be combined in a matrix and
@@ -523,7 +523,7 @@ def _process_content_streams(*, pdf, container, shorthand=None):
    yield from _find_form_xobject_images(pdf, container, contentsinfo)


-def _page_has_text(text_blocks, page_width, page_height):
+def _page_has_text(text_blocks, page_width, page_height) -> bool:
    """Smarter text detection that ignores text in margins"""

    pw, ph = float(page_width), float(page_height)
@@ -567,7 +567,7 @@ def simplify_textboxes(miner, textbox_getter):


 def _pdf_get_pageinfo(
-    pdf, pageno: int, infile: PathLike, check_pages, detailed_analysis
+    pdf, pageno: int, infile: PathLike, check_pages, detailed_analysis: bool
 ):
    pageinfo: Dict[str, Any] = {}
    pageinfo['pageno'] = pageno
@@ -696,41 +696,41 @@ class PageInfo:
        )

    @property
-    def pageno(self):
+    def pageno(self) -> int:
        return self._pageno

    @property
-    def has_text(self):
+    def has_text(self) -> bool:
        return self._pageinfo['has_text']

    @property
-    def has_corrupt_text(self):
+    def has_corrupt_text(self) -> bool:
        if not self._detailed_analysis:
            raise NotImplementedError('Did not do detailed analysis')
        return any(tbox.is_corrupt for tbox in self._pageinfo['textboxes'])

    @property
-    def has_vector(self):
+    def has_vector(self) -> bool:
        return self._pageinfo['has_vector']

    @property
-    def width_inches(self):
+    def width_inches(self) -> Decimal:
        return self._pageinfo['width_inches']

    @property
-    def height_inches(self):
+    def height_inches(self) -> Decimal:
        return self._pageinfo['height_inches']

    @property
-    def width_pixels(self):
+    def width_pixels(self) -> int:
        return int(round(float(self.width_inches) * self.dpi.x))

    @property
-    def height_pixels(self):
+    def height_pixels(self) -> int:
        return int(round(float(self.height_inches) * self.dpi.y))

    @property
-    def rotation(self):
+    def rotation(self) -> int:
        return self._pageinfo.get('rotate', None)

    @rotation.setter
@@ -744,7 +744,9 @@ class PageInfo:
    def images(self):
        return self._pageinfo['images']

-    def get_textareas(self, visible=None, corrupt=None):
+    def get_textareas(
+        self, visible: Optional[bool] = None, corrupt: Optional[bool] = None
+    ):
        def predicate(obj, want_visible, want_corrupt):
            result = True
            if want_visible is not None:
@@ -767,15 +769,15 @@ class PageInfo:
        )

    @property
-    def dpi(self):
+    def dpi(self) -> Resolution:
        return self._pageinfo.get('dpi', Resolution(0.0, 0.0))

    @property
-    def userunit(self):
+    def userunit(self) -> Decimal:
        return self._pageinfo.get('userunit', None)

    @property
-    def min_version(self):
+    def min_version(self) -> str:
        if self.userunit is not None:
            return '1.6'
        else:
@@ -795,9 +797,9 @@ class PdfInfo:
    def __init__(
        self,
        infile,
-        detailed_analysis=False,
-        progbar=False,
-        max_workers=None,
+        detailed_analysis: bool = False,
+        progbar: bool = False,
+        max_workers: int = None,
        check_pages=None,
    ):
        self._infile = infile
@@ -828,29 +830,29 @@ class PdfInfo:
        return self._pages

    @property
-    def min_version(self):
+    def min_version(self) -> str:
        # The minimum PDF is the maximum version that any particular page needs
        return max(page.min_version for page in self.pages)

    @property
-    def has_userunit(self):
+    def has_userunit(self) -> bool:
        return any(page.userunit != 1.0 for page in self.pages)

    @property
-    def has_acroform(self):
+    def has_acroform(self) -> bool:
        return self._has_acroform

    @property
-    def filename(self):
+    def filename(self) -> Union[str, Path]:
        if not isinstance(self._infile, (str, Path)):
            raise NotImplementedError("can't get filename from stream")
        return self._infile

    @property
-    def needs_rendering(self):
+    def needs_rendering(self) -> bool:
        return self._needs_rendering

-    def __getitem__(self, item):
+    def __getitem__(self, item) -> PageInfo:
        return self._pages[item]

    def __len__(self):
@@ -221,7 +221,7 @@ class TextPositionTracker(PDFLayoutAnalyzer):
 def get_page_analysis(infile, pageno, pscript5_mode):
    rman = pdfminer.pdfinterp.PDFResourceManager(caching=True)
    dev = TextPositionTracker(
-        rman, laparams=LAParams(all_texts=True, detect_vertical=True)
+        rman, laparams=LAParams(all_texts=True, detect_vertical=True, boxes_flow=None)
    )
    interp = pdfminer.pdfinterp.PDFPageInterpreter(rman, dev)

@@ -22,13 +22,13 @@ from pathlib import Path
 from typing import TYPE_CHECKING, AbstractSet, List, Optional

 import pluggy
-from PIL import Image

 from ocrmypdf.helpers import Resolution

 if TYPE_CHECKING:
    from ocrmypdf._jobcontext import PageContext
    from ocrmypdf.pdfinfo import PdfInfo
+    from PIL import Image

 hookspec = pluggy.HookspecMarker('ocrmypdf')

@@ -118,7 +118,7 @@ def rasterize_pdf_page(
        rotation: Cardinal angle, clockwise, to rotate page
        filter_vector: If True, remove vector graphics objects
    Returns:
-        output_file
+        Path: output_file if successful
    Note:
        This hook will be called from child processes. Modifying global state
        will not affect the main process or other child processes.
@@ -126,7 +126,7 @@ def rasterize_pdf_page(


@hookspec(firstresult=True)
-def filter_ocr_image(page: 'PageContext', image: Image) -> Image:
+def filter_ocr_image(page: 'PageContext', image: 'Image') -> 'Image':
    """Called to filter the image before it is sent to OCR.

    This is the image that OCR sees, not what the user sees when they view the
@@ -159,20 +159,46 @@ def filter_page_image(page: 'PageContext', image_filename: Path) -> Path:


 OrientationConfidence = namedtuple('OrientationConfidence', ('angle', 'confidence'))
+"""Expresses an OCR engine's confidence in page rotation.
+
+Attributes:
+    angle (int): The clockwise angle (0, 90, 180, 270) that the page should be
+        rotated. 0 means no rotation.
+    confidence (float): How confident the OCR engine is that this the correct
+        rotation. 0 is not confident, 15 is very confident. Arbitrary units.
+"""


 class OcrEngine(ABC):
+    """A class representing an OCR engine with capabilities similar to Tesseract OCR.
+
+    This could be used to create a plugin for another OCR engine instead of
+    Tesseract OCR.
+    """
+
    @abstractstaticmethod
    def version() -> str:
        """Returns the version of the OCR engine."""

    @abstractstaticmethod
    def creator_tag(options: Namespace) -> str:
-        """Returns the creator tag to identify this software's role in creating the PDF."""
+        """Returns the creator tag to identify this software's role in creating the PDF.
+
+        This tag will be inserted in the XMP metadata and DocumentInfo dictionary
+        as appropriate. Ideally you should include the name of the OCR engine and its
+        version. The text should not contain line breaks. This is to help developers
+        like yourself identify the software that produced this file.
+
+        OCRmyPDF will always prepend its name to this value.
+        """

    @abstractmethod
    def __str__(self):
-        """Returns name of OCR engine and version."""
+        """Returns name of OCR engine and version.
+
+        This is used when OCRmyPDF wants to mention the name of the OCR engine
+        to the user, usually in an error message.
+        """

    @abstractstaticmethod
    def languages(options: Namespace) -> AbstractSet[str]:
@@ -248,5 +274,5 @@ def generate_pdfa(
        pdfa_part: The desired PDF/A compliance level, such as ``'2B'``.

    Returns:
-        output_file: If successful, the hook should return ``output_file``.
+        Path: If successful, the hook should return ``output_file``.
    """
@@ -15,23 +15,23 @@
 # You should have received a copy of the GNU General Public License
 # along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.

+"""Utilities to measure OCR quality"""
+
+
 import re
 from typing import Iterable

-"""Utilities to measure OCR quality"""
-

 class OcrQualityDictionary:
    """Manages a dictionary for simple OCR quality checks."""

-    def __init__(self, *, wordlist: Iterable[str] = []):
+    def __init__(self, *, wordlist: Iterable[str]):
        """Construct a dictionary from a list of words.

        Words for which capitalization is important should be capitalized in the
        dictionary. Words that contain spaces or other punctuation will never match.
        """
-        self.dictionary = set()
-        self.dictionary.update(w for w in wordlist)
+        self.dictionary = set(wordlist)

    def measure_words_matched(self, ocr_text: str) -> float:
        """Check how many unique words in the OCR text match a dictionary.
@@ -36,18 +36,12 @@ log = logging.getLogger(__name__)


 def run(args, *, env=None, **kwargs):
-    """Wrapper around subprocess.run()
-
-    The main purpose of this wrapper is to log subprocess output.
-
-    Secondly we have to account for behavioral differences in Windows in particular.
-    Creating symbolic links in Windows requires administrator privileges and
-    may not work if for some reason we're using a FAT file system or the temporary
-    folder is on a different drive from the working folder. The test suite
-    works around this by creating shim Python scripts that perform the same function
-    as a symbolic link, but those shims require support on this side, to ensure
-    we call them with Python.
+    """Wrapper around :py:func:`subprocess.run`

+    The main purpose of this wrapper is to log subprocess output in an orderly
+    fashion that indentifies the responsible subprocess. An additional
+    task is that this function goes to greater lengths to find possible Windows
+    locations of our dependencies when they are not on the system PATH.
    """
    if not env:
        env = os.environ
@@ -106,8 +100,18 @@ def _fix_windows_args(program, args, env):


@lru_cache(maxsize=None)
-def get_version(program, *, version_arg='--version', regex=r'(\d+(\.\d+)*)', env=None):
-    """Get the version of the specified program"""
+def get_version(
+    program: str, *, version_arg: str = '--version', regex=r'(\d+(\.\d+)*)', env=None
+):
+    """Get the version of the specified program
+
+    Arguments:
+        program: The program to version check.
+        version_arg: The argument needed to ask for its version, e.g. ``--version``.
+        regex: A regular expression to parse the program's output and obtain the
+            version.
+        env: Custom ``os.environ`` in which to run program.
+    """
    args_prog = [program, version_arg]
    try:
        proc = run(
@@ -256,7 +256,7 @@ def test_missing_docinfo(resources, outpdf):
        'eng',
        '--skip-text',
        '--plugin',
-        'tests/plugins/tesseract_noop.py',
+        Path('tests/plugins/tesseract_noop.py'),
    )
    assert result == ExitCode.ok
Author	SHA1	Message	Date
James R. Barlow	1722cb579d	v10.2.1 release notes	2020-07-01 03:26:57 -07:00
James R. Barlow	dc42beb6a8	More typing improvements Typing fixes bugs.	2020-06-30 15:02:30 -07:00
James R. Barlow	378f543619	TextPositionTracker: set boxes_flow=None We don't care about the order of lines in our analysis, and this is an expensive calculation in pdfminer.	2020-06-30 04:20:58 -07:00
James R. Barlow	62924ee280	Improve API documentation	2020-06-30 04:20:14 -07:00
James R. Barlow	86a73191b0	Plugin manager: accept Path(plugin)	2020-06-30 04:17:30 -07:00
James R. Barlow	86875997b8	Fix more mypy errors	2020-06-29 02:17:14 -07:00
James R. Barlow	b939584c7a	quality: fixing typing issues	2020-06-29 01:45:45 -07:00
James R. Barlow	bbd174071d	readme: markdown cleanup	2020-06-29 01:45:27 -07:00
James R. Barlow	e5b6fe1317	pyproject.toml: weird line wrapping?	2020-06-29 01:45:12 -07:00
James R. Barlow	f15d9049eb	install: add Mageia Closes #586. Thanks to @yannick56	2020-06-26 23:28:26 -07:00
James R. Barlow	7630c93e5b	install: drop Ubuntu 14.04 steps Bit rot must have set in.	2020-06-26 23:27:42 -07:00
James R. Barlow	638d68aa8a	docs: move Windows ahead of FreeBSD	2020-06-26 22:49:34 -07:00
James R. Barlow	a92dde058a	docs: promote one liner installs, reorg Windows	2020-06-26 22:47:44 -07:00
James R. Barlow	580f2ebb4b	Python 3.9beta is now known to work (Fedora)	2020-06-26 00:06:58 -07:00
James R. Barlow	01cae7a584	docs: Update Fedora versions	2020-06-23 02:08:24 -07:00