v11.6.1 release notes

They're unlikely to be handled well by our recompressors. It seems
that JBIG2 cannot handle very small widths.

Fixes #732

.coveragerc

@@ -1,24 +0,0 @@
-[paths]
-source =
-	src
-	*/site-packages
-
-[run]
-branch = true
-parallel = true
-concurrency =
-	thread
-	multiprocessing
-source =
-	src/ocrmypdf
-
-[report]
-exclude_lines =
-	pragma: no cover
-	def __repr__
-	raise AssertionError
-	raise NotImplementedError
-	if 0:
-	if False:
-	if __name__ == .__main__.:
-	if TYPE_CHECKING:

.github/ISSUE_TEMPLATE/1-general-issues.md

@@ -0,0 +1,32 @@
+---
+name: General issues
+about: Installation, packages, dependencies, "nothing works", test suite failures...
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+What's the problem?
+
+**To Reproduce**
+Steps to reproduce the behavior.
+
+**Expected behavior**
+What did you expected to happen?
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**System (please complete the following information):**
+ - OS:
+ - Python version:
+ - OCRmyPDF version:
+
+**Installation**
+How did you install OCRmyPDF? Did you install it from your operating system's
+package manager, or using pip?
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/2-problem-with-a-specific-input-file.md

@@ -1,6 +1,6 @@
 ---
-name: Bug report
-about: Create a report to help us improve
+name: Problem with a specific input file
+about: Something went wrong while trying to OCR a specific file
 title: ''
 labels: ''
 assignees: ''
@@ -20,13 +20,13 @@ ocrmypdf  ...arguments... input.pdf output.pdf
 Run with verbosity or higher `-v1` to see more detailed logging. This information may be helpful.
 
 **Example file**
-Include an input PDF or image that demonstrates your issue.
+If your issue is a problem that affects only certain files, and we will require an input file (PDF or image) that demonstrates your issue.
 
-Please provide an input file with no personal or confidential information. At your option you may `GPG-encrypt the file <https://github.com/jbarlow83/OCRmyPDF/wiki>` for OCRmyPDF's author only.
+Please provide an input file with no personal or confidential information. At your option you may [GPG-encrypt the file](https://github.com/jbarlow83/OCRmyPDF/wiki) for OCRmyPDF's author only.
 
 Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
 
-(Exceptions: Issues with installation, command line argument parsing, test suite failures.Issues without example files usually cannot be resolved.)
+*(Issues without example files usually cannot be resolved. It's like reporting an issue against a web browser without providing a URL.)*
 
 **Expected behavior**
 A clear and concise description of what you expected to happen.

.github/ISSUE_TEMPLATE/3-feature_request.md

@@ -2,7 +2,7 @@
 name: Feature request
 about: Suggest an idea for this project
 title: ''
-labels: enhancement
+labels: ''
 assignees: ''
 
 ---
@@ -13,5 +13,8 @@ A clear and concise description of what the problem is. Ex. I'm always frustrate
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.
 
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
 **Additional context**
 Add any other context or screenshots about the feature request here.

.github/issue_template.md

@@ -1,33 +0,0 @@
-**Describe the issue**
-A clear and concise description of what the issue is.
-
-**To Reproduce**
-What command line were you trying to run?
-
-```bash
-ocrmypdf  ...arguments... input.pdf output.pdf
-```
-
-**Example file**
-Please include an example *input* PDF (or image). The input file is more helpful.
-
-Please check any or all that apply about the test file:
-
-- [ ] This is the input file
-- [ ] The file contains no personal or confidential information
-- [ ] I am the copyright holder for this file
-- [ ] I permit this file to be included in the OCRmyPDF test suite under the CC-BY-SA 4.0 license
-- [ ] I am not the copyright holder, but this file is available under a free software license
-
-Files that are not free for inclusion in this project are quite welcome, but we like to collect free files for our test suite when possible. Please do *not* submit files with confidential information. At your option you may encrypt files for OCRmyPDF's author only.
-
-**Expected behavior**
-A clear and concise description of what you expected to happen. Include screenshots if applicable.
-
-**System:**
-
-- OS: [e.g. Linux, macOS]
-- OCRmyPDF Version: [e.g. v7.4.0]
-
-**Additional context**
-Add any other context about the problem here.

.gitignore

@@ -7,6 +7,7 @@
 !.gitignore
 !.pre-commit-config.yaml
 !.readthedocs.yml
+!.github/
 
 # Dev scratch
 *.ipynb

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v3.1.0
+    rev: v3.4.0
     hooks:
       - id: check-case-conflict
       - id: check-merge-conflict
@@ -12,12 +12,12 @@ repos:
     hooks:
       - id: seed-isort-config
   - repo: https://github.com/pre-commit/mirrors-isort
-    rev: v5.0.5 # pick the isort version you'd like to use from https://github.com/pre-commit/mirrors-isort/releases
+    rev: v5.7.0 # pick the isort version you'd like to use from https://github.com/pre-commit/mirrors-isort/releases
     hooks:
       - id: isort
   - repo: https://github.com/psf/black
-    rev: 19.10b0
+    rev: 20.8b1
     hooks:
       - id: black
-        language_version: python3.8
+        language_version: python
         exclude: ^src/ocrmypdf/lib/_leptonica.py

README.md

@@ -92,6 +92,9 @@ apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified lan
 
 # Arch Linux users
 pacman -S tesseract-data-eng tesseract-data-deu # Example: Install the English and German language packs
+
+# brew macOS users
+brew install tesseract-lang
 ```
 
 You can then pass the `-l LANG` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple languages can be requested.
@@ -127,11 +130,15 @@ OCRmyPDF would not be the software that it is today without companies and users
 
 ## License
 
-The OCRmyPDF software is licensed under the GNU GPLv3. Certain files are covered by other licenses, as noted in their source files.
+The OCRmyPDF software is licensed under the Mozilla Public License 2.0
+(MPL-2.0). This license permits integration of OCRmyPDF with other code,
+included commercial and closed source, but asks you to publish source-level
+modifications you make to OCRmyPDF.
 
-The license for each test file varies, and is noted in tests/resources/README.rst. The documentation is licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA 4.0).
-
-OCRmyPDF versions prior to 6.0 were distributed under the MIT License.
+Some components of OCRmyPDF have other licenses, as noted in those files and the
+``debian/copyright`` file. Most files in ``misc/`` use the MIT license, and the
+documentation and test files are generally licensed under Creative Commons
+ShareAlike 4.0 (CC-BY-SA 4.0).
 
 ## Disclaimer
 

azure-pipelines.yml

@@ -22,6 +22,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -30,7 +32,7 @@ stages:
               choco install --yes --no-progress --pre tesseract
               choco install --yes --no-progress python3
               choco install --yes --no-progress ghostscript
-              choco install --yes --no-progress pngquant
+              # choco install --yes --no-progress pngquant
             displayName: "Install system packages"
           - pwsh: |
               refreshenv
@@ -59,6 +61,8 @@ stages:
               python.version: "3.7"
             Python38:
               python.version: "3.8"
+            Python39:
+              python.version: "3.9"
         steps:
           - task: UsePythonVersion@0
             inputs:
@@ -149,12 +153,6 @@ stages:
       - job: "macOS_Mojave"
         pool:
           vmImage: "macos-10.14"
-        strategy:
-          matrix:
-            Python37:
-              python.version: ""
-            Python38:
-              python.version: "python@3.8"
         steps:
           # https://github.com/actions/virtual-environments/issues/664
           # - task: UsePythonVersion@0
@@ -162,12 +160,8 @@ stages:
           #     versionSpec: "$(python.version)"
           - bash: |
               brew update
-              brew unlink python@2
-              if [ "$(python.version)" != "" ]; then
-                brew upgrade $(python.version)
-              else
-                echo "Using Python `python3 --version`"
-              fi
+              brew upgrade python
+              echo "Using `python3 --version`"
             displayName: "Update brew and Python"
           - bash: |
               brew install \
@@ -177,8 +171,7 @@ stages:
                 leptonica \
                 openjpeg \
                 pngquant \
-                tesseract \
-                unpaper
+                tesseract
             displayName: "Install system packages"
           - bash: |
               pip3 install --upgrade pip

debian/copyright

@@ -5,15 +5,42 @@ Source: https://github.com/jbarlow83/OCRmyPDF
 
 Files: *
 Copyright:
- (C) 2013-2017 The OCRmyPDF Authors
- (C) 2013-2016, 2015-2017 2016, 2017, 2017-2018, 2018 James R. Barlow
-License: GPL-3+
+ (C) 2013-2015 Julien Pfefferkorn
+ (C) 2015-2020 James R. Barlow
+ (C) 2019 Martin Wind
+License: MPL-2.0
+
+Files: misc/*
+Copyright:
+ (C) 2020 James R. Barlow
+License: Expat
+
+Files: misc/completion/ocrmypdf.bash
+Copyright:
+ (C) 2019 Frank Pille
+ (C) 2020 Alex Willner
+License: Expat
+
+Files: misc/completion/ocrmypdf.fish
+Copyright:
+ (C) 2020 James R. Barlow
+License: Expat
+
+Files: misc/batch.py
+Copyright:
+ (C) 2016 findingorder: https://github.com/findingorder
+License: Expat
+
+Files: misc/synology.py
+Copyright:
+ (C) github.com/Enantiomerie
+License: Expat
 
 Files: misc/watcher.py
 Copyright:
  (C) 2019 Ian Alexander: https://github.com/ianalexander
  (C) 2020 James R. Barlow
-License: GPL-3+
+License: Expat
 
 Files: misc/webservice.py
 Copyright: (C) 2019 James R. Barlow
@@ -33,11 +60,6 @@ Copyright: (C) 2010 Jonathan Brinley <jonathanbrinley@gmail.com>
  (C) 2015-16 James R. Barlow
 License: Expat
 
-Files: src/ocrmypdf/pdfa.py
-Copyright: (C) 2015 James R. Barlow
- (C) 1986-2017 The authors of GhostScript
-License: GPL-3+
-
 Files: src/ocrmypdf/_unicodefun.py
 Copyright: (C) 2014 Armin Ronacher
  (C) 2017 James R. Barlow
@@ -132,6 +154,13 @@ Files: debian/*
 Copyright: (C) 2016 Sean Whitton <spwhitton@spwhitton.name>
 License: GPL-3+
 
+License: MPL-2.0
+ This Source Code Form is subject to the terms of the Mozilla Public
+ License, v. 2.0.
+ .
+ On Debian systems the full text of the MPL-2.0 can be found in
+ /usr/share/common-licenses/MPL-2.0.
+
 License: GPL-3+
  This program is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by

docs/advanced.rst

@@ -125,8 +125,7 @@ include:
 .. envvar:: OMP_THREAD_LIMIT
 
    Controls the number of threads Tesseract will use. OCRmyPDF will
-   manage this environment if it is not already set. (Currently, it will
-   set it to 1 because this gives the best results in testing.)
+   manage this environment variable if it is not already set.
 
 For example, if you have a development build of Tesseract don't wish to
 use the system installation, you can launch OCRmyPDF as follows:
@@ -315,7 +314,7 @@ message is:
 .. code-block:: none
 
     Temporary working files retained at:
-    /tmp/com.github.ocrmypdf.u20wpz07
+    /tmp/ocrmypdf.io.u20wpz07
 
 The organization of this folder is an implementation detail and subject
 to change between releases. However the general organization is that

docs/api.rst

@@ -20,7 +20,8 @@ and largely have the same functions.
 
     import ocrmypdf
 
-    ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
+    if __name__ == '__main__':  # To ensure correct behavior on Windows and macOS
+        ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
 
 With a few exceptions, all of the command line arguments are available
 and may be passed as equivalent keywords.
@@ -35,33 +36,37 @@ The :func:`ocrmypdf.ocr` function runs OCRmyPDF similar to command line
 execution. To do this, it will:
 
 - create a monitoring thread
-- create worker processes (forking itself)
-- manage the signal flags of worker processes
+- create worker processes (on Linux, forking itself; on Windows and macOS, by
+  spawning)
+- manage the signal flags of its worker processes
 - execute other subprocesses (forking and executing other programs)
 
 The Python process that calls ``ocrmypdf.ocr()`` must be sufficiently
-privileged to perform these actions. If it is not, ``ocrmypdf()`` will
-fail.
+privileged to perform these actions.
 
 There is no currently no option to manage how jobs are scheduled other
 than the argument ``jobs=`` which will limit the number of worker
 processes.
 
-Forking a child process to call ``ocrmypdf.ocr()`` is suggested. That
+Creating a child process to call ``ocrmypdf.ocr()`` is suggested. That
 way your application will survive and remain interactive even if
-OCRmyPDF does not.
+OCRmyPDF fails for any reason.
 
 Programs that call ``ocrmypdf.ocr()`` should also install a SIGBUS signal
 handler (except on Windows), to raise an exception if access to a memory
 mapped file fails. OCRmyPDF may use memory mapping.
 
+``ocrmypdf.ocr()`` will take a threading lock to prevent multiple runs of itself
+in the same Python interpreter process. This is not thread-safe, because of how
+OCRmyPDF's plugins and Python's library import system work. If you need to parallelize
+OCRmyPDF, use processes.
+
 .. warning::
 
-    On Windows, the script that calls ``ocrmypdf.ocr()`` must be protected
-    by an "ifmain" guard (``if __name__ == '__main__'``) or you must use
-    ``ocrmypdf.ocr(...use_threads=True)``. If you do not take at least one
-    of these steps, Windows process semantics will prevent OCRmyPDF from working
-    correctly.
+    On Windows and macOS, the script that calls ``ocrmypdf.ocr()`` must be
+    protected by an "ifmain" guard (``if __name__ == '__main__'``). If you do
+    not take at least one of these steps, process semantics will prevent
+    OCRmyPDF from working correctly.
 
 Logging
 -------

docs/apiref.rst

@@ -5,6 +5,15 @@ API Reference
 This page summarizes the rest of the public API. Generally speaking this
 should mainly of interest to plugin developers.
 
+ocrmypdf
+========
+
+.. autoclass:: ocrmypdf.PageContext
+    :members:
+
+.. autoclass:: ocrmypdf.PdfContext
+    :members:
+
 ocrmypdf.exceptions
 ===================
 
@@ -17,6 +26,9 @@ ocrmypdf.helpers
 
 .. automodule:: ocrmypdf.helpers
     :members:
+    :noindex: deprecated
+
+    .. autodecorator:: deprecated
 
 ocrmypdf.hocrtransform
 ======================

docs/batch.rst

@@ -127,7 +127,7 @@ Users may need to customize the script to meet their requirements.
     "OCR_ON_SUCCESS_DELETE", "This will delete the input file if the exit code is 0 (OK)"
     "OCR_OUTPUT_DIRECTORY_YEAR_MONTH", "This will place files in the output in ``{output}/{year}/{month}/{filename}``"
     "OCR_DESKEW", "Apply deskew to crooked input PDFs"
-    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={"rotate_pages": true}'``.
+    "OCR_JSON_SETTINGS", "A JSON string specifying any other arguments for ``ocrmypdf.ocr``, e.g. ``'OCR_JSON_SETTINGS={""rotate_pages"": true}'``."
     "OCR_POLL_NEW_FILE_SECONDS", "Polling interval"
     "OCR_LOGLEVEL", "Level of log messages to report"
 

docs/contributing.rst

@@ -32,7 +32,8 @@ If you are proposing a change that will require a new Python dependency, we
 prefer dependencies that are already packaged by Debian or Red Hat. This makes
 life much easier for our downstream package maintainers.
 
-Python dependencies must also be GPLv3 compatible.
+Python dependencies must also be license-compatible. GPLv3 or AGPLv3 are likely
+incompatible with the project's license, but LGPLv3 is compatible.
 
 New non-Python dependencies
 ===========================
@@ -55,3 +56,12 @@ of that platform.
 
 Packager maintainers, please ensure that the command line completion scripts in
 ``misc/`` are installed.
+
+Copyright and license
+=====================
+
+For contributions over 10 lines of code, please include your name to list of
+copyright holders for that file. The core program is licensed under MPL-2.0,
+test files and documentation under CC-BY-SA 4.0, and miscellaneous files under
+MIT. Please contribute code only that you wrote and you have the permission to
+contribute or license to us.

docs/docker.rst

@@ -1,3 +1,5 @@
+.. _docker:
+
 =====================
 OCRmyPDF Docker image
 =====================
@@ -163,8 +165,7 @@ complete. This may entail setting a long timeout; this interface is more
 useful for internal HTTP API calls.
 
 Unlike the rest of OCRmyPDF, this web service is licensed under the
-Affero GPLv3 (AGPLv3) since Ghostscript, a dependency of OCRmyPDF, is
-also licensed in this way.
+Affero GPLv3 (AGPLv3) since Ghostscript is also licensed in this way.
 
 In addition to the above, please read our
 :ref:`general remarks on using OCRmyPDF as a service <ocr-service>`.

docs/index.rst

@@ -1,7 +1,7 @@
 OCRmyPDF documentation
 ======================
 
-OCRmyPDF adds an optical charcter recognition (OCR) text layer to scanned PDF
+OCRmyPDF adds an optical character recognition (OCR) text layer to scanned PDF
 files, allowing them to be searched.
 
 PDF is the best format for storing and exchanging scanned documents.

docs/installation.rst

@@ -190,7 +190,8 @@ of ocrmypdf, and install the following dependencies:
         python3-reportlab \
         qpdf \
         tesseract-ocr \
-        zlib1g
+        zlib1g \
+        unpaper
 
 We will need a newer version of ``pip`` then was available for Ubuntu 18.04:
 
@@ -494,10 +495,6 @@ Installing on Windows
 Native Windows
 --------------
 
-.. note::
-
-    It is easier to install OCRmyPDF on Windows Subsystem for Linux.
-
 .. note::
 
     Administrator privileges will be required for some of these steps.
@@ -508,31 +505,40 @@ You must install the following for Windows:
 * Tesseract 4.0 or later
 * Ghostscript 9.50 or later
 
-You can install these with the Chocolatey package manager:
+Using the `Chocolatey <https://chocolatey.org/>`_ package manager, install the
+following when running in an Administrator command prompt:
 
 * ``choco install python3``
 * ``choco install --pre tesseract``
 * ``choco install ghostscript``
+* ``choco install pngquant`` (optional)
 
-Also consider adding:
+The commands above will install Python 3.x (latest version), Tesseract, Ghostscript
+and pngquant. Chocolatey may also need to install the Windows Visual C++ Runtime
+DLLs or other Windows patches, and may require a reboot.
 
-* ``choco install pngquant``
-
-Windows 10 64-bit and 64-bit versions of applications are recommended. Earlier
-versions of Windows and 32-bit versions of these programs are not tested, and not
-supported at this time.
-
-OCRmyPDF will check for Tesseract-OCR and Ghostscript in your Program Files folder.
-If they are in some other location, you may need to modify the ``PATH``
-environment variable so Tesseract, Ghostscript, and other any optional executables can
-be found. You can enter it in the command line or
-`follow these directions <https://www.computerhope.com/issues/ch000549.htm#dospath>`_
-to make the change persistent and system-wide.
-
-You may then use pip to install ocrmypdf:
+You may then use ``pip`` to install ocrmypdf. (This can performed by a user or
+Administrator.):
 
 * ``pip install ocrmypdf``
 
+Chocolatey automatically selects appropriate versions of these applications. If you
+are installing them manually, please install 64-bit versions of all applications for
+64-bit Windows, or 32-bit versions of all applications for 32-bit Windows. Mixing
+the "bitness" of these programs will lead to errors.
+
+OCRmyPDF will check the Windows Registry and standard locations in your Program Files
+for third party software it needs (specifically, Tesseract and Ghostscript). To
+override the versions OCRmyPDF selects, you can modify the ``PATH`` environment
+variable. `Follow these directions <https://www.computerhope.com/issues/ch000549.htm#dospath>`_
+to change the PATH.
+
+.. warning::
+
+    As of early 2021, users have reported problems with the Microsoft Store version of
+    Python affected most third party Python packages including OCRmyPDF. Please use
+    Python downloaded from Python.org or Chocolatey as recommended here.
+
 Windows Subsystem for Linux
 ---------------------------
 
@@ -603,7 +609,7 @@ However, the OCR-to-text-layer functionality is available.
 Docker
 ------
 
-You can also :ref:`Install the Docker <docker-install>` container on Windows. Ensure that
+You can also :ref:`Install the Docker <docker>` container on Windows. Ensure that
 your command prompt can run the docker "hello world" container.
 
 Installing on FreeBSD
@@ -629,7 +635,7 @@ Installing the Docker image
 For some users, installing the Docker image will be easier than
 installing all of OCRmyPDF's dependencies.
 
-See `OCRmyPDF Docker Image <docker>`__ for more information.
+See :ref:`docker` for more information.
 
 Installing with Python pip
 ==========================
@@ -637,7 +643,7 @@ Installing with Python pip
 OCRmyPDF is delivered by PyPI because it is a convenient way to install
 the latest version. However, PyPI and ``pip`` cannot address the fact
 that ``ocrmypdf`` depends on certain non-Python system libraries and
-programs being instsalled.
+programs being installed.
 
 For best results, first install `your platform's
 version <https://repology.org/metapackage/ocrmypdf/versions>`__ of

docs/introduction.rst

@@ -186,6 +186,11 @@ Ghostscript also imposes some limitations:
 -  Ghostscript's PDF/A conversion removes any XMP metadata that is not
    one of the standard XMP metadata namespaces for PDFs. In particular,
    PRISM Metdata is removed.
+-  Ghostscript's PDF/A conversion seems to remove or deactivate
+   hyperlinks and other active content.
+
+You can use ``--output-type pdf`` to disable PDF/A conversion and produce
+a standard, non-archival PDF.
 
 Regarding OCRmyPDF itself:
 

docs/pdfsecurity.rst

@@ -64,7 +64,7 @@ malicious user could upload a chosen PDF. In particular, it is not
 necessarily secure against PDF malware or PDFs that cause denial of
 service. OCRmyPDF relies on Ghostscript, and therefore, if deployed
 online one should be prepared to comply with Ghostscript's Affero GPL
-license, OCRmyPDF's GPL license, and any other licenses.
+license, and any other licenses.
 
 Setting aside these concerns, a side effect of OCRmyPDF is it may
 incidentally sanitize PDFs that contain certain types of malware. It

docs/plugins.rst

@@ -65,6 +65,27 @@ similar to ``pytest`` packages such as ``pytest-cov`` (the package) and
     ``ocrmypdf-`` (for the package name on PyPI) and ``ocrmypdf_`` (for the
     module), just like pytest plugins.
 
+Setuptools plugins
+==================
+
+You can also create a plugin that OCRmyPDF will always automatically load if both are
+installed in the same virtual environment, using a setuptools entrypoint.
+
+Your package's ``setup.py`` would need to contain the following, for a plugin
+named ``ocrmypdf-exampleplugin``:
+
+.. code-block:: python
+
+    # sample ./setup.py file
+    from setuptools import setup
+
+    setup(
+        name="ocrmypdf-exampleplugin",
+        packages=["exampleplugin"],
+        # the following makes a plugin available to pytest
+        entry_points={"ocrmypdf": ["exampleplugin = exampleplugin.pluginmodule"]},
+    )
+
 Plugin requirements
 ===================
 

docs/release_notes.rst

@@ -12,9 +12,223 @@ may be unreliable. Use the API to depend on precise behavior.
 The public API may be useful in scripts that launch OCRmyPDF processes or that
 wish to use some of its features for working with PDFs.
 
-Note that it is licensed under GPLv3, so scripts that
-``import ocrmypdf`` and are released publicly should probably also be
-licensed under GPLv3.
+v11.6.1
+=======
+
+-  Fixed an issue with attempting optimize unusually narrow-width images by excluding
+   these images from optimization (#732).
+-  Remove an obsolete compatibility shim for a version of pikepdf that is no longer
+   supported.
+
+v11.6.0
+=======
+
+-  OCRmyPDF will now automatically register plugins from the same virtual environment
+   with an appropriate setuptools entrypoint.
+-  Refactor the plugin manager to remove unnecessary complications and make plugin
+   registration more automatic.
+-  ``PageContext`` and ``PdfContext`` are now formally part of the API, as they
+   should have been, since they were part of ``ocrmypdf.pluginspec``.
+
+v11.5.0
+=======
+
+-  Fixed an issue where the output page size might differ by a fractional amount
+   due to rounding, when ``--force-ocr`` was used and the page contained objects
+   with multiple resolutions.
+-  When determining the resolution at which to rasterize a page, we now consider
+   printed text on the page as requiring a higher resolution. This fixes issues
+   with certain pages being rendered with unacceptably low resolution text, but
+   may increase output file sizes in some workflows where low resolution text
+   is acceptable.
+-  Added a workaround to fix an exception that occurs when trying to
+   ``import ocrmypdf.leptonica`` on Apple ARM silicon (or potentially, other
+   platforms that do not permit write+executable memory).
+
+v11.4.5
+=======
+
+-  Fixed an issue where files may not be closed when the API is used.
+-  Improved ``setup.cfg`` with better settings for test coverage.
+
+v11.4.4
+=======
+
+-  Fixed ``AttributeError: 'NoneType' object has no attribute 'userunit'``, issue #700,
+   related to OCRmyPDF not properly forwarded an error message from pdfminer.six.
+-  Adjusted typing of some arguments.
+-  ``ocrmypdf.ocr`` now takes a ``threading.Lock`` for reasons outlined in the
+   documentation.
+
+v11.4.3
+=======
+
+-  Removed a redundant debug message.
+-  Test suite now asserts that most patched functions are called when they should be.
+-  Test suite now skips a test that fails on two particular versions of piekpdf.
+
+v11.4.2
+=======
+
+-  Fixed support for Cygwin, hopefully.
+-  watcher.py: Fixed an issue with the OCR_LOGLEVEL not being interpreted.
+
+v11.4.1
+=======
+
+-  Fixed an issue where invalid pages ranges passed using the ``pages`` argument,
+   such as "1-0" would cause unhandled exceptions.
+-  Accepted a user-contributed to the Synology demo script in misc/synology.py.
+-  Clarified documentation about change of temporary file location ``ocrmypdf.io``.
+-  Fixed Python wheel tag which was incorrectly set to py35 even though we long
+   since dropped support for Python 3.5.
+
+v11.4.0
+=======
+
+-  When looking for Tesseract and Ghostscript, we now check the Windows Registry to
+   see if their installers registered the location of their executables. This should
+   help Windows users who have installed these programs to non-standard
+   locations.
+-  We now report on the progress of PDF/A conversion, since this operation is
+   sometimes slow.
+-  Improved command line completions.
+-  The prefix of the temporary folder OCRmyPDF creates has been changed from
+   ``com.github.ocrmypdf`` to ``ocrmypdf.io``. Scripts that chose to depend on this
+   prefix may need to be adjusted. (This has always been an implementation detail so is
+   not considered part of the semantic versioning "contract".)
+-  Fixed issue #692, where a particular file with malformed fonts would flood an
+   internal message cue by generating so many debug messages.
+-  Fixed an exception on processing hOCR files with no page record. Tesseract
+   is not known to generate such files.
+
+v11.3.4
+=======
+
+-  Fixed an error message 'called readLinearizationData for file that is not
+   linearized' that may occur when pikepdf 2.1.0 is used. (Upgrading to pikepdf
+   2.1.1 also fixes the issue.)
+-  File watcher now automatically includes ``.PDF`` in addition to ``.pdf`` to
+   better support case sensitive file systems.
+-  Some documentation and comment improvements.
+
+v11.3.3
+=======
+
+-  If unpaper outputs non-UTF-8 data, quietly fix this rather than choke on the
+   conversion. (Possibly addresses #671.)
+
+v11.3.2
+=======
+
+-  Explicitly require pikepdf 2.0.0 or newer when running on Python 3.9. (There are
+   concerns about the stability of pybind11 2.5.x with Python 3.9, which is used in
+   pikepdf 1.x.)
+-  Fixed another issue related to page rotation.
+-  Fixed an issue where image marked as image masks were not properly considered
+   as optimization candidates.
+-  On some systems, unpaper seems to be unable to process the PNGs we offer it
+   as input. We now convert the input to PNM format, which unpaper always accepts.
+   Fixes #665 and #667.
+-  DPI sent to unpaper is now rounded to a more reasonable number of decimal digits.
+-  Debug and error messages from unpaper were being suppressed.
+-  Some documentation tweaks.
+
+v11.3.1
+=======
+
+-  Declare support for new versions: pdfminer.six 20201018 and pikepdf 2.x
+-  Fix warning related to ``--pdfa-image-compression`` that appears at the wrong
+   time.
+
+v11.3.0
+=======
+
+-  The "OCR" step is describing as "Image processing" in the output messages when
+   OCR is disabled, to better explain the application's behavior.
+-  Debug logs are now only created when run as a command line, and not when OCR
+   is performed for an API call. It is the calling application's responsibility
+   to set up logging.
+-  For PDFs with a low number of pages, we gathered information about the input PDF
+   in a thread rather than process (when there are more pages). When run as a
+   thread, we did not close the file handle to the working PDF, leaking one file
+   handle per call of ``ocrmypdf.ocr``.
+-  Fixed an issue where debug messages send by child worker processes did not match
+   the log settings of parent process, causing messages to be dropped. This affected
+   macOS and Windows only where the parent process is not forked.
+-  Fixed the hookspec of rasterize_pdf_page to remove default parameters that
+   were not handled in an expected way by pluggy.
+-  Fixed another issue with automatic page rotation (#658) due to the issue above.
+
+v11.2.1
+=======
+
+-  Fixed an issue where optimization of a 1-bit image with a color palette or
+   associated ICC that was optimized to JBIG2 could have its colors inverted.
+
+v11.2.0
+=======
+
+-  Fixed an issue with optimizing PNG-type images that had soft masks or image masks.
+   This is a regression introduced in (or about) v11.1.0.
+-  Improved type checking of the ``plugins`` parameter for the ``ocrmypdf.ocr``
+   API call.
+
+v11.1.2
+=======
+
+-  Fixed hOCR renderer writing the text in roughly reverse order. This should not
+   affect reasonably smart PDF readers that properly locate the position of all
+   text, but may confuse those that rely on the order of objects in the content
+   stream. (#642)
+
+v11.1.1
+=======
+
+-  We now avoid using named temporary files when using pngquant allowing containerized
+   pngquant installs to be used.
+-  Clarified an error message.
+-  Highest number of 1's in a release ever!
+
+v11.1.0
+=======
+
+-  Fixed page rotation issues: #634, #589.
+-  Fixed some cases where optimization created an invalid image such as a
+   1-bit "RGB" image: #629, #620.
+-  Page numbers are now displayed in debug logs when pages are being grafted.
+-  ocrmypdf.optimize.rewrite_png and ocrmypdf.optimize.rewrite_png_as_g4 were
+   marked deprecated. Strictly speaking these should have been internal APIs,
+   but they were never hidden.
+-  As a precaution, pikepdf mmap-based file access has been disabled due to a
+   rare race condition that causes a crash when certain objects are deallocated.
+   The problem is likely in pikepdf's dependency pybind11.
+-  Extended the example plugin to demonstrate conversion to mono.
+
+v11.0.2
+=======
+
+-  Fixed issue #612, TypeError exception. Fixed by eliminating unnecessary repair of
+   input PDF metadata in memory.
+
+v11.0.1
+=======
+
+-  Blacklist pdfminer.six 20200720, which has a regression fixed in 20200726.
+-  Approve img2pdf 0.4 as it passes tests.
+-  Clarify that the GPL-3 portion of pdfa.py was removed with the changes in v11.0.0;
+   the debian/copyright file did not properly annotate this change.
+
+v11.0.0
+=======
+
+-  Project license changed to Mozilla Public License 2.0. Some miscellaneous
+   code is now under MIT license and non-code content/media remains under
+   CC-BY-SA 4.0. License changed with approval of all people who were found
+   to have contributed to GPLv3 licensed sections of the project. (#600)
+-  Because the license changed, this is being treated as a major version number
+   change; however, there are no known breaking changes in functional behavior
+   or API compared to v10.x.
 
 v10.3.3
 =======
@@ -32,7 +246,7 @@ v10.3.2
 v10.3.1
 =======
 
--  Fixed a number of test suite failures with pdfminer.six older than veresion 20200420.
+-  Fixed a number of test suite failures with pdfminer.six older than veresion 20200402.
 -  Enabled support for pdfminer.six 20200720.
 
 v10.3.0
@@ -375,7 +589,7 @@ v9.0.0
 
 -  Added a high level API for applications that want to integrate OCRmyPDF.
    Special thanks to Martin Wind (@mawi1988) whose made significant contributions
-   to this effort. OCRmyPDF is GPLv3-licensed.
+   to this effort.
 -  Added progress bars for long-running steps. ■■■■■■■□□
 -  We now create linearized ("fast web view") PDFs by default. The new parameter
    ``--fast-web-view`` provides control over when this feature is applied.
@@ -1002,8 +1216,8 @@ v6.1.0
 v6.0.0
 ======
 
--  The software license has been changed to GPLv3. Test resource files
-   and some individual sources may have other licenses.
+-  The software license has been changed to GPLv3 [it has since changed again].
+   Test resource files and some individual sources may have other licenses.
 -  OCRmyPDF now depends on
    `PyMuPDF <https://pymupdf.readthedocs.io/en/latest/installation/>`__.
    Including PyMuPDF is the primary reason for the change to GPLv3.

misc/batch.py

@@ -1,5 +1,23 @@
 #!/usr/bin/env python3
-# Original version by DeliciousPickle@github; modified
+# Copyright 2016 findingorder: https://github.com/findingorder
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
 
 # This script must be edited to meet your needs.
 

misc/completion/ocrmypdf.bash

@@ -1,5 +1,26 @@
 # ocrmypdf completion                                     -*- shell-script -*-
 
+# Copyright 2019 Frank Pille
+# Copyright 2020 Alex Willner
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
 set -o errexit
 
 _ocrmypdf()

misc/completion/ocrmypdf.fish

@@ -1,3 +1,23 @@
+# Copyright 2020 James R. Barlow
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
 complete -c ocrmypdf -x -n '__fish_is_first_arg' -l version
 complete -c ocrmypdf -x -n '__fish_is_first_arg' -s h -s "?" -l help
 
@@ -39,7 +59,8 @@ complete -c ocrmypdf -x -l output-type -a '(__fish_ocrmypdf_output_type)' -d "se
 
 function __fish_ocrmypdf_pdf_renderer
     echo -e "auto\t"(_ "auto select PDF renderer")
-    echo -e "hocr\t"(_ "use hocr renderer")
+    echo -e "hocr\t"(_ "use hOCR renderer")
+    echo -e "hocrdebug\t"(_ "uses hOCR renderer in debug mode, showing recognized text")
     echo -e "sandwich\t"(_ "use sandwich renderer")
 end
 complete -c ocrmypdf -x -l pdf-renderer -a '(__fish_ocrmypdf_pdf_renderer)' -d "select PDF renderer options"
@@ -115,4 +136,4 @@ complete -c ocrmypdf -r -l user-words -d "specify location of user words file"
 complete -c ocrmypdf -r -l user-patterns -d "specify location of user patterns file"
 complete -c ocrmypdf -x -l fast-web-view -d "if file size if above this amount in MB, linearize PDF"
 
-complete -c ocrmypdf -x -a "(__fish_complete_suffix .pdf)"
+complete -c ocrmypdf -x -a "(__fish_complete_suffix .pdf; __fish_complete_suffix .PDF; __fish_complete_suffix .jpg; __fish_complete_suffix .png)"

misc/docker-compose.example.yml

@@ -9,7 +9,7 @@ services:
       - "/media/scan:/input"
       - "/mnt/scan:/output"
     environment:
-      - OCR_OUTPUT_DIRECTORY_YEAR_MONT=0
+      - OCR_OUTPUT_DIRECTORY_YEAR_MONTH=0
     user: "<SET TO YOUR USER ID>:<SET TO YOUR GROUP ID>"
     entrypoint: python3
     command: watcher.py

misc/example_plugin.py

@@ -1,17 +1,41 @@
 # © 2020 James R Barlow: https://github.com/jbarlow83
 #
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
 #
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
 #
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+
+"""
+An example of an OCRmyPDF plugin.
+
+This plugin adds two new command line arguments
+    --grayscale-ocr: converts the image to grayscale before performing OCR on it
+        (This is occasionally useful for images whose color confounds OCR. It only
+        affects the image shown to OCR. The image is not saved.)
+    --mono-page: converts pages all pages in the output file to black and white
+
+To use this from the command line:
+    ocrmypdf --plugin path/to/example_plugin.py --mono-page input.pdf output.pdf
+
+To use this as an API:
+    import ocrmypdf
+    ocrmypdf.ocr('input.pdf', 'output.pdf',
+        plugins=['path/to/example_plugin.py'], mono_page=True
+    )
+"""
 
 import logging
 
@@ -25,6 +49,7 @@ log = logging.getLogger(__name__)
 @hookimpl
 def add_options(parser):
     parser.add_argument('--grayscale-ocr', action='store_true')
+    parser.add_argument('--mono-page', action='store_true')
 
 
 @hookimpl
@@ -47,7 +72,13 @@ def filter_ocr_image(page, image):
 
 @hookimpl
 def filter_page_image(page, image_filename):
-    output = image_filename.with_suffix('.jpg')
-    with Image.open(image_filename) as im:
-        im.save(output)
-    return output
+    if page.options.mono_page:
+        with Image.open(image_filename) as im:
+            im = im.convert('1')
+            im.save(image_filename)
+        return image_filename
+    else:
+        output = image_filename.with_suffix('.jpg')
+        with Image.open(image_filename) as im:
+            im.save(output)
+        return output

misc/synology.py

@@ -1,5 +1,23 @@
 #!/bin/env python3
-# Contributed by github.com/Enantiomerie
+# Copyright 2017 github.com/Enantiomerie
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
 
 # This script must be edited to meet your needs.
 
@@ -61,8 +79,10 @@ for dir_name, subdirs, file_list in os.walk(start_dir):
                 stdout=output_file,
                 stderr=subprocess.PIPE,
                 check=False,
+                text=True,
+                errors='ignore',
             )
-        logging.info(proc.stderr.read())
+        logging.info(proc.stderr)
         os.chmod(full_path_ocr, 0o664)
         os.chmod(full_path, 0o664)
         full_path_ocr_archive = sys.argv[2]

misc/watcher.py

@@ -1,18 +1,23 @@
 # Copyright (C) 2019 Ian Alexander: https://github.com/ianalexander
 # Copyright (C) 2020 James R Barlow: https://github.com/jbarlow83
 #
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
 #
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
 #
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
 
 import json
 import logging
@@ -39,8 +44,8 @@ DESKEW = bool(os.getenv('OCR_DESKEW', ''))
 OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = int(os.getenv('OCR_POLL_NEW_FILE_SECONDS', '1'))
 USE_POLLING = bool(os.getenv('OCR_USE_POLLING', ''))
-LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO').upper()
-PATTERNS = ['*.pdf']
+LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO')
+PATTERNS = ['*.pdf', '*.PDF']
 
 log = logging.getLogger('ocrmypdf-watcher')
 
@@ -112,7 +117,12 @@ class HandleObserverEvent(PatternMatchingEventHandler):
 
 def main():
     ocrmypdf.configure_logging(
-        verbosity=ocrmypdf.Verbosity.default, manage_root_logger=True
+        verbosity=(
+            ocrmypdf.Verbosity.default
+            if LOGLEVEL != 'DEBUG'
+            else ocrmypdf.Verbosity.debug
+        ),
+        manage_root_logger=True,
     )
     log.setLevel(LOGLEVEL)
     log.info(
@@ -130,7 +140,7 @@ def main():
         f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
         f"USE_POLLING: {USE_POLLING}\n"
-        f"LOGLEVEL: {LOGLEVEL}\n"
+        f"LOGLEVEL: {LOGLEVEL}"
     )
 
     if 'input_file' in OCR_JSON_SETTINGS or 'output_file' in OCR_JSON_SETTINGS:

requirements/main.txt

@@ -1,12 +1,12 @@
 # requirements.txt can be used to replicate the developer's build environment
 # setup.py lists a separate set of requirements that are looser to simplify
 # installation
-cffi == 1.14.0
+cffi == 1.14.3
 coloredlogs == 14.0  # technically optional
-img2pdf == 0.3.6
-pdfminer.six == 20200517
-pikepdf == 1.16.1
+img2pdf == 0.4.0
+pdfminer.six == 20201018
+pikepdf == 2.0.0
 pluggy == 0.13.1
-Pillow == 7.1.2
-reportlab == 3.5.42
-tqdm == 4.46.1
+Pillow == 8.0.1
+reportlab == 3.5.55
+tqdm == 4.51.0

setup.cfg

@@ -1,5 +1,5 @@
 [bdist_wheel]
-python-tag = py35
+python-tag = py36
 
 [aliases]
 test=pytest
@@ -15,6 +15,8 @@ filterwarnings =
 	ignore:.*XMLParser.*:DeprecationWarning
 markers =
 	slow
+addopts =
+	-n auto
 
 [isort]
 multi_line_output=3
@@ -27,3 +29,30 @@ known_third_party = PIL,_cffi_backend,cffi,flask,img2pdf,pdfminer,pikepdf,pkg_re
 
 [metadata]
 license_file = LICENSE
+
+[coverage:paths]
+source =
+	src/
+
+[coverage:run]
+branch = true
+parallel = true
+concurrency = multiprocessing
+source =
+	src/ocrmypdf
+
+[coverage:report]
+# Regexes for lines to exclude from consideration
+exclude_lines =
+    # Have to re-enable the standard pragma
+    pragma: no cover
+
+    # Don't complain if tests don't hit defensive assertion code:
+    raise AssertionError
+    raise NotImplementedError
+
+    # Don't complain if non-runnable code isn't run:
+    if 0:
+	if False:
+    if __name__ == .__main__.:
+	if TYPE_CHECKING:

setup.py

@@ -2,20 +2,10 @@
 # -*- coding: utf-8 -*-
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from __future__ import print_function, unicode_literals
 
@@ -60,7 +50,7 @@ setup(
         "Intended Audience :: End Users/Desktop",
         "Intended Audience :: Science/Research",
         "Intended Audience :: System Administrators",
-        "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+        "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",
         "Operating System :: MacOS :: MacOS X",
         "Operating System :: Microsoft :: Windows :: Windows 10",
         "Operating System :: POSIX",
@@ -73,7 +63,6 @@ setup(
     python_requires=' >= 3.6',
     setup_requires=[  # can be removed whenever we can drop pip 9 support
         'cffi >= 1.9.1',  # to build the leptonica module
-        'pytest-runner',  # to enable python setup.py test
         'setuptools_scm',  # so that version will work
         'setuptools_scm_git_archive',  # enable version from github tarballs
     ],
@@ -82,17 +71,18 @@ setup(
     install_requires=[
         'cffi >= 1.9.1',  # must be a setup and install requirement
         'coloredlogs >= 14.0',  # strictly optional
-        'img2pdf >= 0.3.0, < 0.4',  # pure Python, so track HEAD closely
-        'pdfminer.six >= 20191110, <= 20200726',
-        'pikepdf >= 1.14.0, < 2',
+        'img2pdf >= 0.3.0, < 0.5',  # pure Python, so track HEAD closely
+        'pdfminer.six >= 20191110, != 20200720, <= 20201018',
+        "pikepdf >= 1.14.0, < 3 ; python_version < '3.9'",
+        "pikepdf >= 2.0.0 ; python_version >= '3.9'",
         'Pillow >= 7.0.0',
-        'pluggy >= 0.13.0',
+        'pluggy >= 0.13.0, < 1.0',
         'reportlab >= 3.3.0',  # oldest released version with sane image handling
         'tqdm >= 4',
     ],
     tests_require=tests_require,
     entry_points={'console_scripts': ['ocrmypdf = ocrmypdf.__main__:run']},
-    package_data={'ocrmypdf': ['data/sRGB.icc']},
+    package_data={'ocrmypdf': ['data/sRGB.icc', 'py.typed']},
     include_package_data=True,
     zip_safe=False,
     project_urls={

src/ocrmypdf/__init__.py

@@ -1,24 +1,14 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 from pluggy import HookimplMarker as _HookimplMarker
 
 from ocrmypdf import helpers, hocrtransform, leptonica, pdfa, pdfinfo
+from ocrmypdf._jobcontext import PageContext, PdfContext
 from ocrmypdf._version import PROGRAM_NAME, __version__
 from ocrmypdf.api import Verbosity, configure_logging, ocr
 from ocrmypdf.exceptions import (

src/ocrmypdf/__main__.py

@@ -1,20 +1,10 @@
 #!/usr/bin/env python3
 # © 2015-19 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import os

src/ocrmypdf/_concurrent.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import logging.handlers
@@ -22,6 +12,7 @@ import os
 import signal
 import sys
 import threading
+from contextlib import suppress
 from multiprocessing import Pool as ProcessPool
 from multiprocessing.dummy import Pool as ThreadPool
 from typing import Callable, Iterable, Optional
@@ -59,19 +50,20 @@ def process_sigbus(*args):
     raise InputFileError("A worker process lost access to an input file")
 
 
-def process_init(queue, user_init):
+def process_init(queue, user_init, loglevel):
     """Initialize a process pool worker"""
 
     # Ignore SIGINT (our parent process will kill us gracefully)
     signal.signal(signal.SIGINT, signal.SIG_IGN)
 
     # Install SIGBUS handler (so our parent process can abort somewhat gracefully)
-    if hasattr(signal, 'SIGBUS'):
+    with suppress(AttributeError):  # Windows and Cygwin do not have SIGBUS
         signal.signal(signal.SIGBUS, process_sigbus)
 
     # Reconfigure the root logger for this process to send all messages to a queue
     h = logging.handlers.QueueHandler(queue)
     root = logging.getLogger()
+    root.setLevel(loglevel)
     root.handlers = []
     root.addHandler(h)
 
@@ -79,9 +71,10 @@ def process_init(queue, user_init):
         user_init()
 
 
-def thread_init(_queue, user_init):
+def thread_init(_queue, user_init, _loglevel):
     # As a thread, block SIGBUS so the main thread deals with it...
-    if hasattr(signal, 'SIGBUS'):
+    with suppress(AttributeError):
+        # Windows and Cygwin do not have pthread_sigmask or SIGBUS
         signal.pthread_sigmask(signal.SIG_BLOCK, {signal.SIGBUS})
     if user_init:
         user_init()
@@ -112,19 +105,15 @@ def exec_progress_pool(
         pool = pool_class(
             processes=max_workers,
             initializer=initializer,
-            initargs=(log_queue, task_initializer),
+            initargs=(log_queue, task_initializer, logging.getLogger("").level),
         )
         try:
             results = pool.imap_unordered(task, task_arguments)
-            while True:
-                try:
-                    result = results.next()
-                    if task_finished:
-                        task_finished(result, pbar)
-                    else:
-                        pbar.update()
-                except StopIteration:
-                    break
+            for result in results:
+                if task_finished:
+                    task_finished(result, pbar)
+                else:
+                    pbar.update()
         except KeyboardInterrupt:
             # Terminate pool so we exit instantly
             pool.terminate()

src/ocrmypdf/_exec/__init__.py

@@ -1,18 +1,8 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Manage third party executables"""

src/ocrmypdf/_exec/ghostscript.py

@@ -1,19 +1,9 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Interface to Ghostscript executable"""
 
@@ -25,34 +15,34 @@ from os import fspath
 from pathlib import Path
 from shutil import which
 from subprocess import PIPE, CalledProcessError
-from typing import Optional, cast
+from typing import Optional
 
 from PIL import Image
 
 from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
 from ocrmypdf.helpers import Resolution
-from ocrmypdf.subprocess import get_version, run
+from ocrmypdf.subprocess import get_version, run, run_polling_stderr
 
 log = logging.getLogger(__name__)
 
+missing_gs_error = """
+---------------------------------------------------------------------
+This error normally occurs when ocrmypdf find can't Ghostscript.
+Please ensure Ghostscript is installed and its location is added to
+the system PATH environment variable.
+
+For details see:
+    https://ocrmypdf.readthedocs.io/en/latest/installation.html
+---------------------------------------------------------------------
+"""
+
 _gswin = None
 if os.name == 'nt':
     _gswin = which('gswin64c')
     if not _gswin:
         _gswin = which('gswin32c')
         if not _gswin:
-            raise MissingDependencyError(
-                """
-                ---------------------------------------------------------------------
-                This error normally occurs when ocrmypdf can't Ghostscript.  Please
-                ensure Ghostscript is installed and its location is added to the
-                system PATH environment variable.
-
-                For details see:
-                    https://ocrmypdf.readthedocs.io/en/latest/installation.html
-                ---------------------------------------------------------------------
-                """
-            )
+            raise MissingDependencyError(missing_gs_error)
     _gswin = Path(_gswin).stem
 
 GS = _gswin if _gswin else 'gs'
@@ -66,16 +56,14 @@ def version():
 def jpeg_passthrough_available() -> bool:
     """Returns True if the installed version of Ghostscript supports JPEG passthru
 
-    Prior to 9.23, Ghostscript decode and re-encoded JPEGs internally. In 9.23
+    Prior to 9.23, Ghostscript decoded and re-encoded JPEGs internally. In 9.23
     it gained the ability to keep JPEGs unmodified. However, the 9.23
     implementation was buggy and would deletes the last two bytes of images in
     some cases, as reported here.
     https://bugs.ghostscript.com/show_bug.cgi?id=699216
 
     The issue was fixed for 9.24, hence that is the first version we consider
-    the feature available. (However, we don't use 9.24 at all, so the first
-    version that allows JPEG passthrough is 9.25.
-
+    the feature available. (Ghostscript 9.24 has its own problems is blacklisted.)
     """
     return version() >= '9.24'
 
@@ -91,8 +79,8 @@ def rasterize_pdf(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int = 1,
-    page_dpi: Resolution = None,
-    rotation: int = None,
+    page_dpi: Optional[Resolution] = None,
+    rotation: Optional[int] = None,
     filter_vector: bool = False,
 ):
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units."""
@@ -132,8 +120,6 @@ def rasterize_pdf(
         stderr = p.stderr.decode(errors='replace')
         if _gs_error_reported(stderr):
             log.error(stderr)
-        elif stderr:
-            log.debug(stderr)
 
     with Image.open(BytesIO(p.stdout)) as im:
         if rotation is not None:
@@ -151,13 +137,43 @@ def rasterize_pdf(
         im.save(fspath(output_file), dpi=page_dpi)
 
 
+class GhostscriptFollower:
+    re_process = re.compile(r"Processing pages \d+ through (\d+).")
+    re_page = re.compile(r"Page (\d+)")
+
+    def __init__(self, progressbar_class):
+        self.count = 0
+        self.progressbar_class = progressbar_class
+        self.progressbar = None
+
+    def __call__(self, line):
+        if not self.progressbar_class:
+            return
+        if not self.progressbar:
+            m = self.re_process.match(line.strip())
+            if m:
+                self.count = int(m.group(1))
+                self.progressbar = self.progressbar_class(
+                    total=self.count, desc="PDF/A conversion", unit='page'
+                )
+                return
+        else:
+            m = self.re_page.match(line.strip())
+            if m:
+                self.progressbar.update()
+
+
 def generate_pdfa(
     pdf_pages,
     output_file: os.PathLike,
     compression: str,
     pdf_version: str = '1.5',
     pdfa_part: str = '2',
+    progressbar_class=None,
 ):
+    # Ghostscript's compression is all or nothing. We can either force all images
+    # to JPEG, force all to Flate/PNG, or let it decide how to encode the images.
+    # In most case it's best to let it decide.
     compression_args = []
     if compression == 'jpeg':
         compression_args = [
@@ -185,8 +201,9 @@ def generate_pdfa(
     strategy = 'RGB' if version() >= '9.19' else '/RGB'
 
     if version() == '9.23':
-        # 9.23: new feature JPEG passthrough is broken in some cases, best to
-        # disable it always
+        # 9.23: added JPEG passthrough as a new feature, but with a bug that
+        # incorrectly formats some images. Fixed as of 9.24. So we disable this
+        # feature for 9.23.
         # https://bugs.ghostscript.com/show_bug.cgi?id=699216
         compression_args.append('-dPassThroughJPEGImages=false')
 
@@ -196,7 +213,6 @@ def generate_pdfa(
     args_gs = (
         [
             GS,
-            "-dQUIET",
             "-dBATCH",
             "-dNOPAUSE",
             "-dSAFER",
@@ -216,16 +232,28 @@ def generate_pdfa(
         ]
     )
     args_gs.extend(fspath(s) for s in pdf_pages)  # Stringify Path objs
+
     try:
         with Path(output_file).open('wb') as output:
-            p = run(args_gs, stdout=output, stderr=PIPE, check=True)
+            p = run_polling_stderr(
+                args_gs,
+                stdout=output,
+                stderr=PIPE,
+                check=True,
+                text=True,
+                encoding='utf-8',
+                errors='replace',
+                callback=GhostscriptFollower(progressbar_class),
+            )
     except CalledProcessError as e:
         # Ghostscript does not change return code when it fails to create
         # PDF/A - check PDF/A status elsewhere
-        log.error(e.stderr.decode(errors='replace'))
-        raise SubprocessOutputError('Ghostscript PDF/A rendering failed')
+        log.error(e.stderr)
+        raise SubprocessOutputError('Ghostscript PDF/A rendering failed') from e
     else:
-        stderr = p.stderr.decode('utf-8', errors='replace')
+        stderr = p.stderr
+        # If there is an error we log the whole stderr, except for filtering
+        # duplicates.
         if _gs_error_reported(stderr):
             last_part = None
             repcount = 0
@@ -238,11 +266,3 @@ def generate_pdfa(
                 else:
                     repcount += 1
                 last_part = part
-        elif 'overprint mode not set' in stderr:
-            # Unless someone is going to print PDF/A documents on a
-            # magical sRGB printer I can't see the removal of overprinting
-            # being a problem....
-            log.debug(
-                "Ghostscript had to remove PDF 'overprinting' from the "
-                "input file to complete PDF/A conversion. "
-            )

src/ocrmypdf/_exec/jbig2enc.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Interface to jbig2 executable"""
 

src/ocrmypdf/_exec/pngquant.py

@@ -1,24 +1,16 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Interface to pngquant executable"""
 
-from os import fspath
-from tempfile import NamedTemporaryFile
+from contextlib import contextmanager
+from io import BytesIO
+from pathlib import Path
+from subprocess import PIPE
 
 from PIL import Image
 
@@ -38,34 +30,32 @@ def available():
     return True
 
 
-def quantize(input_file, output_file, quality_min, quality_max):
-    input_file = fspath(input_file)
-    output_file = fspath(output_file)
-    if input_file.endswith('.jpg'):
-        with Image.open(input_file) as im, NamedTemporaryFile(suffix='.png') as tmp:
-            im.save(tmp)
-            args = [
-                'pngquant',
-                '--force',
-                '--skip-if-larger',
-                '--output',
-                output_file,
-                '--quality',
-                f'{quality_min}-{quality_max}',
-                '--',
-                tmp.name,
-            ]
-            run(args)
+@contextmanager
+def input_as_png(input_file: Path):
+    if not input_file.name.endswith('.png'):
+        with Image.open(input_file) as im:
+            bio = BytesIO()
+            im.save(bio, format='png')
+            bio.seek(0)
+            yield bio
     else:
+        with open(input_file, 'rb') as f:
+            yield f
+
+
+def quantize(input_file: Path, output_file: Path, quality_min: int, quality_max: int):
+    with input_as_png(input_file) as input_stream:
         args = [
             'pngquant',
             '--force',
             '--skip-if-larger',
-            '--output',
-            output_file,
             '--quality',
             f'{quality_min}-{quality_max}',
-            '--',
-            input_file,
+            '--',  # pngquant: stop processing arguments
+            '-',  # pngquant: stream input and output
         ]
-        run(args)
+        result = run(args, stdin=input_stream, stdout=PIPE, stderr=PIPE, check=False)
+
+    if result.returncode == 0:
+        # input_file could be the same as output_file, so we defer the write
+        output_file.write_bytes(result.stdout)

src/ocrmypdf/_exec/tesseract.py

@@ -1,19 +1,9 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Interface to Tesseract executable"""
 
@@ -24,7 +14,7 @@ from collections import namedtuple
 from os import fspath
 from pathlib import Path
 from subprocess import PIPE, STDOUT, CalledProcessError, TimeoutExpired
-from typing import List
+from typing import List, Optional
 
 from PIL import Image
 
@@ -110,7 +100,12 @@ def get_languages():
     args_tess = ['tesseract', '--list-langs']
     try:
         proc = run(
-            args_tess, universal_newlines=True, stdout=PIPE, stderr=STDOUT, check=True
+            args_tess,
+            text=True,
+            stdout=PIPE,
+            stderr=STDOUT,
+            logs_errors_to_stdout=True,
+            check=True,
         )
         output = proc.stdout
     except CalledProcessError as e:
@@ -123,7 +118,7 @@ def get_languages():
     return set(lang.strip() for lang in rest)
 
 
-def tess_base_args(langs: List[str], engine_mode: int) -> List[str]:
+def tess_base_args(langs: List[str], engine_mode: Optional[int]) -> List[str]:
     args = ['tesseract']
     if langs:
         args.extend(['-l', '+'.join(langs)])
@@ -132,7 +127,7 @@ def tess_base_args(langs: List[str], engine_mode: int) -> List[str]:
     return args
 
 
-def get_orientation(input_file: Path, engine_mode: int, timeout: float):
+def get_orientation(input_file: Path, engine_mode: Optional[int], timeout: float):
     args_tesseract = tess_base_args(['osd'], engine_mode) + [
         '--psm',
         '0',

src/ocrmypdf/_exec/unpaper.py

@@ -1,19 +1,9 @@
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 # unpaper documentation:
 # https://github.com/Flameeyes/unpaper/blob/master/doc/basic-concepts.md
@@ -23,10 +13,11 @@
 import logging
 import os
 import shlex
+from decimal import Decimal
 from pathlib import Path
-from subprocess import PIPE, STDOUT, CalledProcessError
+from subprocess import PIPE, STDOUT
 from tempfile import TemporaryDirectory
-from typing import Tuple
+from typing import List, Optional, Tuple, Union
 
 from PIL import Image
 
@@ -34,10 +25,12 @@ from ocrmypdf.exceptions import MissingDependencyError, SubprocessOutputError
 from ocrmypdf.subprocess import get_version
 from ocrmypdf.subprocess import run as external_run
 
+DecFloat = Union[Decimal, float]
+
 log = logging.getLogger(__name__)
 
 
-def version():
+def version() -> str:
     return get_version('unpaper')
 
 
@@ -63,23 +56,25 @@ def _setup_unpaper_io(tmpdir: Path, input_file: Path) -> Tuple[Path, Path]:
         except KeyError:
             raise MissingDependencyError(
                 "Failed to convert image to a supported format."
-            ) from e
+            ) from None
 
-        if im_modified or input_file.suffix != '.png':
-            input_png = tmpdir / 'input.png'
-            im.save(input_png, format='PNG', compress_level=1)
+        if im_modified or input_file.suffix != '.pnm':
+            input_pnm = tmpdir / 'input.pnm'
+            im.save(input_pnm, format='PPM')
         else:
             # No changes, PNG input, just use the file we already have
-            input_png = input_file
+            input_pnm = input_file
         output_pnm = tmpdir / f'output{suffix}'
-    return input_png, output_pnm
+    return input_pnm, output_pnm
 
 
-def run(input_file, output_file, dpi, mode_args):
-    args_unpaper = ['unpaper', '-v', '--dpi', str(dpi)] + mode_args
+def run(
+    input_file: Path, output_file: Path, dpi: DecFloat, mode_args: List[str]
+) -> None:
+    args_unpaper = ['unpaper', '-v', '--dpi', str(round(dpi, 6))] + mode_args
 
     with TemporaryDirectory() as tmpdir:
-        input_png, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
+        input_pnm, output_pnm = _setup_unpaper_io(Path(tmpdir), input_file)
 
         # To prevent any shenanigans from accepting arbitrary parameters in
         # --unpaper-args, we:
@@ -88,41 +83,40 @@ def run(input_file, output_file, dpi, mode_args):
         # 3) append absolute paths for the input and output file
         # This should ensure that a user cannot clobber some other file with
         # their unpaper arguments (whether intentionally or otherwise)
-        args_unpaper.extend([os.fspath(input_png), os.fspath(output_pnm)])
+        args_unpaper.extend([os.fspath(input_pnm), os.fspath(output_pnm)])
+        external_run(
+            args_unpaper,
+            close_fds=True,
+            check=True,
+            stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
+            stdout=PIPE,  # and cannot send file output to stdout
+            cwd=tmpdir,
+            logs_errors_to_stdout=True,
+        )
         try:
-            proc = external_run(
-                args_unpaper,
-                check=True,
-                close_fds=True,
-                universal_newlines=True,
-                stderr=STDOUT,  # unpaper writes logging output to stdout and stderr
-                cwd=tmpdir,  # and cannot send file output to stdout
-                stdout=PIPE,
-            )
-        except CalledProcessError as e:
-            log.debug(e.stderr)
-            raise e from e
-        else:
-            log.debug(proc.stderr)
-            try:
-                with Image.open(output_pnm) as imout:
-                    imout.save(output_file, dpi=(dpi, dpi))
-            except (FileNotFoundError, OSError):
-                raise SubprocessOutputError(
-                    "unpaper: failed to produce the expected output file. "
-                    + " Called with: "
-                    + str(args_unpaper)
-                ) from None
+            with Image.open(output_pnm) as imout:
+                imout.save(output_file, dpi=(dpi, dpi))
+        except (FileNotFoundError, OSError):
+            raise SubprocessOutputError(
+                "unpaper: failed to produce the expected output file. "
+                + " Called with: "
+                + str(args_unpaper)
+            ) from None
 
 
-def validate_custom_args(args: str):
+def validate_custom_args(args: str) -> List[str]:
     unpaper_args = shlex.split(args)
-    if any('/' in arg for arg in unpaper_args):
+    if any(('/' in arg or arg == '.' or arg == '..') for arg in unpaper_args):
         raise ValueError('No filenames allowed in --unpaper-args')
     return unpaper_args
 
 
-def clean(input_file, output_file, dpi, unpaper_args=None):
+def clean(
+    input_file: Path,
+    output_file: Path,
+    dpi: DecFloat,
+    unpaper_args: Optional[List[str]] = None,
+):
     default_args = [
         '--layout',
         'none',

src/ocrmypdf/_graft.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 from contextlib import suppress
@@ -135,34 +125,38 @@ class OcrGrafter:
                 del self.pdf_base.pages[-1]
             emplaced_page = True
 
+        # Calculate if the text is misaligned compared to the content
         if emplaced_page:
             content_rotation = autorotate_correction
         text_rotation = autorotate_correction
         text_misaligned = (text_rotation - content_rotation) % 360
         log.debug(
-            f"Rotations for page {pageno}: [text, auto, misalign, content] = "
-            f"{text_rotation}, {autorotate_correction}, "
-            f"{text_misaligned}, {content_rotation}"
+            f"Text rotation: (text, autorotate, content) -> text misalignment = "
+            f"({text_rotation}, {autorotate_correction}, {content_rotation}) -> {text_misaligned}"
         )
 
         if textpdf and self.font:
-            # Graft the text layer onto this page, whether new or old
+            # Graft the text layer onto this page, whether new or old, possibly
+            # rotating the text layer by the amount is misaligned.
             strip_old = self.context.options.redo_ocr
             self._graft_text_layer(
                 page_num=pageno + 1,
                 textpdf=textpdf,
                 font=self.font,
                 font_key=self.font_key,
-                rotation=text_misaligned,
+                text_rotation=text_misaligned,
                 procset=self.procset,
                 strip_old_text=strip_old,
             )
 
-        # Correct the rotation if applicable
-        self.pdf_base.pages[pageno].Rotate = (
-            content_rotation - autorotate_correction
-        ) % 360
-
+        # Correct the overall page rotation if needed, now that the text and content
+        # are aligned
+        page_rotation = (content_rotation - autorotate_correction) % 360
+        self.pdf_base.pages[pageno].Rotate = page_rotation
+        log.debug(
+            f"Page rotation: (content, auto) -> page = "
+            f"({content_rotation}, {autorotate_correction}) -> {page_rotation}"
+        )
         if self.emplacements % MAX_REPLACE_PAGES == 0:
             self.save_and_reload()
 
@@ -236,7 +230,7 @@ class OcrGrafter:
         font: pikepdf.Object,
         font_key: pikepdf.Object,
         procset: pikepdf.Object,
-        rotation: int,
+        text_rotation: int,
         strip_old_text: bool,
     ):
         """Insert the text layer from text page 0 on to pdf_base at page_num"""
@@ -266,13 +260,13 @@ class OcrGrafter:
             corner = pikepdf.PdfMatrix().translated(mediabox[0], mediabox[1])
             # -rotation because the input is a clockwise angle and this formula
             # uses CCW
-            rotation = -rotation % 360
-            rotate = pikepdf.PdfMatrix().rotated(rotation)
+            text_rotation = -text_rotation % 360
+            rotate = pikepdf.PdfMatrix().rotated(text_rotation)
 
             # Because of rounding of DPI, we might get a text layer that is not
             # identically sized to the target page. Scale to adjust. Normally this
             # is within 0.998.
-            if rotation in (90, 270):
+            if text_rotation in (90, 270):
                 wt, ht = ht, wt
             scale_x = wp / wt
             scale_y = hp / ht

src/ocrmypdf/_jobcontext.py

@@ -1,34 +1,31 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import os
 import shutil
 import sys
 from argparse import Namespace
 from copy import copy
-from io import IOBase
 from pathlib import Path
 from typing import Iterator
 
+from pluggy import PluginManager
+
 from ocrmypdf.pdfinfo import PdfInfo
+from ocrmypdf.pdfinfo.info import PageInfo
 
 
 class PdfContext:
-    """Holds our context for a particular run of the pipeline"""
+    """Holds the context for a particular run of the pipeline."""
+
+    options: Namespace  #: The specified options for processing this PDF.
+    origin: Path  #: The filename of the original input file.
+    pdfinfo: PdfInfo  #: Detailed data for this PDF.
+    plugin_manager: PluginManager  #: PluginManager for processing the current PDF.
 
     def __init__(
         self,
@@ -45,21 +42,33 @@ class PdfContext:
         self.plugin_manager = plugin_manager
 
     def get_path(self, name: str) -> Path:
+        """Generate a ``Path`` for an intermediate file involved in processing.
+
+        The path will be in a temporary folder that is common for all processing
+        of this particular PDF.
+        """
         return self.work_folder / name
 
     def get_page_contexts(self) -> Iterator['PageContext']:
+        """Get all ``PageContext`` for this PDF."""
         npages = len(self.pdfinfo)
         for n in range(npages):
             yield PageContext(self, n)
 
 
 class PageContext:
-    """Holds our context for a page
+    """Holds our context for a page.
 
     Must be pickable, so stores only intrinsic/simple data elements or those
-    capable of their serializing themselves via __getstate__.
+    capable of their serializing themselves via ``__getstate__``.
     """
 
+    options: Namespace  #: The specified options for processing this PDF.
+    origin: Path  #: The filename of the original input file.
+    pageno: int  #: This page number (zero-based).
+    pageinfo: PageInfo  #: Information on this page.
+    plugin_manager: PluginManager  #: PluginManager for processing the current PDF.
+
     def __init__(self, pdf_context: PdfContext, pageno):
         self.work_folder = pdf_context.work_folder
         self.origin = pdf_context.origin
@@ -69,6 +78,11 @@ class PageContext:
         self.plugin_manager = pdf_context.plugin_manager
 
     def get_path(self, name: str) -> Path:
+        """Generate a ``Path`` for a file that is part of processing this page.
+
+        The path will be based in a common temporary folder and have a prefix based
+        on the page number.
+        """
         return self.work_folder / ("%06d_%s" % (self.pageno + 1, name))
 
     def __getstate__(self):

src/ocrmypdf/_logging.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import sys

src/ocrmypdf/_pipeline.py

@@ -1,19 +1,9 @@
 # © 2016 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import os
@@ -29,6 +19,7 @@ import img2pdf
 import pikepdf
 from pikepdf.models.metadata import encode_pdf_date
 from PIL import Image, ImageColor, ImageDraw
+from tqdm import tqdm
 
 from ocrmypdf import leptonica
 from ocrmypdf._exec import unpaper
@@ -159,7 +150,7 @@ def get_pdfinfo(
     progbar=False,
     max_workers=None,
     check_pages=None,
-):
+) -> PdfInfo:
     try:
         return PdfInfo(
             input_file,
@@ -215,17 +206,21 @@ def validate_pdfinfo_options(context: PdfContext):
     context.plugin_manager.hook.validate(pdfinfo=pdfinfo, options=options)
 
 
+def _vector_page_dpi(pageinfo):
+    return VECTOR_PAGE_DPI if pageinfo.has_vector or pageinfo.has_text else 0.0
+
+
 def get_page_dpi(pageinfo, options):
     "Get the DPI when nonsquare DPI is tolerable"
     xres = max(
         pageinfo.dpi.x or VECTOR_PAGE_DPI,
         options.oversample or 0.0,
-        VECTOR_PAGE_DPI if pageinfo.has_vector else 0.0,
+        _vector_page_dpi(pageinfo),
     )
     yres = max(
         pageinfo.dpi.y or VECTOR_PAGE_DPI,
         options.oversample or 0,
-        VECTOR_PAGE_DPI if pageinfo.has_vector else 0.0,
+        _vector_page_dpi(pageinfo),
     )
     return Resolution(float(xres), float(yres))
 
@@ -239,7 +234,7 @@ def get_page_square_dpi(pageinfo, options) -> Resolution:
         max(
             (xres * userunit) or VECTOR_PAGE_DPI,
             (yres * userunit) or VECTOR_PAGE_DPI,
-            VECTOR_PAGE_DPI if pageinfo.has_vector else 0.0,
+            _vector_page_dpi(pageinfo),
             options.oversample or 0.0,
         )
     )
@@ -252,7 +247,7 @@ def get_canvas_square_dpi(pageinfo, options) -> Resolution:
         max(
             (pageinfo.dpi.x) or VECTOR_PAGE_DPI,
             (pageinfo.dpi.y) or VECTOR_PAGE_DPI,
-            VECTOR_PAGE_DPI if pageinfo.has_vector else 0.0,
+            _vector_page_dpi(pageinfo),
             options.oversample or 0.0,
         )
     )
@@ -342,8 +337,10 @@ def rasterize_preview(input_file: Path, page_context: PageContext):
         output_file=output_file,
         raster_device='jpeggray',
         raster_dpi=canvas_dpi,
-        page_dpi=page_dpi,
         pageno=page_context.pageinfo.pageno + 1,
+        page_dpi=page_dpi,
+        rotation=0,
+        filter_vector=False,
     )
     return output_file
 
@@ -443,7 +440,7 @@ def rasterize(
 
     device = colorspaces[device_idx]
 
-    log.debug(f"Rasterize with {device}")
+    log.debug(f"Rasterize with {device}, rotation {correction}")
 
     # Produce the page image with square resolution or else deskew and OCR
     # will not work properly.
@@ -595,12 +592,17 @@ def create_pdf_page_from_image(image: Path, page_context: PageContext):
     # except that the hocr renderer does not understand non-square DPI. The
     # sandwich renderer would be fine.
     output_file = page_context.get_path('visible.pdf')
-    dpi = get_page_square_dpi(page_context.pageinfo, page_context.options)
-    layout_fun = img2pdf.get_fixed_dpi_layout_fun(dpi)
+
+    pageinfo = page_context.pageinfo
+    pagesize = 72.0 * float(pageinfo.width_inches), 72.0 * float(pageinfo.height_inches)
+    if pageinfo.rotation % 180 == 90:
+        pagesize = pagesize[1], pagesize[0]
 
     # This create a single page PDF
     with open(image, 'rb') as imfile, open(output_file, 'wb') as pdf:
         log.debug('convert')
+
+        layout_fun = img2pdf.get_layout_fun(pagesize)
         img2pdf.convert(
             imfile, with_pdfrw=False, layout_fun=layout_fun, outputstream=pdf
         )
@@ -609,14 +611,17 @@ def create_pdf_page_from_image(image: Path, page_context: PageContext):
 
 
 def render_hocr_page(hocr: Path, page_context: PageContext):
+    options = page_context.options
     output_file = page_context.get_path('ocr_hocr.pdf')
-    dpi = get_page_square_dpi(page_context.pageinfo, page_context.options)
+    dpi = get_page_square_dpi(page_context.pageinfo, options)
+    debug_mode = options.pdf_renderer == 'hocrdebug'
+
     hocrtransform = HocrTransform(hocr, dpi.x)  # square
     hocrtransform.to_pdf(
         output_file,
         image_filename=None,
-        show_bounding_boxes=False,
-        invisible_text=True,
+        show_bounding_boxes=False if not debug_mode else True,
+        invisible_text=True if not debug_mode else False,
         interword_spaces=True,
     )
     return output_file
@@ -717,6 +722,7 @@ def convert_to_pdfa(input_pdf: Path, input_ps_stub: Path, context: PdfContext):
         output_file=output_file,
         compression=options.pdfa_image_compression,
         pdfa_part=options.output_type[-1],  # is pdfa-1, pdfa-2, or pdfa-3
+        progressbar_class=tqdm if options.progress_bar else None,
     )
 
     return output_file
@@ -759,17 +765,17 @@ def metadata_fixup(working_file: Path, context: PdfContext):
             if 'xmp:CreateDate' not in meta:
                 meta['xmp:CreateDate'] = meta.get('xmp:ModifyDate', '')
 
-            # Ghostscript likes to set title to Untitled if omitted from input.
-            # Reverse this, because PDF/A TechNote 0003:Metadata in PDF/A-1
-            # and the XMP Spec do not make this recommendation.
-            if meta.get('dc:title') == 'Untitled':
-                with original.open_metadata() as original_meta:
-                    if 'dc:title' not in original_meta:
+            with original.open_metadata(
+                set_pikepdf_as_editor=False, update_docinfo=False, strict=False
+            ) as meta_original:
+                if meta.get('dc:title') == 'Untitled':
+                    # Ghostscript likes to set title to Untitled if omitted from input.
+                    # Reverse this, because PDF/A TechNote 0003:Metadata in PDF/A-1
+                    # and the XMP Spec do not make this recommendation.
+                    if 'dc:title' not in meta_original:
                         del meta['dc:title']
-
-            meta_original = original.open_metadata()
-            missing = set(meta_original.keys()) - set(meta.keys())
-            report_on_metadata(missing)
+                missing = set(meta_original.keys()) - set(meta.keys())
+                report_on_metadata(missing)
 
         pdf.save(
             output_file,

src/ocrmypdf/_plugin_manager.py

@@ -1,30 +1,21 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import argparse
 import importlib
 import importlib.util
+import pkgutil
 import sys
-from functools import partial
 from pathlib import Path
-from typing import Callable, List, Tuple, Union
+from typing import List, Tuple, Union
 
 import pluggy
 
+import ocrmypdf.builtin_plugins
 from ocrmypdf import pluginspec
 from ocrmypdf.cli import get_parser, plugins_only_parser
 
@@ -40,62 +31,69 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
     """
 
     def __init__(
-        self, *args, setup_func: Callable[[pluggy.PluginManager], None], **kwargs
+        self,
+        *args,
+        plugins: List[Union[str, Path]],
+        builtins: bool = True,
+        **kwargs,
     ):
-        self._init_args = args
-        self._setup_func = setup_func
-        self._init_kwargs = kwargs
+        self.__init_args = args
+        self.__init_kwargs = kwargs
+        self.__plugins = plugins
+        self.__builtins = builtins
         super().__init__(*args, **kwargs)
-        setup_func(self)
+        self.setup_plugins()
 
     def __getstate__(self):
         state = dict(
-            _init_args=self._init_args,
-            _setup_func=self._setup_func,
-            _init_kwargs=self._init_kwargs,
+            init_args=self.__init_args,
+            plugins=self.__plugins,
+            builtins=self.__builtins,
+            init_kwargs=self.__init_kwargs,
         )
         return state
 
     def __setstate__(self, state):
         self.__init__(
-            *state['_init_args'],
-            setup_func=state['_setup_func'],
-            **state['_init_kwargs'],
+            *state['init_args'],
+            plugins=state['plugins'],
+            builtins=state['builtins'],
+            **state['init_kwargs'],
         )
 
+    def setup_plugins(self):
+        self.add_hookspecs(pluginspec)
 
-def _setup_plugins(
-    pm: pluggy.PluginManager, plugins: List[Union[str, Path]], builtins: bool = True
-):
-    pm.add_hookspecs(pluginspec)
+        # 1. Register builtins
+        if self.__builtins:
+            for module in pkgutil.iter_modules(ocrmypdf.builtin_plugins.__path__):
+                name = f'ocrmypdf.builtin_plugins.{module.name}'
+                module = importlib.import_module(name)
+                self.register(module)
 
-    all_plugins: List[Union[str, Path]] = []
-    if builtins:
-        all_plugins.extend(
-            [
-                'ocrmypdf.builtin_plugins.ghostscript',
-                'ocrmypdf.builtin_plugins.tesseract_ocr',
-            ]
-        )
-    all_plugins.extend(plugins)
-    for name in all_plugins:
-        if isinstance(name, Path) or name.endswith('.py'):
-            # Import by filename
-            module_name = Path(name).stem
-            spec = importlib.util.spec_from_file_location(module_name, name)
-            module = importlib.util.module_from_spec(spec)
-            sys.modules[module_name] = module
-            spec.loader.exec_module(module)
-        else:
-            # Import by dotted module name
-            module = importlib.import_module(name)
-        pm.register(module)
+        # 2. Register setuptools plugins
+        self.load_setuptools_entrypoints('ocrmypdf')
+
+        # 3. Register plugins specified on command line
+        for name in self.__plugins:
+            if isinstance(name, Path) or name.endswith('.py'):
+                # Import by filename
+                module_name = Path(name).stem
+                spec = importlib.util.spec_from_file_location(module_name, name)
+                module = importlib.util.module_from_spec(spec)
+                sys.modules[module_name] = module
+                spec.loader.exec_module(module)
+            else:
+                # Import by dotted module name
+                module = importlib.import_module(name)
+            self.register(module)
 
 
-def get_plugin_manager(plugins: List[str], builtins=True):
+def get_plugin_manager(plugins: List[Union[str, Path]], builtins=True):
     pm = OcrmypdfPluginManager(
         project_name='ocrmypdf',
-        setup_func=partial(_setup_plugins, plugins=plugins, builtins=builtins),
+        plugins=plugins,
+        builtins=builtins,
     )
     return pm
 

src/ocrmypdf/_sync.py

@@ -1,19 +1,9 @@
 # © 2016 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import logging.handlers
@@ -65,6 +55,7 @@ from ocrmypdf._validation import (
 )
 from ocrmypdf.exceptions import ExitCode, ExitCodeException
 from ocrmypdf.helpers import (
+    NeverRaise,
     available_cpu_count,
     check_pdf,
     pikepdf_enable_mmap,
@@ -216,12 +207,13 @@ def exec_page_sync(page_context: PageContext):
             visible_image_out, page_context
         )
 
-    if options.pdf_renderer == 'hocr':
+    if options.pdf_renderer.startswith('hocr'):
         (hocr_out, text_out) = ocr_engine_hocr(ocr_image_out, page_context)
         ocr_out = render_hocr_page(hocr_out, page_context)
-
-    if options.pdf_renderer == 'sandwich':
+    elif options.pdf_renderer == 'sandwich':
         (ocr_out, text_out) = ocr_engine_textonly_pdf(ocr_image_out, page_context)
+    else:
+        raise NotImplementedError(f"pdf_renderer {options.pdf_renderer}")
 
     return PageResult(
         pageno=page_context.pageno,
@@ -254,7 +246,8 @@ def exec_concurrent(context: PdfContext):
     """Execute the pipeline concurrently"""
 
     # Run exec_page_sync on every page context
-    max_workers = min(len(context.pdfinfo), context.options.jobs)
+    options = context.options
+    max_workers = min(len(context.pdfinfo), options.jobs)
     if max_workers > 1:
         log.info("Start processing %d pages concurrently", max_workers)
 
@@ -262,25 +255,29 @@ def exec_concurrent(context: PdfContext):
     ocrgraft = OcrGrafter(context)
 
     def update_page(result: PageResult, pbar):
-        sidecars[result.pageno] = result.text
-        pbar.update()
-        ocrgraft.graft_page(
-            pageno=result.pageno,
-            image=result.pdf_page_from_image,
-            textpdf=result.ocr,
-            autorotate_correction=result.orientation_correction,
-        )
-        pbar.update()
+        try:
+            tls.pageno = result.pageno + 1
+            sidecars[result.pageno] = result.text
+            pbar.update()
+            ocrgraft.graft_page(
+                pageno=result.pageno,
+                image=result.pdf_page_from_image,
+                textpdf=result.ocr,
+                autorotate_correction=result.orientation_correction,
+            )
+            pbar.update()
+        finally:
+            tls.pageno = None
 
     exec_progress_pool(
-        use_threads=context.options.use_threads,
+        use_threads=options.use_threads,
         max_workers=max_workers,
         tqdm_kwargs=dict(
             total=(2 * len(context.pdfinfo)),
-            desc='OCR',
+            desc='OCR' if options.tesseract_timeout > 0 else 'Image processing',
             unit='page',
             unit_scale=0.5,
-            disable=not context.options.progress_bar,
+            disable=not options.progress_bar,
         ),
         task_initializer=partial(worker_init, PIL.Image.MAX_IMAGE_PIXELS),
         task=exec_page_sync,
@@ -289,28 +286,30 @@ def exec_concurrent(context: PdfContext):
     )
 
     # Output sidecar text
-    if context.options.sidecar:
+    if options.sidecar:
         text = merge_sidecars(sidecars, context)
         # Copy text file to destination
-        copy_final(text, context.options.sidecar, context)
+        copy_final(text, options.sidecar, context)
 
     # Merge layers to one single pdf
     pdf = ocrgraft.finalize()
 
     # PDF/A and metadata
+    log.info("Postprocessing...")
     pdf = post_process(pdf, context)
 
     # Copy PDF file to destination
-    copy_final(pdf, context.options.output_file, context)
+    copy_final(pdf, options.output_file, context)
 
 
-class NeverRaise(Exception):
-    """An exception that is never raised"""
+def configure_debug_logging(log_filename: Path, prefix: str = ''):
+    """
+    Create a debug log file at a specified location.
 
-    pass  # pylint: disable=unnecessary-pass
-
-
-def configure_debug_logging(log_filename, prefix=''):
+    Arguments:
+        log_filename: Where to the put the log file.
+        prefix: The logging domain prefix that should be sent to the log.
+    """
     log_file_handler = logging.FileHandler(log_filename, delay=True)
     log_file_handler.setLevel(logging.DEBUG)
     formatter = logging.Formatter(
@@ -331,12 +330,19 @@ def run_pipeline(options, *, plugin_manager, api=False):
     if not plugin_manager:
         plugin_manager = get_plugin_manager(options.plugins)
 
-    work_folder = Path(mkdtemp(prefix="com.github.ocrmypdf."))
+    work_folder = Path(mkdtemp(prefix="ocrmypdf.io."))
     debug_log_handler = None
-    if (options.keep_temporary_files or options.verbose >= 1) and not os.environ.get(
-        'PYTEST_CURRENT_TEST', ''
+    if (
+        (options.keep_temporary_files or options.verbose >= 1)
+        and not os.environ.get('PYTEST_CURRENT_TEST', '')
+        and not api
     ):
-        debug_log_handler = configure_debug_logging(Path(work_folder) / "debug.log")
+        # Debug log for command line interface only with verbose output
+        # See https://github.com/pytest-dev/pytest/issues/5502 for why we skip this
+        # when pytest is running
+        debug_log_handler = configure_debug_logging(
+            Path(work_folder) / "debug.log"
+        )  # pragma: no cover
 
     pikepdf_enable_mmap()
 

src/ocrmypdf/_validation.py

@@ -1,20 +1,9 @@
 #!/usr/bin/env python3
 # © 2015-17 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 import locale
@@ -24,7 +13,7 @@ import sys
 import unicodedata
 from pathlib import Path
 from shutil import copyfileobj
-from typing import Tuple
+from typing import List, Set, Tuple, Union
 
 import pikepdf
 import PIL
@@ -89,7 +78,7 @@ def check_options_languages(options, ocr_engine_languages):
 def check_options_output(options):
     is_latin = options.languages.issubset(HOCR_OK_LANGS)
 
-    if options.pdf_renderer == 'hocr' and not is_latin:
+    if options.pdf_renderer.startswith('hocr') and not is_latin:
         msg = (
             "The 'hocr' PDF renderer is known to cause problems with one "
             "or more of the languages in your document.  Use "
@@ -147,10 +136,10 @@ def check_options_preprocessing(options):
             raise BadArgsError(str(e))
 
 
-def _pages_from_ranges(ranges):
+def _pages_from_ranges(ranges: str) -> Set[int]:
     if is_iterable_notstr(ranges):
         return set(ranges)
-    pages = []
+    pages: List[int] = []
     page_groups = ranges.replace(' ', '').split(',')
     for g in page_groups:
         if not g:
@@ -161,9 +150,18 @@ def _pages_from_ranges(ranges):
             pages.append(int(g) - 1)
         else:
             try:
-                pages.extend(range(int(start) - 1, int(end)))
+                new_pages = list(range(int(start) - 1, int(end)))
+                if not new_pages:
+                    raise BadArgsError(f"invalid page subrange '{start}-{end}'")
+                pages.extend(new_pages)
             except ValueError:
-                raise BadArgsError("invalid page range")
+                raise BadArgsError("invalid page range") from None
+
+    if not pages:
+        raise BadArgsError(
+            f"The string of page ranges '{ranges}' did not contain any recognizable "
+            f"page ranges."
+        )
 
     if not monotonic(pages):
         log.warning(
@@ -224,12 +222,12 @@ def check_options_optimizing(options):
 
 
 def check_options_advanced(options):
-    if options.pdfa_image_compression != 'auto' and options.output_type.startswith(
+    if options.pdfa_image_compression != 'auto' and not options.output_type.startswith(
         'pdfa'
     ):
         log.warning(
-            "--pdfa-image-compression argument has no effect when "
-            "--output-type is not 'pdfa', 'pdfa-1', or 'pdfa-2'"
+            "--pdfa-image-compression argument only applies when "
+            "--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
         )
 
 
@@ -319,17 +317,6 @@ def check_closed_streams(options):  # pragma: no cover
     return True
 
 
-def log_page_orientations(pdfinfo):
-    direction = {0: 'n', 90: 'e', 180: 's', 270: 'w'}
-    orientations = []
-    for n, page in enumerate(pdfinfo):
-        angle = page.rotation or 0
-        if angle != 0:
-            orientations.append('{0}{1}'.format(n + 1, direction.get(angle, '')))
-    if orientations:
-        log.info('Page orientations detected: %s', ' '.join(orientations))
-
-
 def create_input_file(options, work_folder: Path) -> Tuple[Path, str]:
     if options.input_file == '-':
         # stdin

src/ocrmypdf/_version.py

@@ -1,19 +1,8 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 import pkg_resources

src/ocrmypdf/api.py

@@ -1,26 +1,19 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import os
 import sys
+import threading
 from enum import IntEnum
+from io import IOBase
 from pathlib import Path
-from typing import BinaryIO, Iterable, Union
+from typing import AnyStr, BinaryIO, Iterable, Optional, Union
+from warnings import warn
 
 from ocrmypdf._logging import PageNumberFilter, TqdmConsole
 from ocrmypdf._plugin_manager import get_plugin_manager
@@ -35,7 +28,10 @@ except ModuleNotFoundError:
     coloredlogs = None
 
 
-PathOrIO = Union[BinaryIO, os.PathLike, str, bytes]
+StrPath = Union[os.PathLike, AnyStr]
+PathOrIO = Union[BinaryIO, StrPath]
+
+_api_lock = threading.Lock()
 
 
 class Verbosity(IntEnum):
@@ -54,16 +50,28 @@ def configure_logging(
 ):
     """Set up logging.
 
-    Library users may wish to use this function if they want their log output to be
-    similar to ocrmypdf command line interface. If not used, the external application
-    should configure logging on its own.
+    Before calling :func:`ocrmypdf.ocr()`, you can use this function to
+    configure logging, if you want ocrmypdf's output to look like the ocrmypdf
+    command line interface. It will register log handlers, log filters, and
+    formatters, configure color logging to standard error, and adjust the log
+    levels of third party libraries. Details of this are fine-tuned and subject
+    to change. The ``verbosity`` argument is equivalent to the argument
+    ``--verbose`` and applies those settings.
 
-    ocrmypdf will perform all of its logging under the ``"ocrmypdf"`` logging namespace.
-    In addition, ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library
-    user may wish to configure both; note that pdfminer is extremely chatty at the log
-    level ``logging.INFO``.
+    If this function is not called, ocrmypdf will not configure logging, and it
+    is up to the caller of ``ocrmypdf.ocr()`` to set up logging as it wishes using
+    the Python standard library's logging module. If this function is called,
+    the caller may of course make further adjustments to logging.
 
-    Library users may perform additional configuration afterwards.
+    Regardless of whether this function is called, ocrmypdf will perform all of
+    its logging under the ``"ocrmypdf"`` logging namespace. In addition,
+    ocrmypdf imports pdfminer, which logs under ``"pdfminer"``. A library user
+    may wish to configure both; note that pdfminer is extremely chatty at the
+    log level ``logging.INFO``.
+
+    This function does not set up the ``debug.log`` log file that the command
+    line interface does at certain verbosity levels. Applications should configure
+    their own debug logging.
 
     Args:
         verbosity (Verbosity): Verbosity level.
@@ -170,14 +178,14 @@ def create_options(
         else:
             raise TypeError(f"{arg}: {val} ({type(val)})")
 
-    try:
-        cmdline.append(os.fspath(input_file))
-    except TypeError:
+    if isinstance(input_file, (BinaryIO, IOBase)):
         cmdline.append('stream://input_file')
-    try:
-        cmdline.append(os.fspath(output_file))
-    except TypeError:
+    else:
+        cmdline.append(os.fspath(input_file))
+    if isinstance(output_file, (BinaryIO, IOBase)):
         cmdline.append('stream://output_file')
+    else:
+        cmdline.append(os.fspath(output_file))
 
     parser._api_mode = True
     options = parser.parse_args(cmdline)
@@ -199,7 +207,7 @@ def ocr(  # pylint: disable=unused-argument
     language: Iterable[str] = None,
     image_dpi: int = None,
     output_type=None,
-    sidecar: os.PathLike = None,
+    sidecar: Optional[StrPath] = None,
     jobs: int = None,
     use_threads: bool = None,
     title: str = None,
@@ -236,7 +244,7 @@ def ocr(  # pylint: disable=unused-argument
     user_words: os.PathLike = None,
     user_patterns: os.PathLike = None,
     fast_web_view: float = None,
-    plugins: Iterable[str] = None,
+    plugins: Iterable[StrPath] = None,
     keep_temporary_files: bool = None,
     progress_bar: bool = None,
     **kwargs,
@@ -258,7 +266,7 @@ def ocr(  # pylint: disable=unused-argument
             read.
         output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
             interpreted as file system path to the output file. If the object
-            appears to be a writable stream (with methods such as ``.read()`` and
+            appears to be a writable stream (with methods such as ``.write()`` and
             ``.seek()``), the output will be written to this stream. If
             ``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
             (provided that standard output does not seem to be a terminal device).
@@ -290,18 +298,29 @@ def ocr(  # pylint: disable=unused-argument
     """
     if not plugins:
         plugins = []
+    elif isinstance(plugins, (str, Path)):
+        plugins = [plugins]
     else:
         plugins = list(plugins)
 
-    parser = get_parser()
-    _plugin_manager = get_plugin_manager(plugins)
-    _plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
-
-    create_options_kwargs = {
-        k: v for k, v in locals().items() if not k.startswith('_') and k != 'kwargs'
-    }
+    # No new variable names should be assigned until these two steps are run
+    create_options_kwargs = {k: v for k, v in locals().items() if k != 'kwargs'}
     create_options_kwargs.update(kwargs)
 
-    options = create_options(**create_options_kwargs)
-    check_options(options, _plugin_manager)
-    return run_pipeline(options=options, plugin_manager=_plugin_manager, api=True)
+    parser = get_parser()
+    create_options_kwargs['parser'] = parser
+
+    with _api_lock:
+        # We can't allow multiple ocrmypdf.ocr() threads to run in parallel, because
+        # they might install different plugins, and generally speaking we have areas
+        # of code that use global state.
+
+        plugin_manager = get_plugin_manager(plugins)
+        plugin_manager.hook.add_options(parser=parser)  # pylint: disable=no-member
+
+        if 'verbose' in kwargs:
+            warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
+
+        options = create_options(**create_options_kwargs)
+        check_options(options, plugin_manager)
+        return run_pipeline(options=options, plugin_manager=plugin_manager, api=True)

src/ocrmypdf/builtin_plugins/__init__.py

@@ -1,16 +1,5 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.

src/ocrmypdf/builtin_plugins/ghostscript.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 
@@ -49,13 +39,11 @@ def check_options(options):
     if gs_version < '9.20' and options.output_type != 'pdf' and not is_latin:
         # https://bugs.ghostscript.com/show_bug.cgi?id=696874
         # Ghostscript < 9.20 fails to encode multibyte characters properly
-        msg = (
-            "The installed version of Ghostscript does not work correctly "
-            "with the OCR languages you specified. Use --output-type pdf or "
+        log.warning(
+            f"The installed version of Ghostscript ({gs_version}) does not work "
+            "correctly with the OCR languages you specified. Use --output-type pdf or "
             "upgrade to Ghostscript 9.20 or later to avoid this issue."
         )
-        msg += f"Found Ghostscript {gs_version}"
-        log.warning(msg)
 
     if options.output_type == 'pdfa':
         options.output_type = 'pdfa-2'
@@ -73,9 +61,9 @@ def rasterize_pdf_page(
     raster_device,
     raster_dpi,
     pageno,
-    page_dpi=None,
-    rotation=None,
-    filter_vector=False,
+    page_dpi,
+    rotation,
+    filter_vector,
 ):
     ghostscript.rasterize_pdf(
         input_file,
@@ -91,12 +79,21 @@ def rasterize_pdf_page(
 
 
 @hookimpl
-def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
+def generate_pdfa(
+    pdf_pages,
+    pdfmark,
+    output_file,
+    compression,
+    pdf_version,
+    pdfa_part,
+    progressbar_class,
+):
     ghostscript.generate_pdfa(
         pdf_pages=[*pdf_pages, pdfmark],
         output_file=output_file,
         compression=compression,
         pdf_version=pdf_version,
         pdfa_part=pdfa_part,
+        progressbar_class=progressbar_class,
     )
     return output_file

src/ocrmypdf/builtin_plugins/tesseract_ocr.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import os
@@ -131,9 +121,7 @@ def validate(pdfinfo, options):
         os.environ['OMP_THREAD_LIMIT'] = str(tess_threads)
     else:
         tess_threads = int(os.environ['OMP_THREAD_LIMIT'])
-
-    if tess_threads > 1:
-        log.info("Using Tesseract OpenMP thread limit %d", tess_threads)
+    log.debug("Using Tesseract OpenMP thread limit %d", tess_threads)
 
 
 class TesseractOcrEngine(OcrEngine):

src/ocrmypdf/cli.py

@@ -1,27 +1,20 @@
 # © 2015-19 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import argparse
+from typing import Optional, Type, TypeVar
 
 from ocrmypdf._version import PROGRAM_NAME as _PROGRAM_NAME
 from ocrmypdf._version import __version__ as _VERSION
 
+T = TypeVar('T')
 
-def numeric(basetype, min_=None, max_=None):
+
+def numeric(basetype: Type[T], min_: Optional[T] = None, max_: Optional[T] = None):
     """Validator for numeric params"""
     min_ = basetype(min_) if min_ is not None else None
     max_ = basetype(max_) if max_ is not None else None
@@ -417,7 +410,7 @@ Online documentation is located at:
     )
     advanced.add_argument(
         '--pdf-renderer',
-        choices=['auto', 'hocr', 'sandwich'],
+        choices=['auto', 'hocr', 'sandwich', 'hocrdebug'],
         default='auto',
         help="Choose OCR PDF renderer - the default option is to let OCRmyPDF "
         "choose.  See documentation for discussion.",

src/ocrmypdf/exceptions.py

@@ -1,19 +1,8 @@
 # © 2016 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 from enum import IntEnum

src/ocrmypdf/helpers.py

@@ -1,19 +1,9 @@
 # © 2016 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import multiprocessing
@@ -27,7 +17,7 @@ from functools import wraps
 from io import StringIO
 from math import isclose
 from pathlib import Path
-from typing import Any, Sequence, TypeVar
+from typing import Any, Sequence
 
 import pikepdf
 
@@ -68,9 +58,18 @@ class Resolution(namedtuple('Resolution', ('x', 'y'))):
         return f"Resolution({self.x}x{self.y} dpi)"
 
 
+class NeverRaise(Exception):
+    """An exception that is never raised"""
+
+
 def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike):
-    """
-    Helper function: relinks soft symbolic link if necessary
+    """Create a symbolic link at ``soft_link_name``, which references ``input_file``.
+
+    Think of this as copying ``input_file`` to ``soft_link_name`` with less overhead.
+
+    Use symlinks safely. Self-linking loops are prevented. On Windows, file copy is
+    used since symlinks may require administrator privileges. An existing link at the
+    destination is removed.
     """
     input_file = os.fspath(input_file)
     soft_link_name = os.fspath(soft_link_name)
@@ -78,8 +77,8 @@ def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike):
     # Guard against soft linking to oneself
     if input_file == soft_link_name:
         log.warning(
-            "No symbolic link made. You are using "
-            "the original data directory as the working directory."
+            "No symbolic link created. You are using  the original data directory "
+            "as the working directory."
         )
         return
 
@@ -120,7 +119,7 @@ def is_iterable_notstr(thing: Any) -> bool:
 
 
 def monotonic(L: Sequence) -> bool:
-    """Does list increase monotonically?"""
+    """Does this sequence increase monotonically?"""
     return all(b > a for a, b in zip(L, L[1:]))
 
 
@@ -179,7 +178,8 @@ def is_file_writable(test_file: os.PathLike) -> bool:
 def check_pdf(input_file: Path) -> bool:
     """Check if a PDF complies with the PDF specification.
 
-    Checks for proper formatting and proper linearization.
+    Checks for proper formatting and proper linearization. Uses pikepdf (which in
+    turn, uses QPDF) to perform the checks.
     """
     pdf = None
     try:
@@ -196,17 +196,25 @@ def check_pdf(input_file: Path) -> bool:
                 log.warning(msg)
 
         sio = StringIO()
-        linearize = None
+        linearize_msgs = ''
         try:
+            # If linearization is missing entirely, we do not complain. We do
+            # complain if linearization is present but incorrect.
             pdf.check_linearization(sio)
         except RuntimeError:
             pass
+        except (
+            getattr(pikepdf, 'ForeignObjectError')
+            if pikepdf.__version__ == '2.1.0'  # This version may throw wrong exception
+            else NeverRaise
+        ):
+            pass
         else:
-            linearize = sio.getvalue()
-            if linearize:
-                log.warning(linearize)
+            linearize_msgs = sio.getvalue()
+            if linearize_msgs:
+                log.warning(linearize_msgs)
 
-        if not messages and not linearize:
+        if not messages and not linearize_msgs:
             return True
         return False
     finally:
@@ -214,24 +222,26 @@ def check_pdf(input_file: Path) -> bool:
             pdf.close()
 
 
-T = TypeVar('T')
-
-
-def clamp(n: T, smallest: T, largest: T) -> T:
-    """Clamps the value of n to between smallest and largest."""
+def clamp(n, smallest, largest):  # mypy doesn't understand types for this
+    """Clamps the value of ``n`` to between ``smallest`` and ``largest``."""
     return max(smallest, min(n, largest))
 
 
 def pikepdf_enable_mmap():
-    try:
-        if pikepdf._qpdf.set_access_default_mmap(True):
-            log.debug("pikepdf mmap enabled")
-    except AttributeError:
-        log.debug("pikepdf mmap not available")
+    # try:
+    #     if pikepdf._qpdf.set_access_default_mmap(True):
+    #         log.debug("pikepdf mmap enabled")
+    # except AttributeError:
+    #     log.debug("pikepdf mmap not available")
+    # We found a race condition probably related to pybind issue #2252 that can
+    # cause a crash. For now, disable pikepdf mmap to be on the safe side.
+    # Fix is not in pybind11 2.6.0
+    log.debug("pikepdf mmap disabled")
+    return
 
 
 def deprecated(func):
-    """Warn that function is deprecated"""
+    """Warn that function is deprecated."""
 
     @wraps(func)
     def new_func(*args, **kwargs):

src/ocrmypdf/hocrtransform.py

@@ -31,18 +31,26 @@
 import argparse
 import os
 import re
-from collections import namedtuple
 from itertools import chain
 from math import atan, cos, sin
 from pathlib import Path
-from typing import Optional, Tuple, Union
+from typing import Any, NamedTuple, Optional, Tuple, Union
 from xml.etree import ElementTree
 
 from reportlab.lib.colors import black, cyan, magenta, red
 from reportlab.lib.units import inch
 from reportlab.pdfgen.canvas import Canvas
 
-Rect = namedtuple('Rect', ['x1', 'y1', 'x2', 'y2'])
+Element = ElementTree.Element
+
+
+class Rect(NamedTuple):  # pylint: disable=inherit-non-class
+    """A rectangle for managing PDF coordinates."""
+
+    x1: Any
+    y1: Any
+    x2: Any
+    y2: Any
 
 
 class HocrTransformError(Exception):
@@ -105,7 +113,7 @@ class HocrTransform:
         else:
             return ''
 
-    def _get_element_text(self, element):
+    def _get_element_text(self, element: Element):
         """
         Return the textual content of the element and its children
         """
@@ -119,7 +127,7 @@ class HocrTransform:
         return text
 
     @classmethod
-    def element_coordinates(cls, element) -> Rect:
+    def element_coordinates(cls, element: Element) -> Rect:
         """
         Returns a tuple containing the coordinates of the bounding box around
         an element
@@ -133,7 +141,7 @@ class HocrTransform:
         return out
 
     @classmethod
-    def baseline(cls, element) -> Tuple[float, float]:
+    def baseline(cls, element: Element) -> Tuple[float, float]:
         """
         Returns a tuple containing the baseline slope and intercept.
         """
@@ -149,7 +157,7 @@ class HocrTransform:
         """
         return Rect._make((c / self.dpi * inch) for c in pxl)
 
-    def _child_xpath(self, html_tag, html_class=None):
+    def _child_xpath(self, html_tag: str, html_class: Optional[str] = None) -> str:
         xpath = f".//{self.xmlns}{html_tag}"
         if html_class:
             xpath += f"[@class='{html_class}']"
@@ -167,7 +175,10 @@ class HocrTransform:
     def topdown_position(self, element):
         pxl_line_coords = self.element_coordinates(element)
         line_box = self.pt_from_pixel(pxl_line_coords)
-        return -line_box.y2
+        # Coordinates here are still in the hocr coordinate system, so 0 on the y axis
+        # is the top of the page and increasing values of y will move towards the
+        # bottom of the page.
+        return line_box.y2
 
     def to_pdf(
         self,
@@ -277,13 +288,15 @@ class HocrTransform:
     def _do_line(
         self,
         pdf: Canvas,
-        line,
+        line: Optional[Element],
         elemclass: str,
         fontname: str,
         invisible_text: bool,
         interword_spaces: bool,
         show_bounding_boxes: bool,
     ):
+        if not line:
+            return
         pxl_line_coords = self.element_coordinates(line)
         line_box = self.pt_from_pixel(pxl_line_coords)
         line_height = line_box.y2 - line_box.y1

src/ocrmypdf/leptonica.py

@@ -3,20 +3,10 @@
 #
 # © 2013-16: jbarlow83 from Github (https://github.com/jbarlow83)
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 #
 # Python FFI wrapper for Leptonica library
 
@@ -37,15 +27,16 @@ from tempfile import TemporaryFile
 
 from ocrmypdf.exceptions import MissingDependencyError
 from ocrmypdf.lib._leptonica import ffi
-from ocrmypdf.subprocess import shim_paths_with_program_files
 
 # pylint: disable=protected-access
 
 logger = logging.getLogger(__name__)
 
 if os.name == 'nt':
+    from ocrmypdf.subprocess._windows import shim_env_path
+
     libname = 'liblept-5'
-    os.environ['PATH'] = shim_paths_with_program_files()
+    os.environ['PATH'] = shim_env_path()
 else:
     libname = 'lept'
 _libpath = find_library(libname)
@@ -68,6 +59,24 @@ if not _libpath:
         ---------------------------------------------------------------------
         """
     )
+if os.name == 'nt':
+    # On Windows, recent versions of libpng require zlib. We have to make sure
+    # the zlib version being loaded is the same one that libpng was built with.
+    # This tries to import zlib from Tesseract's installation folder, falling back
+    # to find_library() if liblept is being loaded from somewhere else.
+    # Loading zlib from other places could cause a version mismatch
+    _zlib_path = os.path.join(os.path.dirname(_libpath), 'zlib1.dll')
+    if not os.path.exists(_zlib_path):
+        _zlib_path = find_library('zlib')
+    try:
+        zlib = ffi.dlopen(_zlib_path)
+    except ffi.error as e:
+        raise MissingDependencyError(
+            """
+            Could not load the zlib library. It could be that Tesseract is not installed properly,
+            we can't find the installation on your system PATH environment variable.
+            """
+        ) from e
 try:
     lept = ffi.dlopen(_libpath)
     lept.setMsgSeverity(lept.L_SEVERITY_WARNING)
@@ -161,20 +170,6 @@ tls = threading.local()
 tls.trap = None
 
 
-@ffi.callback("void(char *)")
-def _stderr_handler(cstr):
-    msg = ffi.string(cstr).decode(errors='replace')
-    if msg.startswith("Error"):
-        logger.error(msg)
-    elif msg.startswith("Warning"):
-        logger.warning(msg)
-    else:
-        logger.debug(msg)
-    if tls.trap is not None:
-        tls.trap.append(msg)
-    return
-
-
 class _LeptonicaErrorTrap_Queue:
     def __init__(self):
         self.queue = deque()
@@ -204,9 +199,25 @@ class _LeptonicaErrorTrap_Queue:
 
 
 try:
+
+    @ffi.callback("void(char *)")
+    def _stderr_handler(cstr):
+        msg = ffi.string(cstr).decode(errors='replace')
+        if msg.startswith("Error"):
+            logger.error(msg)
+        elif msg.startswith("Warning"):
+            logger.warning(msg)
+        else:
+            logger.debug(msg)
+        if tls.trap is not None:
+            tls.trap.append(msg)
+        return
+
     lept.leptSetStderrHandler(_stderr_handler)
-except ffi.error:
+except (ffi.error, MemoryError):
     # Pre-1.79 Leptonica does not have leptSetStderrHandler
+    # And some platforms, notably Apple ARM 64, do not allow the write+execute
+    # memory needed to set up the callback function.
     _LeptonicaErrorTrap = _LeptonicaErrorTrap_Redirect
 else:
     # 1.79 have this new symbol

src/ocrmypdf/lib/__init__.py

@@ -1,18 +1,8 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Bindings to external libraries"""

src/ocrmypdf/lib/compile_leptonica.py

@@ -1,20 +1,10 @@
 #!/usr/bin/env python3
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from pathlib import Path
 

src/ocrmypdf/optimize.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import sys
@@ -23,7 +13,6 @@ from functools import partial
 from os import fspath
 from pathlib import Path
 from typing import (
-    Any,
     Callable,
     Dict,
     Iterator,
@@ -34,9 +23,9 @@ from typing import (
     Optional,
     Sequence,
     Tuple,
-    Union,
 )
 
+import img2pdf
 import pikepdf
 from pikepdf import Dictionary, Name, Object, Pdf, PdfImage
 from PIL import Image
@@ -47,7 +36,7 @@ from ocrmypdf._concurrent import exec_progress_pool
 from ocrmypdf._exec import jbig2enc, pngquant
 from ocrmypdf._jobcontext import PdfContext
 from ocrmypdf.exceptions import OutputFileAccessError
-from ocrmypdf.helpers import safe_symlink
+from ocrmypdf.helpers import deprecated, safe_symlink
 
 log = logging.getLogger(__name__)
 
@@ -58,7 +47,7 @@ DEFAULT_PNG_QUALITY = 70
 Xref = NewType('Xref', int)
 
 
-class XrefExt(NamedTuple):
+class XrefExt(NamedTuple):  # pylint: disable=inherit-non-class
     xref: Xref
     ext: str
 
@@ -85,23 +74,29 @@ def extract_image_filter(
     if image.Subtype != Name.Image:
         return None
     if image.Length < 100:
-        log.debug("Skipping small image, xref %s", xref)
+        log.debug(f"Skipping small image, xref {xref}")
+        return None
+    if image.Width < 8 or image.Height < 8:  # Issue 732
+        log.debug(f"Skipping oddly sized image, xref {xref}")
         return None
 
     pim = PdfImage(image)
 
     if len(pim.filter_decodeparms) > 1:
-        log.debug("Skipping multiply filtered, xref %s", xref)
+        log.debug(f"Skipping multiply filtered image, xref {xref}")
         return None
     filtdp = pim.filter_decodeparms[0]
 
     if pim.bits_per_component > 8:
+        log.debug(f"Skipping wide gamut image, xref {xref}")
         return None  # Don't mess with wide gamut images
 
     if filtdp[0] == Name.JPXDecode:
+        log.debug(f"Skipping JPEG2000 iamge, xref {xref}")
         return None  # Don't do JPEG2000
 
     if Name.Decode in image:
+        log.debug(f"Skipping image with Decode table, xref {xref}")
         return None  # Don't mess with custom Decode tables
 
     return pim, filtdp
@@ -120,14 +115,29 @@ def extract_image_jbig2(
         and filtdp[0] != Name.JBIG2Decode
         and jbig2enc.available()
     ):
-        try:
-            imgname = root / f'{xref:08d}'
-            with imgname.open('wb') as f:
-                ext = pim.extract_to(stream=f)
-            imgname.rename(imgname.with_suffix(ext))
-        except pikepdf.UnsupportedImageTypeError:
-            return None
-        return XrefExt(xref, ext)
+        # Save any colorspace associated with the image, so that we
+        # will export a pure 1-bit PNG with no palette or ICC profile.
+        # Showing the palette or ICC to jbig2enc will cause it to perform
+        # colorspace transform to 1bpp, which will conflict the palette or
+        # ICC if it exists.
+        colorspace = pim.obj.get(pikepdf.Name.ColorSpace, None)
+        if colorspace is not None or pim.image_mask:
+            try:
+                # Set to DeviceGray temporarily; we already in 1 bpc.
+                pim.obj.ColorSpace = pikepdf.Name.DeviceGray
+                imgname = root / f'{xref:08d}'
+                with imgname.open('wb') as f:
+                    ext = pim.extract_to(stream=f)
+                imgname.rename(imgname.with_suffix(ext))
+            except pikepdf.UnsupportedImageTypeError:
+                return None
+            finally:
+                # Restore image colorspace after temporarily setting it to DeviceGray
+                if colorspace is not None:
+                    pim.obj.ColorSpace = colorspace
+                else:
+                    del pim.obj.ColorSpace
+            return XrefExt(xref, ext)
     return None
 
 
@@ -143,11 +153,6 @@ def extract_image_generic(
     if pim.bits_per_component == 1:
         return None
 
-    try:
-        pim.indexed  # pikepdf 1.6.3 can't handle [/Indexed [/Array...]]
-    except NotImplementedError:
-        return None
-
     if filtdp[0] == Name.DCTDecode and options.optimize >= 2:
         # This is a simple heuristic derived from some training data, that has
         # about a 70% chance of guessing whether the JPEG is high quality,
@@ -203,7 +208,10 @@ def extract_image_generic(
 
 
 def extract_images(
-    pike: Pdf, root: Path, options, extract_fn: Callable[..., Optional[XrefExt]],
+    pike: Pdf,
+    root: Path,
+    options,
+    extract_fn: Callable[..., Optional[XrefExt]],
 ) -> Iterator[Tuple[int, XrefExt]]:
     """Extract image using extract_fn
 
@@ -237,7 +245,9 @@ def extract_images(
                 # Ignore soft masks
                 smask_xref = Xref(image.SMask.objgen[0])
                 exclude_xrefs.add(smask_xref)
+                log.debug(f"Skipping image {smask_xref} because it is an SMask")
             include_xrefs.add(xref)
+            log.debug(f"Treating {xref} as an optimization candidate")
             if xref not in pageno_for_xref:
                 pageno_for_xref[xref] = pageno
 
@@ -282,10 +292,6 @@ def extract_images_jbig2(pike: Pdf, root: Path, options) -> Dict[int, List[XrefE
         group = pageno // options.jbig2_page_group_size
         jbig2_groups[group].append(xref_ext)
 
-    # Elide empty groups
-    jbig2_groups = {
-        group: xrefs for group, xrefs in jbig2_groups.items() if len(xrefs) > 0
-    }
     log.debug("Optimizable images: JBIG2 groups: %s", (len(jbig2_groups),))
     return jbig2_groups
 
@@ -403,6 +409,47 @@ def transcode_jpegs(pike: Pdf, jpegs: Sequence[Xref], root: Path, options) -> No
         im_obj.write(compdata.read(), filter=Name.DCTDecode)
 
 
+def _transcode_png(pike: Pdf, filename: Path, xref: Xref) -> bool:
+    output = filename.with_suffix('.png.pdf')
+    with output.open('wb') as f:
+        img2pdf.convert(fspath(filename), outputstream=f)
+
+    with pikepdf.open(output) as pdf_image:
+        foreign_image = next(pdf_image.pages[0].images.values())
+        local_image = pike.copy_foreign(foreign_image)
+
+        im_obj = pike.get_object(xref, 0)
+        im_obj.write(
+            local_image.read_raw_bytes(),
+            filter=local_image.Filter,
+            decode_parms=local_image.DecodeParms,
+        )
+
+        # Don't copy keys from the new image...
+        del_keys = set(im_obj.keys()) - set(local_image.keys())
+        # ...except for the keep_fields, which are essential to displaying
+        # the image correctly and preserving its metadata. (/Decode arrays
+        # and /SMaskInData are implicitly discarded prior to this point.)
+        keep_fields = {
+            '/ID',
+            '/Intent',
+            '/Interpolate',
+            '/Mask',
+            '/Metadata',
+            '/OC',
+            '/OPI',
+            '/SMask',
+            '/StructParent',
+        }
+        del_keys -= keep_fields
+        for key in local_image.keys():
+            if key != Name.Length and str(key) not in keep_fields:
+                im_obj[key] = local_image[key]
+        for key in del_keys:
+            del im_obj[key]
+    return True
+
+
 def transcode_pngs(
     pike: Pdf,
     images: Sequence[Xref],
@@ -445,35 +492,12 @@ def transcode_pngs(
         )
 
     for xref in modified:
-        im_obj = pike.get_object(xref, 0)
-        try:
-            pix = leptonica.Pix.open(png_name(root, xref))
-            if pix.mode == '1':
-                compdata = pix.generate_pdf_ci_data(leptonica.lept.L_G4_ENCODE, 0)
-            else:
-                compdata = leptonica.CompressedData.open(png_name(root, xref))
-        except leptonica.LeptonicaError as e:
-            # Most likely this means file not found, i.e. quantize did not
-            # produce an improved version
-            log.error(e)
-            continue
-
-        # If re-coded image is larger don't use it - we test here because
-        # pngquant knows the size of the temporary output file but not the actual
-        # object in the PDF
-        if len(compdata) > int(im_obj.stream_dict.Length):
-            log.debug(
-                f"pngquant: pngquant did not improve over original image "
-                f"{len(compdata)} > {int(im_obj.stream_dict.Length)}"
-            )
-            continue
-        if compdata.type == leptonica.lept.L_FLATE_ENCODE:
-            rewrite_png(pike, im_obj, compdata)
-        elif compdata.type == leptonica.lept.L_G4_ENCODE:
-            rewrite_png_as_g4(pike, im_obj, compdata)
+        filename = png_name(root, xref)
+        _transcode_png(pike, filename, xref)
 
 
-def rewrite_png_as_g4(pike: Pdf, im_obj: Object, compdata) -> None:
+@deprecated
+def rewrite_png_as_g4(pike: Pdf, im_obj: Object, compdata) -> None:  # pragma: no cover
     im_obj.BitsPerComponent = 1
     im_obj.Width = compdata.w
     im_obj.Height = compdata.h
@@ -493,7 +517,8 @@ def rewrite_png_as_g4(pike: Pdf, im_obj: Object, compdata) -> None:
     return
 
 
-def rewrite_png(pike: Pdf, im_obj: Object, compdata) -> None:
+@deprecated
+def rewrite_png(pike: Pdf, im_obj: Object, compdata) -> None:  # pragma: no cover
     # When a PNG is inserted into a PDF, we more or less copy the IDAT section from
     # the PDF and transfer the rest of the PNG headers to PDF image metadata.
     # One thing we have to do is tell the PDF reader whether a predictor was used
@@ -538,10 +563,10 @@ def rewrite_png(pike: Pdf, im_obj: Object, compdata) -> None:
         # ncolors == 0 means we are using a colorspace without a palette
         if compdata.spp == 1:
             cs = Name.DeviceGray
-        elif compdata.spp == 3:
-            cs = Name.DeviceRGB
         elif compdata.spp == 4:
             cs = Name.DeviceCMYK
+        else:  # spp == 3
+            cs = Name.DeviceRGB
     im_obj.ColorSpace = cs
     im_obj.write(compdata.read(), filter=Name.FlateDecode, decode_parms=dparms)
 
@@ -586,7 +611,7 @@ def optimize(input_file: Path, output_file: Path, context, save_settings) -> Non
         )
     ratio = input_size / output_size
     savings = 1 - output_size / input_size
-    log.info(f"Optimize ratio: {ratio:.2f} savings: {(100 * savings):.1f}%")
+    log.info(f"Optimize ratio: {ratio:.2f} savings: {(savings):.1%}")
 
     if savings < 0:
         log.info("Image optimization did not improve the file - discarded")

src/ocrmypdf/pdfa.py

@@ -1,19 +1,9 @@
-# © 2015 James R. Barlow: github.com/jbarlow83
+# © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """
 Utilities for PDF/A production and confirmation with Ghostspcript.
@@ -21,8 +11,7 @@ Utilities for PDF/A production and confirmation with Ghostspcript.
 
 import base64
 from pathlib import Path
-from string import Template
-from typing import Dict, Union
+from typing import Dict, Iterator, Union
 
 import pikepdf
 import pkg_resources
@@ -32,29 +21,57 @@ ICC_PROFILE_RELPATH = 'data/sRGB.icc'
 SRGB_ICC_PROFILE = pkg_resources.resource_filename('ocrmypdf', ICC_PROFILE_RELPATH)
 
 
-# This is a template written in PostScript which is needed to create PDF/A
-# files, from the Ghostscript documentation. Lines beginning with % are
-# comments. Python substitution variables have a '$' prefix.
-pdfa_def_template = u"""%!
-% Define an ICC profile :
-/ICCProfile $icc_profile
-def
+def _postscript_objdef(
+    alias: str,
+    dictionary: Dict[str, str],
+    *,
+    stream_name: str = None,
+    stream_data: bytes = None,
+) -> Iterator[str]:
+    assert (stream_name is None) == (stream_data is None)
 
-[/_objdef {icc_PDFA} /type /stream /OBJ pdfmark
-[{icc_PDFA} << /N 3 >> /PUT pdfmark
-[{icc_PDFA} ICCProfile /PUT pdfmark
+    objtype = '/stream' if stream_name else '/dict'
 
-% Define the output intent dictionary :
+    if stream_name:
+        assert stream_data is not None
+        a85_data = base64.a85encode(stream_data, adobe=True).decode('ascii')
+        yield f'{stream_name} ' + a85_data
+        yield 'def'
 
-[/_objdef {OutputIntent_PDFA} /type /dict /OBJ pdfmark
-[{OutputIntent_PDFA} <<
-  /Type /OutputIntent             % Must be so (the standard requires).
-  /S /GTS_PDFA1                   % Must be so (the standard requires).
-  /DestOutputProfile {icc_PDFA}            % Must be so (see above).
-  /OutputConditionIdentifier ($icc_identifier)
->> /PUT pdfmark
-[{Catalog} <</OutputIntents [ {OutputIntent_PDFA} ]>> /PUT pdfmark
-"""
+    if alias != '{Catalog}':  # Catalog needs no definition
+        yield f'[/_objdef {alias} /type {objtype} /OBJ pdfmark'
+
+    yield f'[{alias} <<'
+    for key, val in dictionary.items():
+        yield f'  {key} {val}'
+    yield '>> /PUT pdfmark'
+
+    if stream_name:
+        yield f'[{alias} {stream_name[1:]} /PUT pdfmark'
+
+
+def _make_postscript(icc_name: str, icc_data: bytes, colors: int) -> Iterator[str]:
+    yield '%!'
+    yield from _postscript_objdef(
+        '{icc_PDFA}',  # Not an f-string
+        {'/N': str(colors)},
+        stream_name='/ICCProfile',
+        stream_data=icc_data,
+    )
+    yield ''
+    yield from _postscript_objdef(
+        '{OutputIntent_PDFA}',
+        {
+            '/Type': '/OutputIntent',
+            '/S': '/GTS_PDFA1',
+            '/DestOutputProfile': '{icc_PDFA}',
+            '/OutputConditionIdentifier': f'({icc_name})',  # Only f-string
+        },
+    )
+    yield ''
+    yield from _postscript_objdef(
+        '{Catalog}', {'/OutputIntents': '[ {OutputIntent_PDFA} ]'}
+    )
 
 
 def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
@@ -85,13 +102,8 @@ def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
     else:
         raise NotImplementedError("Only supporting sRGB")
 
-    # Read the ICC profile, encode as ASCII85 and convert to a string which we
-    # will insert in the .ps file
     bytes_icc_profile = Path(icc_profile).read_bytes()
-    icc_profile = base64.a85encode(bytes_icc_profile, adobe=True).decode('ascii')
-
-    t = Template(pdfa_def_template)
-    ps = t.substitute(icc_profile=icc_profile, icc_identifier=icc)
+    ps = '\n'.join(_make_postscript(icc, bytes_icc_profile, 3))
 
     # We should have encoded everything to pure ASCII by this point, and
     # to be safe, only allow ASCII in PostScript
@@ -100,7 +112,7 @@ def generate_pdfa_ps(target_filename: Path, icc: str = 'sRGB'):
 
 
 def file_claims_pdfa(filename: Path):
-    """Determines if the file claims to be PDF/A compliant
+    """Determines if the file claims to be PDF/A compliant.
 
     This only checks if the XMP metadata contains a PDF/A marker. It does not
     do full PDF/A validation.

src/ocrmypdf/pdfinfo/__init__.py

@@ -1,19 +1,9 @@
 #!/usr/bin/env python3
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from ocrmypdf.pdfinfo.info import Colorspace, Encoding, PdfInfo

src/ocrmypdf/pdfinfo/info.py

@@ -1,20 +1,10 @@
 #!/usr/bin/env python3
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import re
@@ -25,14 +15,14 @@ from functools import partial
 from math import hypot, isclose
 from os import PathLike
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Union
+from typing import Container, Iterator, Optional, Tuple, Union
 from warnings import warn
 
 import pikepdf
-from pikepdf import PdfMatrix
+from pikepdf import Object, Pdf, PdfMatrix
 
 from ocrmypdf._concurrent import exec_progress_pool
-from ocrmypdf.exceptions import EncryptedPdfError
+from ocrmypdf.exceptions import EncryptedPdfError, InputFileError
 from ocrmypdf.helpers import Resolution, available_cpu_count, pikepdf_enable_mmap
 from ocrmypdf.pdfinfo.layout import get_page_analysis, get_text_boxes
 
@@ -125,7 +115,7 @@ def _normalize_stack(graphobjs):
             yield (operands, operator)
 
 
-def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
+def _interpret_contents(contentstream: Object, initial_shorthand=UNIT_SQUARE):
     """Interpret the PDF content stream.
 
     The stack represents the state of the PDF graphics stack.  We are only
@@ -214,7 +204,7 @@ def _interpret_contents(contentstream, initial_shorthand=UNIT_SQUARE):
     )
 
 
-def _get_dpi(ctm_shorthand, image_size):
+def _get_dpi(ctm_shorthand, image_size) -> Resolution:
     """Given the transformation matrix and image size, find the image DPI.
 
     PDFs do not include image resolution information within image data.
@@ -281,8 +271,14 @@ def _get_dpi(ctm_shorthand, image_size):
 class ImageInfo:
     DPI_PREC = Decimal('1.000')
 
-    def __init__(self, *, name='', pdfimage=None, inline=None, shorthand=None):
-
+    def __init__(
+        self,
+        *,
+        name='',
+        pdfimage: Optional[Object] = None,
+        inline: Optional[Object] = None,
+        shorthand=None,
+    ):
         self._name = str(name)
         self._shorthand = shorthand
 
@@ -292,6 +288,8 @@ class ImageInfo:
         elif pdfimage is not None:
             self._origin = 'xobject'
             pim = pikepdf.PdfImage(pdfimage)
+        else:
+            raise ValueError("Either pdfimage or inline must be set")
         self._width = pim.width
         self._height = pim.height
 
@@ -381,7 +379,7 @@ class ImageInfo:
         ).format(**class_locals)
 
 
-def _find_inline_images(contentsinfo):
+def _find_inline_images(contentsinfo: ContentsInfo) -> Iterator[ImageInfo]:
     "Find inline images in the contentstream"
 
     for n, inline in enumerate(contentsinfo.inline_images):
@@ -390,7 +388,7 @@ def _find_inline_images(contentsinfo):
         )
 
 
-def _image_xobjects(container):
+def _image_xobjects(container) -> Iterator[Tuple[Object, str]]:
     """Search for all XObject-based images in the container
 
     Usually the container is a page, but it could also be a Form XObject
@@ -410,7 +408,7 @@ def _image_xobjects(container):
         return
     xobjs = resources['/XObject'].as_dict()
     for xobj in xobjs:
-        candidate = xobjs[xobj]
+        candidate: Object = xobjs[xobj]
         if not '/Subtype' in candidate:
             continue
         if candidate['/Subtype'] == '/Image':
@@ -418,7 +416,9 @@ def _image_xobjects(container):
             yield (pdfimage, xobj)
 
 
-def _find_regular_images(container, contentsinfo):
+def _find_regular_images(
+    container: Object, contentsinfo: ContentsInfo
+) -> Iterator[ImageInfo]:
     """Find images stored in the container's /Resources /XObject
 
     Usually the container is a page, but it could also be a Form XObject
@@ -442,7 +442,7 @@ def _find_regular_images(container, contentsinfo):
             yield ImageInfo(name=draw.name, pdfimage=pdfimage, shorthand=draw.shorthand)
 
 
-def _find_form_xobject_images(pdf, container, contentsinfo):
+def _find_form_xobject_images(pdf: Pdf, container: Object, contentsinfo: ContentsInfo):
     """Find any images that are in Form XObjects in the container
 
     The container may be a page, or a parent Form XObject.
@@ -474,7 +474,9 @@ def _find_form_xobject_images(pdf, container, contentsinfo):
             )
 
 
-def _process_content_streams(*, pdf, container, shorthand=None):
+def _process_content_streams(
+    *, pdf: Pdf, container: Object, shorthand=None
+) -> Iterator[Union[VectorMarker, TextMarker, ImageInfo]]:
     """Find all individual instances of images drawn in the container
 
     Usually the container is a page, but it may also be a Form XObject.
@@ -536,7 +538,7 @@ def _page_has_text(text_blocks, page_width, page_height) -> bool:
         margin_ratio * ph,  # bottom  (first quadrant: bottom < top)
     )
 
-    def rects_intersect(a, b):
+    def rects_intersect(a, b) -> bool:
         """
         Where (a,b) are 4-tuple rects (left-0, top-1, right-2, bottom-3)
         https://stackoverflow.com/questions/306316/determine-if-two-rectangles-overlap-each-other
@@ -552,7 +554,7 @@ def _page_has_text(text_blocks, page_width, page_height) -> bool:
     return has_text
 
 
-def simplify_textboxes(miner, textbox_getter):
+def simplify_textboxes(miner, textbox_getter) -> Iterator[TextboxInfo]:
     """Extract only limited content from text boxes
 
     We do this to save memory and ensure that our objects are pickleable.
@@ -566,80 +568,18 @@ def simplify_textboxes(miner, textbox_getter):
         yield TextboxInfo(box.bbox, visible, corrupt)
 
 
-def _pdf_get_pageinfo(
-    pdf, pageno: int, infile: PathLike, check_pages, detailed_analysis: bool
-):
-    pageinfo: Dict[str, Any] = {}
-    pageinfo['pageno'] = pageno
-    pageinfo['images'] = []
-
-    page = pdf.pages[pageno]
-    mediabox = [Decimal(d) for d in page.MediaBox.as_list()]
-    width_pt = mediabox[2] - mediabox[0]
-    height_pt = mediabox[3] - mediabox[1]
-
-    check_this_page = pageno in check_pages
-
-    if check_this_page and detailed_analysis:
-        pscript5_mode = str(pdf.docinfo.get('/Creator')).startswith('PScript5')
-        miner = get_page_analysis(infile, pageno, pscript5_mode)
-        pageinfo['textboxes'] = list(simplify_textboxes(miner, get_text_boxes))
-        bboxes = (box.bbox for box in pageinfo['textboxes'])
-
-        pageinfo['has_text'] = _page_has_text(bboxes, width_pt, height_pt)
-    else:
-        pageinfo['textboxes'] = []
-        pageinfo['has_text'] = None  # i.e. "no information"
-
-    userunit = page.get('/UserUnit', Decimal(1.0))
-    if not isinstance(userunit, Decimal):
-        userunit = Decimal(userunit)
-    pageinfo['userunit'] = userunit
-    pageinfo['width_inches'] = width_pt * userunit / Decimal(72.0)
-    pageinfo['height_inches'] = height_pt * userunit / Decimal(72.0)
-
-    try:
-        pageinfo['rotate'] = int(page['/Rotate'])
-    except KeyError:
-        pageinfo['rotate'] = 0
-
-    userunit_shorthand = (userunit, 0, 0, userunit, 0, 0)
-
-    if check_this_page:
-        pageinfo['has_vector'] = False
-        pageinfo['has_text'] = False
-        pageinfo['images'] = []
-        for ci in _process_content_streams(
-            pdf=pdf, container=page, shorthand=userunit_shorthand
-        ):
-            if isinstance(ci, VectorMarker):
-                pageinfo['has_vector'] = True
-            elif isinstance(ci, TextMarker):
-                pageinfo['has_text'] = True
-            elif isinstance(ci, ImageInfo):
-                pageinfo['images'].append(ci)
-            else:
-                raise NotImplementedError()
-    else:
-        pageinfo['has_vector'] = None  # i.e. "no information"
-        pageinfo['has_text'] = None
-        pageinfo['images'] = None
-
-    if pageinfo['images']:
-        dpi = Resolution(0.0, 0.0).take_max(image.dpi for image in pageinfo['images'])
-        pageinfo['dpi'] = dpi
-        pageinfo['width_pixels'] = int(round(dpi.x * float(pageinfo['width_inches'])))
-        pageinfo['height_pixels'] = int(round(dpi.y * float(pageinfo['height_inches'])))
-
-    return pageinfo
-
-
 worker_pdf = None
 
 
-def _pdf_pageinfo_sync_init(infile):
+def _pdf_pageinfo_sync_init(infile: Path, pdfminer_loglevel):
     global worker_pdf  # pylint: disable=global-statement
     pikepdf_enable_mmap()
+
+    logging.getLogger('pdfminer').setLevel(pdfminer_loglevel)
+
+    # If this function is called as a thread initializer, we need a messy hack
+    # to close worker_pdf. If called as a process, it will be released when the
+    # process is terminated.
     worker_pdf = pikepdf.open(infile)
 
 
@@ -653,10 +593,13 @@ def _pdf_pageinfo_sync(args):
 def _pdf_pageinfo_concurrent(
     pdf, infile, progbar, max_workers, check_pages, detailed_analysis=False
 ):
+    global worker_pdf  # pylint: disable=global-statement
     pages = [None] * len(pdf.pages)
 
     def update_pageinfo(result, pbar):
         page = result
+        if not page:
+            raise InputFileError("Could read a page in the PDF")
         pages[page.pageno] = page
         pbar.update()
 
@@ -673,28 +616,108 @@ def _pdf_pageinfo_concurrent(
         # a separate process.
         use_threads = True
 
-    exec_progress_pool(
-        use_threads=use_threads,
-        max_workers=n_workers,
-        tqdm_kwargs=dict(
-            total=total, desc="Scanning contents", unit='page', disable=not progbar
-        ),
-        task_initializer=partial(_pdf_pageinfo_sync_init, infile),
-        task=_pdf_pageinfo_sync,
-        task_arguments=contexts,
-        task_finished=update_pageinfo,
-    )
+    try:
+        exec_progress_pool(
+            use_threads=use_threads,
+            max_workers=n_workers,
+            tqdm_kwargs=dict(
+                total=total, desc="Scanning contents", unit='page', disable=not progbar
+            ),
+            task_initializer=partial(
+                _pdf_pageinfo_sync_init, infile, logging.getLogger('pdfminer').level
+            ),
+            task=_pdf_pageinfo_sync,
+            task_arguments=contexts,
+            task_finished=update_pageinfo,
+        )
+    finally:
+        if worker_pdf and use_threads:
+            assert n_workers == 1, "Should have only one worker when threaded"
+            # This is messy, but if we ran in thread, close worker_pdf
+            worker_pdf.close()
     return pages
 
 
 class PageInfo:
-    def __init__(self, pdf, pageno, infile, check_pages, detailed_analysis=False):
+    def __init__(
+        self,
+        pdf: Pdf,
+        pageno: int,
+        infile: PathLike,
+        check_pages: Container[int],
+        detailed_analysis: bool = False,
+    ):
         self._pageno = pageno
         self._infile = infile
         self._detailed_analysis = detailed_analysis
-        self._pageinfo = _pdf_get_pageinfo(
-            pdf, pageno, infile, check_pages, detailed_analysis
-        )
+        self._gather_pageinfo(pdf, pageno, infile, check_pages, detailed_analysis)
+
+    def _gather_pageinfo(
+        self,
+        pdf: Pdf,
+        pageno: int,
+        infile: PathLike,
+        check_pages: Container[int],
+        detailed_analysis: bool,
+    ):
+        page = pdf.pages[pageno]
+        mediabox = [Decimal(d) for d in page.MediaBox.as_list()]
+        width_pt = mediabox[2] - mediabox[0]
+        height_pt = mediabox[3] - mediabox[1]
+
+        check_this_page = pageno in check_pages
+
+        if check_this_page and detailed_analysis:
+            pscript5_mode = str(pdf.docinfo.get('/Creator')).startswith('PScript5')
+            miner = get_page_analysis(infile, pageno, pscript5_mode)
+            self._textboxes = list(simplify_textboxes(miner, get_text_boxes))
+            bboxes = (box.bbox for box in self._textboxes)
+
+            self._has_text = _page_has_text(bboxes, width_pt, height_pt)
+        else:
+            self._textboxes = []
+            self._has_text = None  # i.e. "no information"
+
+        userunit = page.get('/UserUnit', Decimal(1.0))
+        if not isinstance(userunit, Decimal):
+            userunit = Decimal(userunit)
+        self._userunit = userunit
+        self._width_inches = width_pt * userunit / Decimal(72.0)
+        self._height_inches = height_pt * userunit / Decimal(72.0)
+
+        try:
+            self._rotate = int(page['/Rotate'])
+        except KeyError:
+            self._rotate = 0
+
+        userunit_shorthand = (userunit, 0, 0, userunit, 0, 0)
+
+        if check_this_page:
+            self._has_vector = False
+            self._has_text = False
+            self._images = []
+            for ci in _process_content_streams(
+                pdf=pdf, container=page, shorthand=userunit_shorthand
+            ):
+                if isinstance(ci, VectorMarker):
+                    self._has_vector = True
+                elif isinstance(ci, TextMarker):
+                    self._has_text = True
+                elif isinstance(ci, ImageInfo):
+                    self._images.append(ci)
+                else:
+                    raise NotImplementedError()
+        else:
+            self._has_vector = None  # i.e. "no information"
+            self._has_text = None
+            self._images = None
+
+        self._dpi = None
+        if self._images:
+            dpi = Resolution(0.0, 0.0).take_max(image.dpi for image in self._images)
+            self._dpi = dpi
+            self._width_pixels = int(round(dpi.x * float(self._width_inches)))
+            self._height_pixels = int(round(dpi.y * float(self._height_inches)))
 
     @property
     def pageno(self) -> int:
@@ -702,25 +725,25 @@ class PageInfo:
 
     @property
     def has_text(self) -> bool:
-        return self._pageinfo['has_text']
+        return self._has_text
 
     @property
     def has_corrupt_text(self) -> bool:
         if not self._detailed_analysis:
             raise NotImplementedError('Did not do detailed analysis')
-        return any(tbox.is_corrupt for tbox in self._pageinfo['textboxes'])
+        return any(tbox.is_corrupt for tbox in self._textboxes)
 
     @property
     def has_vector(self) -> bool:
-        return self._pageinfo['has_vector']
+        return self._has_vector
 
     @property
     def width_inches(self) -> Decimal:
-        return self._pageinfo['width_inches']
+        return self._width_inches
 
     @property
     def height_inches(self) -> Decimal:
-        return self._pageinfo['height_inches']
+        return self._height_inches
 
     @property
     def width_pixels(self) -> int:
@@ -732,18 +755,18 @@ class PageInfo:
 
     @property
     def rotation(self) -> int:
-        return self._pageinfo.get('rotate', None)
+        return self._rotate
 
     @rotation.setter
     def rotation(self, value):
         if value in (0, 90, 180, 270, 360, -90, -180, -270):
-            self._pageinfo['rotate'] = value
+            self._rotate = value
         else:
             raise ValueError("rotation must be a cardinal angle")
 
     @property
     def images(self):
-        return self._pageinfo['images']
+        return self._images
 
     def get_textareas(
         self, visible: Optional[bool] = None, corrupt: Optional[bool] = None
@@ -758,24 +781,22 @@ class PageInfo:
                     result = False
             return result
 
-        if 'textboxes' not in self._pageinfo:
+        if not self._textboxes:
             if visible is not None and corrupt is not None:
                 raise NotImplementedError('Incomplete information on textboxes')
-            return self._pageinfo['bboxes']
+            return self._textboxes
 
-        return (
-            obj.bbox
-            for obj in self._pageinfo['textboxes']
-            if predicate(obj, visible, corrupt)
-        )
+        return (obj.bbox for obj in self._textboxes if predicate(obj, visible, corrupt))
 
     @property
     def dpi(self) -> Resolution:
-        return self._pageinfo.get('dpi', Resolution(0.0, 0.0))
+        if self._dpi is None:
+            return Resolution(0.0, 0.0)
+        return self._dpi
 
     @property
     def userunit(self) -> Decimal:
-        return self._pageinfo.get('userunit', None)
+        return self._userunit
 
     @property
     def min_version(self) -> str:
@@ -818,12 +839,12 @@ class PdfInfo:
                 check_pages=check_pages,
                 detailed_analysis=detailed_analysis,
             )
-            self._needs_rendering = pdf.root.get('/NeedsRendering', False)
+            self._needs_rendering = pdf.Root.get('/NeedsRendering', False)
             self._has_acroform = False
-            if '/AcroForm' in pdf.root:
-                if len(pdf.root.AcroForm.get('/Fields', [])) > 0:
+            if '/AcroForm' in pdf.Root:
+                if len(pdf.Root.AcroForm.get('/Fields', [])) > 0:
                     self._has_acroform = True
-                elif '/XFA' in pdf.root.AcroForm:
+                elif '/XFA' in pdf.Root.AcroForm:
                     self._has_acroform = True
 
     @property

src/ocrmypdf/pdfinfo/layout.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import re
 from math import copysign
@@ -31,7 +21,7 @@ from pdfminer.pdffont import PDFSimpleFont, PDFUnicodeNotDefined
 from pdfminer.pdfpage import PDFPage
 from pdfminer.utils import bbox2str, matrix2str
 
-from ocrmypdf.exceptions import EncryptedPdfError
+from ocrmypdf.exceptions import EncryptedPdfError, InputFileError
 
 STRIP_NAME = re.compile(r'[0-9]+')
 
@@ -233,6 +223,7 @@ def get_page_analysis(infile, pageno, pscript5_mode):
     )
     interp = pdfminer.pdfinterp.PDFPageInterpreter(rman, dev)
 
+    patcher = None
     if pscript5_mode:
         patcher = patch.multiple(
             'pdfminer.pdffont.PDFType3Font',
@@ -245,12 +236,17 @@ def get_page_analysis(infile, pageno, pscript5_mode):
 
     try:
         with Path(infile).open('rb') as f:
-            page = PDFPage.get_pages(f, pagenos=[pageno], maxpages=0)
-            interp.process_page(next(page))
-    except PDFTextExtractionNotAllowed:
-        raise EncryptedPdfError()
+            page_iter = PDFPage.get_pages(f, pagenos=[pageno], maxpages=0)
+            page = next(page_iter, None)
+            if page is None:
+                raise InputFileError(
+                    f"pdfminer could not process page {pageno} (counting from 0)."
+                )
+            interp.process_page(page)
+    except PDFTextExtractionNotAllowed as e:
+        raise EncryptedPdfError() from e
     finally:
-        if pscript5_mode:
+        if patcher is not None:
             patcher.stop()
 
     return dev.get_result()

src/ocrmypdf/pluginspec.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from abc import ABC, abstractmethod, abstractstaticmethod
 from argparse import ArgumentParser, Namespace
@@ -69,7 +59,7 @@ def check_options(options: Namespace) -> None:
     Note:
         This hook will be called from the main process, and may modify global state
         before child worker processes are forked.
-	"""
+    """
 
 
 @hookspec
@@ -99,9 +89,9 @@ def rasterize_pdf_page(
     raster_device: str,
     raster_dpi: Resolution,
     pageno: int,
-    page_dpi: Optional[Resolution] = None,
-    rotation: Optional[int] = None,
-    filter_vector: bool = False,
+    page_dpi: Optional[Resolution],
+    rotation: Optional[int],
+    filter_vector: bool,
 ) -> Path:
     """Rasterize one page of a PDF at resolution raster_dpi in canvas units.
 
@@ -164,11 +154,29 @@ def filter_page_image(page: 'PageContext', image_filename: Path) -> Path:
     produced for a given page, this function will not be called. This is not
     the image that will be shown to OCR.
 
-    ocrmypdf will create the PDF page based on the image format used. If you
+    If the function does not want to modify the image, it should return
+    ``image_filename``. The hook may overwrite ``image_filename`` with a new file.
+
+    The output image should preserve the same physical unit dimensions, that is
+    (width * dpi_x, height * dpi_y). That is, if the image is resized, the DPI
+    must be adjusted by the reciprocal. If this is not preserved, the PDF page
+    will be resized and the OCR layer misaligned. OCRmyPDF does not nothing
+    to enforce these constraints; it is up to the plugin to do sensible things.
+
+    OCRmyPDF will create the PDF page based on the image format used. If you
     convert the image to a JPEG, the output page will be created as a JPEG, etc.
-    Note that the ocrmypdf image optimization stage may ultimately chose a
+    If you change the colorspace, that change will be kept. Note that the
+    OCRmyPDF image optimization stage, if enabled, may ultimately chose a
     different format.
 
+    If the return value is a file that does not exist, ``FileNotFoundError``
+    will occur. The return value should be a path to a file in the same folder
+    as ``image_filename``.
+
+    Implementation detail: If the value returned is falsy, OCRmyPDF will ignore
+    the return value and assume the input file was unmodified. This is deprecated.
+    To leave the image unmodified, ``image_filename`` should be returned.
+
     Note:
         This hook will be called from child processes. Modifying global state
         will not affect the main process or other child processes.
@@ -272,6 +280,7 @@ def generate_pdfa(
     compression: str,
     pdf_version: str,
     pdfa_part: str,
+    progressbar_class,
 ) -> Path:
     """Generate a PDF/A.
 
@@ -294,10 +303,21 @@ def generate_pdfa(
             At its own discretion, the PDF/A generator may raise the version,
             but should not lower it.
         pdfa_part: The desired PDF/A compliance level, such as ``'2B'``.
+        progressbar_class: The class of a progress bar with a tqdm-like API. An
+            instance of this class will be initialized when PDF/A conversion
+            begins, using
+            ``instance = progressbar_class(total: int, desc: str, unit:str)``,
+            defining the number of work units, a user-visible description,
+            and the name of the work units ("page"). Then ``instance.update()``
+            will be called when a work unit is completed. If ``None``, no
+            progress information is reported.
 
     Returns:
         Path: If successful, the hook should return ``output_file``.
 
     Note:
         This is a :ref:`firstresult hook<firstresult>`.
+
+    See also:
+        https://github.com/tqdm/tqdm
     """

src/ocrmypdf/py.typed

@@ -0,0 +1 @@
+# ocrmypdf is typed

src/ocrmypdf/quality.py

@@ -1,19 +1,9 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Utilities to measure OCR quality"""
 

src/ocrmypdf/subprocess/__init__.py

@@ -1,33 +1,22 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 """Wrappers to manage subprocess calls"""
 
 import logging
 import os
 import re
-import shutil
 import sys
 from collections.abc import Mapping
 from contextlib import suppress
 from distutils.version import LooseVersion
 from functools import lru_cache
 from pathlib import Path
-from subprocess import PIPE, STDOUT, CalledProcessError
+from subprocess import PIPE, STDOUT, CalledProcessError, CompletedProcess, Popen
 from subprocess import run as subprocess_run
 
 from ocrmypdf.exceptions import MissingDependencyError
@@ -35,14 +24,77 @@ from ocrmypdf.exceptions import MissingDependencyError
 log = logging.getLogger(__name__)
 
 
-def run(args, *, env=None, **kwargs):
+def run(args, *, env=None, logs_errors_to_stdout=False, **kwargs):
     """Wrapper around :py:func:`subprocess.run`
 
     The main purpose of this wrapper is to log subprocess output in an orderly
     fashion that indentifies the responsible subprocess. An additional
     task is that this function goes to greater lengths to find possible Windows
     locations of our dependencies when they are not on the system PATH.
+
+    Arguments should be identical to ``subprocess.run``, except for following:
+
+    Arguments:
+        logs_errors_to_stdout: If True, indicates that the process writes its error
+            messages to stdout rather than stderr, so stdout should be logged
+            if there is an error. If False, stderr is logged. Could be used with
+            stderr=STDOUT, stdout=PIPE for example.
     """
+    args, env, process_log, _text = _fix_process_args(args, env, kwargs)
+
+    stderr = None
+    stderr_name = 'stderr' if not logs_errors_to_stdout else 'stdout'
+    try:
+        proc = subprocess_run(args, env=env, **kwargs)
+    except CalledProcessError as e:
+        stderr = getattr(e, stderr_name, None)
+        raise
+    else:
+        stderr = getattr(proc, stderr_name, None)
+    finally:
+        if process_log.isEnabledFor(logging.DEBUG) and stderr:
+            with suppress(AttributeError, UnicodeDecodeError):
+                stderr = stderr.decode('utf-8', 'replace')
+            if logs_errors_to_stdout:
+                process_log.debug("stdout/stderr = %s", stderr)
+            else:
+                process_log.debug("stderr = %s", stderr)
+    return proc
+
+
+def run_polling_stderr(args, *, callback, check=False, env=None, **kwargs):
+    """Run a process like ``ocrmypdf.subprocess.run``, and poll stderr.
+
+    Every line of produced by stderr will be forwarded to the callback function.
+    The intended use is monitoring progress of subprocesses that output their
+    own progress indicators. In addition, each line will be logged if debug
+    logging is enabled.
+
+    Requires stderr to be opened in text mode for ease of handling errors. In
+    addition the expected encoding= and errors= arguments should be set. Note
+    that if stdout is already set up, it need not be binary.
+    """
+    args, env, process_log, text = _fix_process_args(args, env, kwargs)
+    assert text, "Must use text=True"
+
+    with Popen(args, env=env, **kwargs) as proc:
+        lines = []
+        while proc.poll() is None:
+            for msg in iter(proc.stderr.readline, ''):
+                if process_log.isEnabledFor(logging.DEBUG):
+                    process_log.debug(msg.strip())
+                callback(msg)
+                lines.append(msg)
+        stderr = ''.join(lines)
+
+        if check and proc.returncode != 0:
+            raise CalledProcessError(proc.returncode, args, output=None, stderr=stderr)
+        return CompletedProcess(args, proc.returncode, None, stderr=stderr)
+
+
+def _fix_process_args(args, env, kwargs):
+    assert 'universal_newlines' not in kwargs, "Use text= instead of universal_newlines"
+
     if not env:
         env = os.environ
 
@@ -50,53 +102,23 @@ def run(args, *, env=None, **kwargs):
     program = args[0]
 
     if os.name == 'nt':
-        args = _fix_windows_args(program, args, env)
+        from ocrmypdf.subprocess._windows import fix_windows_args
+
+        args = fix_windows_args(program, args, env)
 
     log.debug("Running: %s", args)
-    process_log = log.getChild('subprocess.' + os.path.basename(program))
-    if sys.version_info < (3, 7) and os.name == 'nt':
-        # Can't use close_fds=True on Windows with Python 3.6 or older
-        # https://bugs.python.org/issue19575, etc.
-        kwargs['close_fds'] = False
-
-    stderr = None
-    try:
-        proc = subprocess_run(args, env=env, **kwargs)
-    except CalledProcessError as e:
-        stderr = getattr(e, 'stderr', None)
-        raise
-    else:
-        stderr = getattr(proc, 'stderr', None)
-    finally:
-        if process_log.isEnabledFor(logging.DEBUG) and stderr:
-            with suppress(AttributeError, UnicodeDecodeError):
-                stderr = stderr.decode('utf-8', 'replace')
-            process_log.debug("stderr = %s", stderr)
-    return proc
-
-
-def _fix_windows_args(program, args, env):
-    """Adjust our desired program and command line arguments for use on Windows"""
-
-    if sys.version_info < (3, 8):
-        # bpo-33617 - Windows needs manual Path -> str conversion
-        args = [os.fspath(arg) for arg in args]
-        program = os.fspath(program)
-
-    # If we are running a .py on Windows, ensure we call it with this Python
-    # (to support test suite shims)
-    if program.lower().endswith('.py'):
-        args = [sys.executable] + args
-
-    paths = os.pathsep.join(os.get_exec_path(env))
-    if not shutil.which(args[0], path=paths):
-        # If the program we want is not on the PATH, add some interesting
-        # locations in %PROGRAMFILES% to the PATH and try again
-        shimmed_path = shim_paths_with_program_files(env)
-        new_args0 = shutil.which(args[0], path=shimmed_path)
-        if new_args0:
-            args[0] = new_args0
-    return args
+    process_log = log.getChild(os.path.basename(program))
+    text = kwargs.get('text', False)
+    if sys.version_info < (3, 7):
+        if os.name == 'nt':
+            # Can't use close_fds=True on Windows with Python 3.6 or older
+            # https://bugs.python.org/issue19575, etc.
+            kwargs['close_fds'] = False
+        if 'text' in kwargs:
+            # Convert run(...text=) to run(...universal_newlines=) for Python 3.6
+            kwargs['universal_newlines'] = kwargs['text']
+            del kwargs['text']
+    return args, env, process_log, text
 
 
 @lru_cache(maxsize=None)
@@ -117,7 +139,7 @@ def get_version(
         proc = run(
             args_prog,
             close_fds=True,
-            universal_newlines=True,
+            text=True,
             stdout=PIPE,
             stderr=STDOUT,
             check=True,
@@ -136,44 +158,18 @@ def get_version(
         raise MissingDependencyError(
             f"Could not find program '{program}' on the PATH"
         ) from e
-    try:
-        version = re.match(regex, output.strip()).group(1)
-    except AttributeError as e:
+
+    match = re.match(regex, output.strip())
+    if not match:
         raise MissingDependencyError(
             f"The program '{program}' did not report its version. "
             f"Message was:\n{output}"
         )
+    version = match.group(1)
 
     return version
 
 
-def shim_paths_with_program_files(env=None):
-    if not env:
-        env = os.environ
-    program_files = env.get('PROGRAMFILES', '')
-    if not program_files:
-        return env.get('PATH', '')
-
-    def path_walker():
-        for path in Path(program_files).iterdir():
-            if not path.is_dir():
-                continue
-            if path.name.lower() == 'tesseract-ocr':
-                yield path
-            elif path.name.lower() == 'gs':
-                yield from (p for p in path.glob('**/bin') if p.is_dir())
-
-    paths = sorted(
-        (p for p in path_walker()), key=lambda p: (p.name, p.parent.name), reverse=True
-    )
-    paths.extend(
-        Path(str_path)
-        for str_path in os.get_exec_path(env)
-        if Path(str_path) not in set(paths)
-    )
-    return os.pathsep.join(str(p) for p in paths)
-
-
 missing_program = '''
 The program '{program}' could not be executed or was not found on your
 system PATH.

src/ocrmypdf/subprocess/_windows.py

@@ -0,0 +1,162 @@
+# © 2020 James R. Barlow: github.com/jbarlow83
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import logging
+import os
+import shutil
+import sys
+from distutils.version import LooseVersion
+from itertools import chain, filterfalse
+from pathlib import Path
+from typing import Any, Callable, Iterator, Optional, Tuple, TypeVar, cast
+
+try:
+    import winreg
+except ModuleNotFoundError as e:
+    raise ModuleNotFoundError("This module is for Windows only") from e
+
+log = logging.getLogger(__name__)
+
+T = TypeVar('T')
+
+
+def registry_enum(
+    key: winreg.HKEYType, enum_fn: Callable[[winreg.HKEYType, int], T]
+) -> Iterator[T]:
+    LIMIT = 999
+    n = 0
+    while n < LIMIT:
+        try:
+            yield enum_fn(key, n)
+            n += 1
+        except OSError:
+            break
+    if n == LIMIT:
+        raise ValueError(f"Too many registry keys under {key}")
+
+
+def registry_subkeys(key: winreg.HKEYType) -> Iterator[str]:
+    return registry_enum(key, winreg.EnumKey)
+
+
+def registry_values(key: winreg.HKEYType) -> Iterator[Tuple[str, Any, int]]:
+    return registry_enum(key, winreg.EnumValue)
+
+
+def registry_path_ghostscript(env=None) -> Iterator[Path]:
+    try:
+        with winreg.OpenKey(
+            winreg.HKEY_LOCAL_MACHINE, r"SOFTWARE\Artifex\GPL Ghostscript"
+        ) as k:
+            latest_gs = max(registry_subkeys(k), key=LooseVersion)
+        with winreg.OpenKey(
+            winreg.HKEY_LOCAL_MACHINE, fr"SOFTWARE\Artifex\GPL Ghostscript\{latest_gs}"
+        ) as k:
+            _, gs_path, _ = next(registry_values(k))
+            yield Path(gs_path) / 'bin'
+    except OSError as e:
+        log.warning(e)
+
+
+def registry_path_tesseract(env=None) -> Iterator[Path]:
+    try:
+        with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r"SOFTWARE\Tesseract-OCR") as k:
+            for subkey, val, _valtype in registry_values(k):
+                if subkey == 'InstallDir':
+                    tesseract_path = Path(val)
+                    yield tesseract_path
+    except OSError as e:
+        log.warning(e)
+
+
+def program_files_paths(env=None) -> Iterator[Path]:
+    if not env:
+        env = os.environ
+    program_files = env.get('PROGRAMFILES', '')
+
+    def path_walker() -> Iterator[Path]:
+        for path in Path(program_files).iterdir():
+            if not path.is_dir():
+                continue
+            if path.name.lower() == 'tesseract-ocr':
+                yield path
+            elif path.name.lower() == 'gs':
+                yield from (p for p in path.glob('**/bin') if p.is_dir())
+
+    return iter(
+        sorted(
+            (p for p in path_walker()),
+            key=lambda p: (p.name, p.parent.name),
+            reverse=True,
+        )
+    )
+
+
+def paths_from_env(env=None) -> Iterator[Path]:
+    return (Path(p) for p in os.get_exec_path(env) if p)
+
+
+def shim_path(new_paths: Callable[[Any], Iterator[Path]], env=None) -> str:
+    if not env:
+        env = os.environ
+    return os.pathsep.join(str(p) for p in new_paths(env) if p)
+
+
+SHIMS = [
+    paths_from_env,
+    registry_path_ghostscript,
+    registry_path_tesseract,
+    program_files_paths,
+]
+
+
+def fix_windows_args(program, args, env):
+    """Adjust our desired program and command line arguments for use on Windows"""
+
+    if sys.version_info < (3, 8):
+        # bpo-33617 - Windows needs manual Path -> str conversion
+        args = [os.fspath(arg) for arg in args]
+        program = os.fspath(program)
+
+    # If we are running a .py on Windows, ensure we call it with this Python
+    # (to support test suite shims)
+    if program.lower().endswith('.py'):
+        args = [sys.executable] + args
+
+    # If the program we want is not on the PATH, check elsewhere
+    for shim in SHIMS:
+        shimmed_path = shim_path(shim, env)
+        new_args0 = shutil.which(args[0], path=shimmed_path)
+        if new_args0:
+            args[0] = new_args0
+            break
+
+    return args
+
+
+def unique_everseen(iterable, key=None):
+    "List unique elements, preserving order. Remember all elements ever seen."
+    # unique_everseen('AAAABBBCCDAABBB') --> A B C D
+    # unique_everseen('ABBCcAD', str.lower) --> A B C D
+    seen = set()
+    seen_add = seen.add
+    if key is None:
+        key = lambda x: x
+    for element in iterable:
+        k = key(element)
+        if k not in seen:
+            seen_add(k)
+            yield element
+
+
+def shim_env_path(env=None):
+    if env is None:
+        env = os.environ
+
+    shim_paths = chain.from_iterable(shim(env) for shim in SHIMS)
+    return os.pathsep.join(
+        str(p) for p in unique_everseen(shim_paths, key=lambda p: str.casefold(str(p)))
+    )

tests/conftest.py

@@ -1,19 +1,9 @@
 # © 2017 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import os
 import platform
@@ -138,7 +128,7 @@ def run_ocrmypdf_api(input_file, output_file, *args):
 
 
 @pytest.helpers.register
-def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
+def run_ocrmypdf(input_file, output_file, *args, text=True):
     "Run ocrmypdf and let caller deal with results"
 
     p_args = (
@@ -147,21 +137,12 @@ def run_ocrmypdf(input_file, output_file, *args, universal_newlines=True):
         + [str(input_file), str(output_file)]
     )
 
-    # Tell subprocess where to find coverage.py configuration
-    # This has no unless except when coverage is running
-    # Details: https://coverage.readthedocs.io/en/coverage-5.0/subprocess.html
-    coverage_rc = Path(__file__).parent.parent / '.coveragerc'
     env = os.environ.copy()
-    if coverage_rc.exists():
-        env['COVERAGE_PROCESS_START'] = os.fspath(coverage_rc)
-    elif not running_in_docker():
-        assert False, "could not find .coveragerc"
-
     p = run(
         p_args,
         stdout=PIPE,
         stderr=PIPE,
-        universal_newlines=universal_newlines,
+        universal_newlines=text,  # When dropping support for Python 3.6 change to text=
         env=env,
         check=False,
     )

tests/plugins/gs_feature_elision.py

@@ -23,21 +23,22 @@ from unittest.mock import patch
 
 from ocrmypdf import hookimpl
 from ocrmypdf.builtin_plugins import ghostscript
-from ocrmypdf.subprocess import run
+from ocrmypdf.subprocess import run_polling_stderr
 
 elision_warning = """GPL Ghostscript 9.20: Setting Overprint Mode to 1
 not permitted in PDF/A-2, overprint mode not set"""
 
 
 def run_append_stderr(*args, **kwargs):
-    proc = run(*args, **kwargs)
-    proc.stderr = b'\n'.join([proc.stderr, elision_warning.encode('utf-8')])
+    proc = run_polling_stderr(*args, **kwargs)
+    proc.stderr += '\n' + elision_warning + '\n'
     return proc
 
 
 @hookimpl
 def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
-    with patch('ocrmypdf._exec.ghostscript.run', new=run_append_stderr):
+    with patch('ocrmypdf._exec.ghostscript.run_polling_stderr') as mock:
+        mock.side_effect = run_append_stderr
         ghostscript.generate_pdfa(
             pdf_pages=pdf_pages,
             pdfmark=pdfmark,
@@ -45,5 +46,7 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
-        return output_file
+        mock.assert_called_once()
+    return output_file

tests/plugins/gs_pdfa_failure.py

@@ -23,7 +23,7 @@ from unittest.mock import patch
 
 from ocrmypdf import hookimpl
 from ocrmypdf.builtin_plugins import ghostscript
-from ocrmypdf.subprocess import run
+from ocrmypdf.subprocess import run_polling_stderr
 
 
 def run_rig_args(args, **kwargs):
@@ -33,13 +33,14 @@ def run_rig_args(args, **kwargs):
     new_args = [
         arg for arg in args if not arg.startswith('-dPDFA') and not arg.endswith('.ps')
     ]
-    proc = run(new_args, **kwargs)
+    proc = run_polling_stderr(new_args, **kwargs)
     return proc
 
 
 @hookimpl
 def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
-    with patch('ocrmypdf._exec.ghostscript.run', new=run_rig_args):
+    with patch('ocrmypdf._exec.ghostscript.run_polling_stderr') as mock:
+        mock.side_effect = run_rig_args
         ghostscript.generate_pdfa(
             pdf_pages=pdf_pages,
             pdfmark=pdfmark,
@@ -47,5 +48,7 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
+        mock.assert_called()
         return output_file

tests/plugins/gs_raster_failure.py

@@ -44,7 +44,8 @@ def rasterize_pdf_page(
     rotation=None,
     filter_vector=False,
 ) -> Path:
-    with patch('ocrmypdf._exec.ghostscript.run', new=raise_gs_fail):
+    with patch('ocrmypdf._exec.ghostscript.run') as mock:
+        mock.side_effect = raise_gs_fail
         ghostscript.rasterize_pdf_page(
             input_file=input_file,
             output_file=output_file,
@@ -55,4 +56,5 @@ def rasterize_pdf_page(
             rotation=rotation,
             filter_vector=filter_vector,
         )
+        mock.assert_called()
         return output_file

tests/plugins/gs_render_failure.py

@@ -34,7 +34,8 @@ def raise_gs_fail(*args, **kwargs):
 
 @hookimpl
 def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdfa_part):
-    with patch('ocrmypdf._exec.ghostscript.run', new=raise_gs_fail):
+    with patch('ocrmypdf._exec.ghostscript.run_polling_stderr') as mock:
+        mock.side_effect = raise_gs_fail
         ghostscript.generate_pdfa(
             pdf_pages=pdf_pages,
             pdfmark=pdfmark,
@@ -42,5 +43,7 @@ def generate_pdfa(pdf_pages, pdfmark, output_file, compression, pdf_version, pdf
             compression=compression,
             pdf_version=pdf_version,
             pdfa_part=pdfa_part,
+            progressbar_class=None,
         )
+        mock.assert_called()
         return output_file

tests/plugins/tesseract_badutf8.py

@@ -26,6 +26,7 @@ that is not UTF-8 compatible, so we are forced to check that we can convert it
 and present it to the user.
 """
 
+from contextlib import contextmanager
 from subprocess import CalledProcessError
 from unittest.mock import patch
 
@@ -42,17 +43,25 @@ def bad_utf8(*args, **kwargs):
     )
 
 
+@contextmanager
+def patch_tesseract_run():
+    with patch('ocrmypdf._exec.tesseract.run') as mock:
+        mock.side_effect = bad_utf8
+        yield
+        mock.assert_called()
+
+
 class BadUtf8OcrEngine(TesseractOcrEngine):
     @staticmethod
     def generate_hocr(input_file, output_hocr, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=bad_utf8):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_hocr(
                 input_file, output_hocr, output_text, options
             )
 
     @staticmethod
     def generate_pdf(input_file, output_pdf, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=bad_utf8):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_pdf(
                 input_file, output_pdf, output_text, options
             )

tests/plugins/tesseract_big_image_error.py

@@ -19,6 +19,7 @@
 # TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
 # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
+from contextlib import contextmanager
 from subprocess import CalledProcessError
 from unittest.mock import patch
 
@@ -35,22 +36,30 @@ def raise_size_exception(*args, **kwargs):
     )
 
 
+@contextmanager
+def patch_tesseract_run():
+    with patch('ocrmypdf._exec.tesseract.run') as mock:
+        mock.side_effect = raise_size_exception
+        yield
+        mock.assert_called()
+
+
 class BigImageErrorOcrEngine(TesseractOcrEngine):
     @staticmethod
     def get_orientation(input_file, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_size_exception):
+        with patch_tesseract_run():
             return TesseractOcrEngine.get_orientation(input_file, options)
 
     @staticmethod
     def generate_hocr(input_file, output_hocr, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_size_exception):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_hocr(
                 input_file, output_hocr, output_text, options
             )
 
     @staticmethod
     def generate_pdf(input_file, output_pdf, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_size_exception):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_pdf(
                 input_file, output_pdf, output_text, options
             )

tests/plugins/tesseract_cache.py

@@ -165,7 +165,7 @@ def cached_run(options, run_args, **run_kwargs):
 
     def clean_sys_argv():
         for arg in run_args[1:]:
-            yield re.sub(r'.*/com.github.ocrmypdf[^/]+[/](.*)', r'$TMPDIR/\1', arg)
+            yield re.sub(r'.*/ocrmypdf[.]io[.][^/]+[/](.*)', r'$TMPDIR/\1', arg)
 
     manifest['args'] = list(clean_sys_argv())
     with (Path(CACHE_ROOT) / 'manifest.jsonl').open('a') as f:

tests/plugins/tesseract_crash.py

@@ -20,6 +20,7 @@
 # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 import signal
+from contextlib import contextmanager
 from subprocess import CalledProcessError
 from unittest.mock import patch
 
@@ -37,22 +38,30 @@ def raise_crash(*args, **kwargs):
     )
 
 
+@contextmanager
+def patch_tesseract_run():
+    with patch('ocrmypdf._exec.tesseract.run') as mock:
+        mock.side_effect = raise_crash
+        yield
+        mock.assert_called()
+
+
 class CrashOcrEngine(TesseractOcrEngine):
     @staticmethod
     def get_orientation(input_file, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_crash):
+        with patch_tesseract_run():
             return TesseractOcrEngine.get_orientation(input_file, options)
 
     @staticmethod
     def generate_hocr(input_file, output_hocr, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_crash):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_hocr(
                 input_file, output_hocr, output_text, options
             )
 
     @staticmethod
     def generate_pdf(input_file, output_pdf, output_text, options):
-        with patch('ocrmypdf._exec.tesseract.run', new=raise_crash):
+        with patch_tesseract_run():
             TesseractOcrEngine.generate_pdf(
                 input_file, output_pdf, output_text, options
             )

tests/test_acroform.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 

tests/test_api.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 from io import BytesIO, StringIO

tests/test_check_pdf.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import pytest
 

tests/test_completion.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from subprocess import PIPE, run
 

tests/test_ghostscript.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 from decimal import Decimal

tests/test_graft.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from unittest.mock import patch
 

tests/test_helpers.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import logging
 import multiprocessing
@@ -22,8 +12,7 @@ from unittest.mock import MagicMock
 
 import pytest
 
-import ocrmypdf.helpers as helpers
-from ocrmypdf.subprocess import shim_paths_with_program_files
+from ocrmypdf import helpers as helpers
 
 
 class TestSafeSymlink:
@@ -47,12 +36,17 @@ class TestSafeSymlink:
 
 
 def test_no_cpu_count(monkeypatch):
+    invoked = False
+
     def cpu_count_raises():
+        nonlocal invoked
+        invoked = True
         raise NotImplementedError()
 
     monkeypatch.setattr(multiprocessing, 'cpu_count', cpu_count_raises)
     with pytest.warns(expected_warning=UserWarning):
         assert helpers.available_cpu_count() == 1
+    assert invoked, "Patched function called during test"
 
 
 def test_deprecated():
@@ -104,7 +98,10 @@ class TestFileIsWritable:
         assert not helpers.is_file_writable(pathmock)
 
 
+@pytest.mark.skipif(os.name != 'nt', reason="Windows test")
 def test_shim_paths(tmp_path):
+    from ocrmypdf.subprocess._windows import shim_env_path
+
     progfiles = tmp_path / 'Program Files'
     progfiles.mkdir()
     (progfiles / 'tesseract-ocr').mkdir()
@@ -113,9 +110,9 @@ def test_shim_paths(tmp_path):
     syspath = tmp_path / 'bin'
     env = {'PROGRAMFILES': str(progfiles), 'PATH': str(syspath)}
 
-    result_str = shim_paths_with_program_files(env=env)
+    result_str = shim_env_path(env=env)
     results = result_str.split(os.pathsep)
-    assert results[0].endswith('tesseract-ocr')
-    assert results[1].endswith(os.path.join('gs', '9.52', 'bin'))
-    assert results[2].endswith(os.path.join('gs', '9.51', 'bin'))
-    assert results[3] == str(syspath)
+    assert results[0] == str(syspath), results
+    assert results[-3].endswith('tesseract-ocr'), results
+    assert results[-2].endswith(os.path.join('gs', '9.52', 'bin')), results
+    assert results[-1].endswith(os.path.join('gs', '9.51', 'bin')), results

tests/test_hocrtransform.py

@@ -1,19 +1,9 @@
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import re
 from io import StringIO

tests/test_image_input.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from unittest.mock import patch
 
@@ -75,11 +65,12 @@ def test_cmyk_no_icc(caplog, resources, no_outpdf):
 def test_img2pdf_fails(resources, no_outpdf):
     with patch(
         'ocrmypdf._pipeline.img2pdf.convert', side_effect=img2pdf.ImageOpenError()
-    ):
+    ) as mock:
         rc = run_ocrmypdf_api(
             resources / 'baiona_gray.png', no_outpdf, '--image-dpi', '200'
         )
         assert rc == ocrmypdf.ExitCode.input_file
+        mock.assert_called()
 
 
 def test_jpeg_in_jpeg_out(resources, outpdf):

tests/test_lept.py

@@ -1,19 +1,8 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 from os import fspath

tests/test_logging.py

@@ -0,0 +1,21 @@
+# © 2021 James R. Barlow: github.com/jbarlow83
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import logging
+
+import pytest
+
+from ocrmypdf._sync import configure_debug_logging
+
+
+def test_debug_logging(tmp_path):
+    # Just exercise the debug logger but don't validate it
+    # See https://github.com/pytest-dev/pytest/issues/5502 for pytest logging quirks
+    prefix = 'test_debug_logging'
+    log = logging.getLogger(prefix)
+    handler = configure_debug_logging(tmp_path / 'test.log', prefix)
+    log.info("test message")
+    log.removeHandler(handler)

tests/test_main.py

@@ -1,19 +1,9 @@
 # © 2015-19 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import os
 import shutil
@@ -102,10 +92,10 @@ def test_skip_ocr(resources, outpdf):
 
 def test_redo_ocr(resources, outpdf):
     in_ = resources / 'graph_ocred.pdf'
-    before = PdfInfo(in_)
+    before = PdfInfo(in_, detailed_analysis=True)
     out = outpdf
     out = check_ocrmypdf(in_, out, '--redo-ocr')
-    after = PdfInfo(out)
+    after = PdfInfo(out, detailed_analysis=True)
     assert before[0].has_text and after[0].has_text
     assert (
         before[0].get_textareas() != after[0].get_textareas()
@@ -608,7 +598,7 @@ def test_compression_preserved(ocrmypdf_exec, resources, image, outpdf):
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
 
@@ -669,7 +659,7 @@ def test_compression_changed(ocrmypdf_exec, resources, image, compression, outpd
             stdout=PIPE,
             stderr=PIPE,
             stdin=input_stream,
-            universal_newlines=True,
+            universal_newlines=True,  # When dropping support for Python 3.6 change to text=
             check=False,
         )
         assert p.returncode == ExitCode.ok, p.stderr

tests/test_metadata.py

@@ -1,19 +1,8 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 
 import datetime
@@ -313,10 +302,13 @@ def test_kodak_toc(resources, outpdf):
 
     p = pikepdf.open(outpdf)
 
-    if pikepdf.Name.First in p.root.Outlines:
-        assert isinstance(p.root.Outlines.First, pikepdf.Dictionary)
+    if pikepdf.Name.First in p.Root.Outlines:
+        assert isinstance(p.Root.Outlines.First, pikepdf.Dictionary)
 
 
+@pytest.mark.skipif(
+    pikepdf.__version__ in ('2.2.2', '2.2.3'), reason="Raises wrong warning"
+)
 def test_metadata_fixup_warning(resources, outdir, caplog):
     options = get_parser().parse_args(
         args=['--output-type', 'pdfa-2', 'graph.pdf', 'out.pdf']
@@ -329,7 +321,7 @@ def test_metadata_fixup_warning(resources, outdir, caplog):
     )
     metadata_fixup(working_file=outdir / 'graph.pdf', context=context)
     for record in caplog.records:
-        assert record.levelname != 'WARNING'
+        assert record.levelname != 'WARNING', "Unexpected warning"
 
     # Now add some metadata that will not be copyable
     graph = pikepdf.open(outdir / 'graph.pdf')

tests/test_optimize.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from os import fspath
 from pathlib import Path
@@ -31,7 +21,15 @@ from ocrmypdf.helpers import Resolution
 
 check_ocrmypdf = pytest.helpers.check_ocrmypdf  # pylint: disable=e1101
 
+needs_pngquant = pytest.mark.skipif(
+    not pngquant.available(), reason="pngquant not installed"
+)
+needs_jbig2enc = pytest.mark.skipif(
+    not jbig2enc.available(), reason="jbig2enc not installed"
+)
 
+
+@needs_pngquant
 @pytest.mark.parametrize('pdf', ['multipage.pdf', 'palette.pdf'])
 def test_basic(resources, pdf, outpdf):
     infile = resources / pdf
@@ -40,6 +38,7 @@ def test_basic(resources, pdf, outpdf):
     assert 0.98 * Path(outpdf).stat().st_size <= Path(infile).stat().st_size
 
 
+@needs_pngquant
 def test_mono_not_inverted(resources, outdir):
     infile = resources / '2400dpi.pdf'
     opt.main(infile, outdir / 'out.pdf', level=3)
@@ -55,7 +54,7 @@ def test_mono_not_inverted(resources, outdir):
         assert im.getpixel((0, 0)) == 255, "Expected white background"
 
 
-@pytest.mark.skipif(not pngquant.available(), reason='need pngquant')
+@needs_pngquant
 def test_jpg_png_params(resources, outpdf):
     check_ocrmypdf(
         resources / 'crom.png',
@@ -73,7 +72,7 @@ def test_jpg_png_params(resources, outpdf):
     )
 
 
-@pytest.mark.skipif(not jbig2enc.available(), reason='need jbig2enc')
+@needs_jbig2enc
 @pytest.mark.parametrize('lossy', [False, True])
 def test_jbig2_lossy(lossy, resources, outpdf):
     args = [
@@ -105,10 +104,8 @@ def test_jbig2_lossy(lossy, resources, outpdf):
         assert len(pim.decode_parms) == 0
 
 
-@pytest.mark.skipif(
-    not jbig2enc.available() or not pngquant.available(),
-    reason='need jbig2enc and pngquant',
-)
+@needs_pngquant
+@needs_jbig2enc
 def test_flate_to_jbig2(resources, outdir):
     # This test requires an image that pngquant is capable of converting to
     # to 1bpp - so use an existing 1bpp image, convert up, confirm it can
@@ -136,6 +133,7 @@ def test_flate_to_jbig2(resources, outdir):
     assert pim.filters[0] == '/JBIG2Decode'
 
 
+@needs_pngquant
 def test_multiple_pngs(resources, outdir):
     with Path.open(outdir / 'in.pdf', 'wb') as inpdf:
         img2pdf.convert(
@@ -151,7 +149,8 @@ def test_multiple_pngs(resources, outdir):
             draw.rectangle((0, 0, im.width, im.height), fill=128)
             im.save(output_file)
 
-    with patch('ocrmypdf.optimize.pngquant.quantize', new=mockquant):
+    with patch('ocrmypdf.optimize.pngquant.quantize') as mock:
+        mock.side_effect = mockquant
         check_ocrmypdf(
             outdir / 'in.pdf',
             outdir / 'out.pdf',
@@ -165,6 +164,7 @@ def test_multiple_pngs(resources, outdir):
             '--plugin',
             'tests/plugins/tesseract_noop.py',
         )
+        mock.assert_called()
 
     with pikepdf.open(outdir / 'in.pdf') as inpdf, pikepdf.open(
         outdir / 'out.pdf'
@@ -173,3 +173,15 @@ def test_multiple_pngs(resources, outdir):
             inim = next(iter(inpdf.pages[n].images.values()))
             outim = next(iter(outpdf.pages[n].images.values()))
             assert len(outim.read_raw_bytes()) < len(inim.read_raw_bytes()), n
+
+
+def test_optimize_off(resources, outpdf):
+    check_ocrmypdf(
+        resources / 'trivial.pdf',
+        outpdf,
+        '--optimize=0',
+        '--output-type',
+        'pdf',
+        '--plugin',
+        'tests/plugins/tesseract_noop.py',
+    )

tests/test_page_numbers.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import pytest
 
@@ -38,6 +28,12 @@ from ocrmypdf.pdfinfo import PdfInfo
         ['1,3,-11', BadArgsError],
         ['1-,', BadArgsError],
         ['start-end', BadArgsError],
+        ['1-0', BadArgsError],
+        ['99-98', BadArgsError],
+        ['0-0', BadArgsError],
+        ['1-0,3-4', BadArgsError],
+        [',', BadArgsError],
+        ['', BadArgsError],
     ],
 )
 def test_pages(pages, result):

tests/test_pdfinfo.py

@@ -1,41 +1,34 @@
 # © 2015 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
 import pickle
+from io import BytesIO
 from math import isclose
 
 import img2pdf
 import pikepdf
 import pytest
 from PIL import Image
+from reportlab.lib.units import inch
 from reportlab.pdfgen.canvas import Canvas
 
 from ocrmypdf import pdfinfo
+from ocrmypdf.exceptions import InputFileError
 from ocrmypdf.pdfinfo import Colorspace, Encoding
+from ocrmypdf.pdfinfo.layout import PDFPage
 
 # pylint: disable=protected-access
 
 
 def test_single_page_text(outdir):
     filename = outdir / 'text.pdf'
-    pdf = Canvas(str(filename), pagesize=(8 * 72, 6 * 72))
+    pdf = Canvas(str(filename), pagesize=(8 * inch, 6 * inch))
     text = pdf.beginText()
     text.setFont('Helvetica', 12)
-    text.setTextOrigin(1 * 72, 3 * 72)
+    text.setTextOrigin(1 * inch, 3 * inch)
     text.textLine(
         "Methink'st thou art a general offence and every" " man should beat thee."
     )
@@ -52,25 +45,32 @@ def test_single_page_text(outdir):
     assert len(page.images) == 0
 
 
-def test_single_page_image(outdir):
-    filename = outdir / 'image-mono.pdf'
-
-    im_tmp = outdir / 'tmp.png'
+@pytest.fixture(scope='session')
+def eight_by_eight():
     im = Image.new('1', (8, 8), 0)
     for n in range(8):
         im.putpixel((n, n), 1)
-    im.save(str(im_tmp), format='PNG')
+    return im
+
+
+def test_single_page_image(eight_by_eight, outpdf):
+    im = eight_by_eight
+    bio = BytesIO()
+    im.save(bio, format='PNG')
+    bio.seek(0)
 
     imgsize = ((img2pdf.ImgSize.dpi, 8), (img2pdf.ImgSize.dpi, 8))
     layout_fun = img2pdf.get_layout_fun(None, imgsize, None, None, None)
 
-    im_bytes = im_tmp.read_bytes()
-    pdf_bytes = img2pdf.convert(
-        im_bytes, producer="img2pdf", with_pdfrw=False, layout_fun=layout_fun
-    )
-    filename.write_bytes(pdf_bytes)
-
-    info = pdfinfo.PdfInfo(filename)
+    with outpdf.open('wb') as f:
+        img2pdf.convert(
+            bio,
+            producer="img2pdf",
+            with_pdfrw=False,
+            layout_fun=layout_fun,
+            outputstream=f,
+        )
+    info = pdfinfo.PdfInfo(outpdf)
 
     assert len(info) == 1
     page = info[0]
@@ -87,16 +87,12 @@ def test_single_page_image(outdir):
     assert isclose(pdfimage.dpi.y, 8)
 
 
-def test_single_page_inline_image(outdir):
+def test_single_page_inline_image(eight_by_eight, outdir):
     filename = outdir / 'image-mono-inline.pdf'
     pdf = Canvas(str(filename), pagesize=(8 * 72, 6 * 72))
 
-    im = Image.new('1', (8, 8), 0)
-    for n in range(8):
-        im.putpixel((n, n), 1)
-
     # Draw image in a 72x72 pt or 1"x1" area
-    pdf.drawInlineImage(im, 0, 0, width=72, height=72)
+    pdf.drawInlineImage(eight_by_eight, 0, 0, width=72, height=72)
     pdf.showPage()
     pdf.save()
 
@@ -189,3 +185,18 @@ def test_stack_abuse():
     with pytest.warns(None):
         with pytest.raises(RuntimeError):
             pdfinfo.info._interpret_contents(stream)
+
+
+def test_pages_issue700(monkeypatch, resources):
+    def get_no_pages(*args, **kwargs):
+        return iter([])
+
+    monkeypatch.setattr(PDFPage, 'get_pages', get_no_pages)
+
+    with pytest.raises(InputFileError, match="pdfminer"):
+        pdfinfo.PdfInfo(
+            resources / 'cardinal.pdf',
+            detailed_analysis=True,
+            progbar=False,
+            max_workers=1,
+        )

tests/test_pipeline.py

@@ -0,0 +1,63 @@
+# © 2021 James R. Barlow: github.com/jbarlow83
+#
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from unittest.mock import Mock
+
+import pytest
+from PIL import Image
+from reportlab.lib.units import inch
+from reportlab.lib.utils import ImageReader
+from reportlab.pdfgen.canvas import Canvas
+
+from ocrmypdf import _pipeline, pdfinfo
+from ocrmypdf.helpers import Resolution
+
+
+@pytest.fixture(scope='session')
+def rgb_image():
+    im = Image.new('RGB', (8, 8))
+    im.putpixel((4, 4), (255, 0, 0))
+    im.putpixel((5, 5), (0, 255, 0))
+    im.putpixel((6, 6), (0, 0, 255))
+    return ImageReader(im)
+
+
+DUMMY_OVERSAMPLE_RESOLUTION = Resolution(42.0, 42.0)
+VECTOR_RESOLUTION = Resolution(_pipeline.VECTOR_PAGE_DPI, _pipeline.VECTOR_PAGE_DPI)
+
+
+@pytest.mark.parametrize(
+    'image, text, vector, result',
+    [
+        (False, False, False, VECTOR_RESOLUTION),
+        (False, True, False, VECTOR_RESOLUTION),
+        (True, False, False, DUMMY_OVERSAMPLE_RESOLUTION),
+        (True, True, False, VECTOR_RESOLUTION),
+        (False, False, True, VECTOR_RESOLUTION),
+        (False, True, True, VECTOR_RESOLUTION),
+        (True, False, True, VECTOR_RESOLUTION),
+        (True, True, True, VECTOR_RESOLUTION),
+    ],
+)
+def test_dpi_needed(image, text, vector, result, rgb_image, outdir):
+
+    c = Canvas(str(outdir / 'dpi.pdf'), pagesize=(5 * inch, 5 * inch))
+    if image:
+        c.drawImage(rgb_image, 1 * inch, 1 * inch, width=1 * inch, height=1 * inch)
+    if text:
+        c.drawString(1 * inch, 4 * inch, "Actual text")
+    if vector:
+        c.ellipse(3 * inch, 3 * inch, 4 * inch, 4 * inch)
+    c.showPage()
+    c.save()
+
+    mock = Mock()
+    mock.oversample = DUMMY_OVERSAMPLE_RESOLUTION[0]
+
+    pi = pdfinfo.PdfInfo(outdir / 'dpi.pdf')
+
+    assert _pipeline.get_canvas_square_dpi(pi[0], mock) == result
+    assert _pipeline.get_page_square_dpi(pi[0], mock) == result

tests/test_preprocessing.py

@@ -1,19 +1,9 @@
 # © 2019 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from math import isclose
 

tests/test_quality.py

@@ -1,23 +1,13 @@
 # © 2020 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 import pytest
 
-import ocrmypdf.quality as qual
+from ocrmypdf import quality as qual
 
 
 def test_quality_measurement():

tests/test_rotation.py

@@ -1,19 +1,9 @@
 # © 2018 James R. Barlow: github.com/jbarlow83
 #
-# This file is part of OCRmyPDF.
-#
-# OCRmyPDF is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# OCRmyPDF is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with OCRmyPDF.  If not, see <http://www.gnu.org/licenses/>.
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this
+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
 
 from io import BytesIO
 from os import fspath
@@ -25,6 +15,7 @@ from PIL import Image
 
 from ocrmypdf import leptonica
 from ocrmypdf._exec import ghostscript, tesseract
+from ocrmypdf._plugin_manager import get_plugin_manager
 from ocrmypdf.helpers import Resolution
 from ocrmypdf.pdfinfo import PdfInfo
 
@@ -250,7 +241,7 @@ def test_rotate_page_level(image_angle, page_angle, resources, outdir):
         '--rotate-pages',
         '--rotate-pages-threshold',
         '0.001',
-        universal_newlines=False,
+        text=False,
     )
     err = err.decode('utf-8', errors='replace')
     assert p.returncode == 0, err
@@ -266,3 +257,33 @@ def test_tesseract_orientation(resources, tmp_path):
     tesseract.get_orientation(  # Test results of this are unreliable
         tmp_path / '000001.png', engine_mode='3', timeout=10
     )
+
+
+def test_rasterize_rotates(resources, tmp_path):
+    pm = get_plugin_manager([])
+
+    img = tmp_path / 'img90.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=90,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (123, 151), "Image not rotated"
+
+    img = tmp_path / 'img180.png'
+    pm.hook.rasterize_pdf_page(
+        input_file=resources / 'graph.pdf',
+        output_file=img,
+        raster_device='pngmono',
+        raster_dpi=Resolution(20, 20),
+        page_dpi=Resolution(20, 20),
+        pageno=1,
+        rotation=180,
+        filter_vector=False,
+    )
+    assert Image.open(img).size == (151, 123), "Image not rotated"