v14.0.4 release notes

Co-authored-by: Andrew Wood <andrew@fluidgravity.co.uk>

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,5 +1,3 @@
-# SPDX-FileCopyrightText: 2022 James R. Barlow
-# SPDX-License-Identifier: CC-BY-SA-4.0
 ---
 name: Feature request
 about: Suggest an idea for this project
@@ -19,11 +17,7 @@ A clear and concise description of what you want to happen.
 A clear and concise description of any alternative solutions or features you've considered. Please include the versions of OCRmyPDF and other supporting programs (Tesseract OCR, Ghostscript) - maybe an alternative already exists in a newer version.
 
 **Example file**
-If your issue concerns how OCRmyPDF processes certain files, and please provide an example file that helps illustrate how OCRmyPDF's output could be improve.
-
-Please provide an input file with no personal or confidential information. At your option you may [GPG-encrypt the file](https://github.com/ocrmypdf/OCRmyPDF/wiki) for OCRmyPDF's author only.
-
-Links to files hosted elsewhere are perfectly acceptable. You could also look in ``tests/resources`` and see if any of those files reproduce your issue.
+If your issue concerns how OCRmyPDF processes certain files, and please provide an example file that helps illustrate how OCRmyPDF's output could be improve. You could also look in ``tests/resources`` and see if any of those files demonstrates your issue.
 
 **Additional context**
 Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/general-issues.md

@@ -1,9 +1,7 @@
-# SPDX-FileCopyrightText: 2022 James R. Barlow
-# SPDX-License-Identifier: CC-BY-SA-4.0
 ---
 name: General issues
 about: Installation, packages, dependencies, "nothing works", test suite failures...
-title: ''
+title: "[BUG]"
 labels: ''
 assignees: ''
 
@@ -25,6 +23,7 @@ If applicable, add screenshots to help explain your problem.
  - OS:
  - Python version:
  - OCRmyPDF version:
+ - Platform: x64 or ARM
 
 **Installation**
 How did you install OCRmyPDF? Did you install it from your operating system's

.github/ISSUE_TEMPLATE/problem-with-specific-file.md

@@ -1,9 +1,7 @@
-# SPDX-FileCopyrightText: 2022 James R. Barlow
-# SPDX-License-Identifier: CC-BY-SA-4.0
 ---
-name: Problem with a specific input file
+name: Problem with specific file
 about: Something went wrong while trying to OCR a specific file
-title: ''
+title: "[BUG]"
 labels: ''
 assignees: ''
 

.github/workflows/build.yml

@@ -33,7 +33,7 @@ jobs:
           - os: ubuntu-latest
             python: "3.10"
           - os: ubuntu-latest
-            python: "3.11.0-rc.2 - 3.11.0"
+            python: "3.11"
           # - os: ubuntu-latest
           #   python: "pypy3.8"
           #- os: ubuntu-latest
@@ -122,7 +122,7 @@ jobs:
     strategy:
       matrix:
         os: [macos-latest]
-        python: ["3.9", "3.10"] # 2022-09-20: pikepdf/lxml not available for cp311
+        python: ["3.10", "3.11"]
 
     env:
       OS: ${{ matrix.os }}
@@ -133,12 +133,8 @@ jobs:
         with:
           fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
 
-      - uses: actions/setup-python@v4
-        name: Install Python
-        with:
-          python-version: ${{ matrix.python }}
-
       - name: Install Homebrew deps
+        continue-on-error: true
         run: |
           brew update
           brew install \
@@ -149,6 +145,11 @@ jobs:
             pngquant \
             tesseract
 
+      - uses: actions/setup-python@v4
+        name: Install Python
+        with:
+          python-version: ${{ matrix.python }}
+
       - name: Install Python packages
         run: |
           python -m pip install --upgrade pip wheel
@@ -177,7 +178,7 @@ jobs:
     strategy:
       matrix:
         os: [windows-latest]
-        python: ["3.9", "3.10"] # 2022-09-20: pikepdf/lxml not available for cp311
+        python: ["3.10", "3.11"]
 
     env:
       OS: ${{ matrix.os }}

.reuse/dep5

@@ -16,6 +16,7 @@ Copyright: (C) 2022 James R. Barlow
 License: MPL-2.0
 
 Files:
+ .github/ISSUE_TEMPLATE/*.md
  docs/images/macos-workflow.png
 Copyright: (C) 2022 James R. Barlow
 License: CC-BY-SA-4.0

docs/advanced.rst

@@ -160,7 +160,7 @@ Changing tesseract configuration variables
 ------------------------------------------
 
 You can override tesseract's default `control
-parameters <https://github.com/tesseract-ocr/tesseract/wiki/ControlParams>`__
+parameters <https://tesseract-ocr.github.io/tessdoc/tess3/ControlParams.html>`__
 with a configuration file.
 
 As an example, this configuration will disable Tesseract's dictionary

docs/batch.rst

@@ -21,8 +21,8 @@ processors. To maximize parallelism without overloading your system with
 processes, consider using ``parallel -j 2`` to limit parallel to running
 two jobs at once.
 
-This command will run all ocrmypdf all files named ``*.pdf`` in the
-current directory and write them to the previous created ``output/``
+This command will run ``ocrmypdf`` on all files named ``*.pdf`` in the
+current directory and write them to the previously created ``output/``
 folder. It will not search subdirectories.
 
 The ``--tag`` argument tells parallel to print the filename as a prefix

docs/installation.rst

@@ -21,7 +21,7 @@ These platforms have one-liner installs:
 +-------------------------------+-----------------------------------------+
 | Windows Subsystem for Linux   | ``apt install ocrmypdf``                |
 +-------------------------------+-----------------------------------------+
-| Fedora                        | ``dnf install ocrmypdf``                |
+| Fedora                        | ``dnf install ocrmypdf tesseract-osd``  |
 +-------------------------------+-----------------------------------------+
 | macOS                         | ``brew install ocrmypdf``               |
 +-------------------------------+-----------------------------------------+
@@ -121,7 +121,7 @@ Users of Fedora 29 or later may simply
 
 .. code-block:: bash
 
-    dnf install ocrmypdf
+    dnf install ocrmypdf tesseract-osd
 
 For full details on version availability, check the `Fedora Package
 Tracker <https://apps.fedoraproject.org/packages/ocrmypdf>`__.
@@ -293,7 +293,7 @@ Homebrew
 
 .. image:: https://img.shields.io/homebrew/v/ocrmypdf.svg
     :alt: homebrew
-    :target: http://brewformulas.org/Ocrmypdf
+    :target: https://formulae.brew.sh/formula/ocrmypdf
 
 OCRmyPDF is now a standard `Homebrew <https://brew.sh>`__ formula. To
 install on macOS:
@@ -375,8 +375,8 @@ Native Windows
 You must install the following for Windows:
 
 * Python 3.8 (64-bit) or later
-* Tesseract 4.1.1 or later
-* Ghostscript 9.50 or later
+* Tesseract 4.1.1 (64-bit) or later
+* Ghostscript 9.50 (64-bit) or later
 
 Using the `Chocolatey <https://chocolatey.org/>`_ package manager, install the
 following when running in an Administrator command prompt:
@@ -395,10 +395,8 @@ Administrator.):
 
 * ``pip install ocrmypdf``
 
-Chocolatey automatically selects appropriate versions of these applications. If you
-are installing them manually, please install 64-bit versions of all applications for
-64-bit Windows, or 32-bit versions of all applications for 32-bit Windows. Mixing
-the "bitness" of these programs will lead to errors.
+Chocolatey automatically selects appropriate versions of these applications. Please make sure
+you are installing the 64-bit versions.
 
 OCRmyPDF will check the Windows Registry and standard locations in your Program Files
 for third party software it needs (specifically, Tesseract and Ghostscript). To
@@ -413,6 +411,10 @@ to change the PATH.
     Please download Python from Python.org or Chocolatey instead, and do not use the
     Microsoft Store version.
 
+.. warning::
+
+    32-bit Windows might work, but is not supported.
+
 Windows Subsystem for Linux
 ---------------------------
 
@@ -636,7 +638,7 @@ system-wide:
 .. code-block:: bash
 
     git clone -b master https://github.com/ocrmypdf/OCRmyPDF.git
-    python3 -m venv
+    python3 -m venv venv
     source venv/bin/activate
     cd OCRmyPDF
     pip install .

docs/introduction.rst

@@ -85,7 +85,7 @@ OCRmyPDF analyzes each page of a PDF to determine the colorspace and
 resolution (DPI) needed to capture all of the information on that page
 without losing content. It uses
 `Ghostscript <http://ghostscript.com/>`__ to rasterize the page, and
-then performs on OCR the rasterized image to create an OCR "layer".
+then performs OCR on the rasterized image to create an OCR "layer".
 The layer is then grafted back onto the original PDF.
 
 While one can use a program like Ghostscript or ImageMagick to get an

docs/release_notes.rst

@@ -28,6 +28,25 @@ tagged yet.
 
 .. |OCRmyPDF PyPI| image:: https://img.shields.io/pypi/v/ocrmypdf.svg
 
+v14.0.4
+=======
+
+-  Fixed :issue:`1066, 1075`, an exception when processing certain malformed PDFs.
+
+v14.0.3
+=======
+
+-  Fixed :issue:`1068`, avoid deleting /dev/null when running as root.
+-  Other documentation fixes.
+
+v14.0.2
+=======
+
+-  Fixed :issue:`1052`, an exception on attempting to process certain nonconforming PDFs.
+-  Explicitly documented that Windows 32-bit is no longer supported.
+-  Fixed source installation instructions.
+-  Other documentation fixes.
+
 v14.0.1
 =======
 

misc/watcher.py

@@ -38,6 +38,7 @@ DESKEW = getenv_bool('OCR_DESKEW')
 OCR_JSON_SETTINGS = json.loads(os.getenv('OCR_JSON_SETTINGS', '{}'))
 POLL_NEW_FILE_SECONDS = int(os.getenv('OCR_POLL_NEW_FILE_SECONDS', '1'))
 USE_POLLING = getenv_bool('OCR_USE_POLLING')
+RETRIES_LOADING_FILE = int(os.getenv('OCR_RETRIES_LOADING_FILE', '5'))
 LOGLEVEL = os.getenv('OCR_LOGLEVEL', 'INFO')
 PATTERNS = ['*.pdf', '*.PDF']
 
@@ -64,7 +65,7 @@ def wait_for_file_ready(file_path):
     # watchdog event before the file is actually fully on disk, causing
     # pikepdf to fail.
 
-    retries = 5
+    retries = RETRIES_LOADING_FILE
     while retries:
         try:
             pdf = pikepdf.open(file_path)
@@ -103,6 +104,8 @@ def execute_ocrmypdf(file_path):
         elif ON_SUCCESS_ARCHIVE:
             log.info(f'OCR is done. Archiving {file_path.name} to {ARCHIVE_DIRECTORY}')
             shutil.move(file_path, f'{ARCHIVE_DIRECTORY}/{file_path.name}')
+        else:
+            log.info('OCR is done')
     else:
         log.info('OCR is done')
 
@@ -140,6 +143,7 @@ def main():
         f"DESKEW: {DESKEW}\n"
         f"ARGS: {OCR_JSON_SETTINGS}\n"
         f"POLL_NEW_FILE_SECONDS: {POLL_NEW_FILE_SECONDS}\n"
+        f"RETRIES_LOADING_FILE: {RETRIES_LOADING_FILE}\n"
         f"USE_POLLING: {USE_POLLING}\n"
         f"LOGLEVEL: {LOGLEVEL}"
     )

src/ocrmypdf/helpers.py

@@ -199,7 +199,7 @@ def is_file_writable(test_file: os.PathLike) -> bool:
             p = p.resolve(strict=False)
 
         # p.is_file() throws an exception in some cases
-        if p.exists() and p.is_file():
+        if p.exists() and (p.is_file() or p.samefile(os.devnull)):
             return os.access(
                 os.fspath(p),
                 os.W_OK,

src/ocrmypdf/pdfinfo/info.py

@@ -477,14 +477,12 @@ def _image_xobjects(container) -> Iterator[tuple[Object, str]]:
     resources = container['/Resources']
     if '/XObject' not in resources:
         return
-    xobjs = resources['/XObject'].as_dict()
-    for xobj in xobjs:
-        candidate: Object = xobjs[xobj]
-        if '/Subtype' not in candidate:
+    for key, candidate in resources['/XObject'].items():
+        if candidate is None or '/Subtype' not in candidate:
             continue
         if candidate['/Subtype'] == '/Image':
             pdfimage = candidate
-            yield (pdfimage, xobj)
+            yield (pdfimage, key)
 
 
 def _find_regular_images(
@@ -527,7 +525,7 @@ def _find_form_xobject_images(pdf: Pdf, container: Object, contentsinfo: Content
     xobjs = resources['/XObject'].as_dict()
     for xobj in xobjs:
         candidate = xobjs[xobj]
-        if candidate['/Subtype'] != '/Form':
+        if candidate is None or candidate['/Subtype'] != '/Form':
             continue
 
         form_xobject = candidate