v15.0.2 release notes

Maintainers of ARM 32-bit in particular don't necessarily need to drop support....

.github/workflows/build.yml

@@ -28,11 +28,14 @@ jobs:
             python: "3.10"
           - os: ubuntu-22.04
             python: "3.11"
-          #- os: ubuntu-latest
-          #  python: "pypy3.9"
           - os: ubuntu-22.04
             python: "3.9"
             tesseract5: true
+          - os: ubuntu-latest
+            python: "3.12-dev"
+            tesseract5: true
+            #- os: ubuntu-latest
+            #  python: "pypy3.9"
 
     env:
       OS: ${{ matrix.os }}
@@ -44,9 +47,10 @@ jobs:
           fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
 
       - uses: actions/setup-python@v4
-        name: Install Python
+        name: Setup Python
         with:
           python-version: ${{ matrix.python }}
+          cache: "pip"
 
       - name: Install Tesseract 5
         if: matrix.tesseract5
@@ -109,7 +113,7 @@ jobs:
     strategy:
       matrix:
         os: [macos-latest]
-        python: ["3.10", "3.11"]
+        python: ["3.10", "3.11", "3.12-dev"]
 
     env:
       OS: ${{ matrix.os }}
@@ -133,9 +137,10 @@ jobs:
             tesseract
 
       - uses: actions/setup-python@v4
-        name: Install Python
+        name: Setup Python
         with:
           python-version: ${{ matrix.python }}
+          cache: "pip"
 
       - name: Install Python packages
         run: |
@@ -165,7 +170,7 @@ jobs:
     strategy:
       matrix:
         os: [windows-latest]
-        python: ["3.10", "3.11"]
+        python: ["3.10", "3.11", "3.12-dev"]
 
     env:
       OS: ${{ matrix.os }}
@@ -177,9 +182,10 @@ jobs:
           fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
 
       - uses: actions/setup-python@v4
-        name: Install Python
+        name: Setup Python
         with:
           python-version: ${{ matrix.python }}
+          cache: "pip"
 
       - name: Install system packages
         run: |
@@ -210,9 +216,10 @@ jobs:
           fetch-depth: "0" # 0=all, needed for setuptools-scm to resolve version tags
 
       - uses: actions/setup-python@v4
-        name: Install Python
+        name: Setup Python
         with:
           python-version: "3.9"
+          cache: "pip"
 
       - name: Make wheels and sdist
         run: |

docs/advanced.rst

@@ -117,14 +117,14 @@ exceed a certain number of megapixels with ``--skip-big``. (A 300 DPI,
 OCR for huge images
 -------------------
 
-Separate from these settings, Tesseract has internal limits on the size
+Tesseract has internal limits on the size
 of images it will process. If you issue
 ``--tesseract-downsample-large-images``, OCRmyPDF will downsample images
 to fit Tesseract limits. (The limits are usually entered only for scanned
 images of oversized media, such as large maps or blueprints exceeding
 110 cm or 43 inches in either dimension, and at high DPI.)
 
-``--tesseract-downsample-above`` adjusts the threshold at which images
+``--tesseract-downsample-above Npixels`` adjusts the threshold at which images
 will be downsampled. By default, only images that exceed any of Tesseract's
 internal limits are downsampled.
 
@@ -195,10 +195,10 @@ In each case OCRmyPDF will search the ``PATH`` environment variable to
 locate the binaries. By modifying the ``PATH`` environment variable, you
 can override the binaries that OCRmyPDF uses.
 
-Changing tesseract configuration variables
+Changing Tesseract configuration variables
 ------------------------------------------
 
-You can override tesseract's default `control
+You can override Tesseract's default `control
 parameters <https://tesseract-ocr.github.io/tessdoc/tess3/ControlParams.html>`__
 with a configuration file.
 
@@ -273,7 +273,7 @@ Unlike ``sandwich`` this renderer is implemented within OCRmyPDF; anyone
 looking to customize how OCR is presented should look here. A major
 disadvantage of this renderer is it not capable of correctly handling
 text outside the Latin alphabet (specifically, it supports the ISO 8859-1
-character). Pull requests to improve the situation are welcome.
+character set). Pull requests to improve the situation are welcome.
 
 Currently, this renderer has the best compatibility with Mozilla's
 PDF.js viewer.
@@ -286,11 +286,31 @@ Rendering and rasterizing options
 .. versionadded:: 14.3.0
 
 The ``--continue-on-soft-render-error`` option allows OCRmyPDF to
-proceed if a page cannot be rasterized rendered. This is useful if you are
+proceed if a page cannot be rasterized/rendered. This is useful if you are
 trying to get the best possible OCR from a PDF that is not well-formed,
 and you are willing to accept some pages that may not visually match the
 input, and that may not OCR well.
 
+Color conversion strategy
+=========================
+
+.. versionadded:: 15.0.0
+
+OCRmyPDF uses Ghostscript to convert PDF to PDF/A. In some cases, this
+conversion requires color conversion. The default strategy is to convert
+using the ``LeaveColorUnchanged`` strategy, which preserves the original
+color space wherever possible (some rare color spaces might still be
+converted).
+
+Usually document scanners produce PDFs in the sRGB color space, and do
+not need to be converted, so the default strategy is appropriate.
+
+Suppose that you have a document that was prepared for professional
+printing in a Separation or CMYK color space, and text was converted to
+curves. In this case, you may want to use a different color conversion
+strategy. The ``--color-conversion-strategy`` option allows you to select a
+different strategy, such as ``RGB``.
+
 Return code policy
 ==================
 

docs/index.rst

@@ -12,7 +12,7 @@ files, allowing them to be searched.
 
 PDF is the best format for storing and exchanging scanned documents.
 Unfortunately, PDFs can be difficult to modify. OCRmyPDF makes it easy to apply
-image processing and OCR to existing PDFs.
+image processing and OCR (recognized, searchable text) to existing PDFs.
 
 .. toctree::
    :maxdepth: 1

docs/installation.rst

@@ -72,7 +72,7 @@ Debian and Ubuntu 20.04 or newer
 | |ubu-2004| |ubu-2204|                         |
 +-----------------------------------------------+
 
-Users of Debian 11, or Ubuntu 20.04 LTS, or newer may simply
+Users of Debian or Ubuntu may simply
 
 .. code-block:: bash
 
@@ -99,11 +99,11 @@ For full details on version availability for your platform, check the
 Fedora
 ------
 
-.. |fedora-35| image:: https://repology.org/badge/version-for-repo/fedora_35/ocrmypdf.svg
+.. |fedora-37| image:: https://repology.org/badge/version-for-repo/fedora_37/ocrmypdf.svg
-    :alt: Fedora 35
+    :alt: Fedora 37
 
-.. |fedora-36| image:: https://repology.org/badge/version-for-repo/fedora_36/ocrmypdf.svg
+.. |fedora-38| image:: https://repology.org/badge/version-for-repo/fedora_38/ocrmypdf.svg
-    :alt: Fedora 36
+    :alt: Fedora 38
 
 .. |fedora-rawhide| image:: https://repology.org/badge/version-for-repo/fedora_rawhide/ocrmypdf.svg
     :alt: Fedore Rawhide
@@ -113,10 +113,10 @@ Fedora
 +-----------------------------------------------+
 | |latest|                                      |
 +-----------------------------------------------+
-| |fedora-35| |fedora-36| |fedora-rawhide|      |
+| |fedora-37| |fedora-38| |fedora-rawhide|      |
 +-----------------------------------------------+
 
-Users of Fedora 29 or later may simply
+Users of Fedora may simply
 
 .. code-block:: bash
 
@@ -155,7 +155,7 @@ user, follow these steps:
 
 If you get the message ``WARNING: The script ocrmypdf is installed in
 '/home/$USER/.local/bin' which is not on PATH.``, you may need to re-login
-or open a new shell, or manually add this to your user's PATH.
+or open a new shell, or manually adjust your PATH.
 
 To add JBIG2 encoding, see :ref:`jbig2`.
 
@@ -382,29 +382,33 @@ Native Windows
 
 You must install the following for Windows:
 
-* Python 3.9 (64-bit) or later
+* Python 64-bit
-* Tesseract 4.1.1 (64-bit) or later
+* Tesseract 64-bit
-* Ghostscript 9.50 (64-bit) or later
+* Ghostscript 64-bit
 
-Using the `Chocolatey <https://chocolatey.org/>`_ package manager, install the
+Using the `winget <https://docs.microsoft.com/en-us/windows/package-manager/winget/>`_
-following when running in an Administrator command prompt:
+package manager:
+
+* ``winget install -e --id Python.Python.3.11``
+* ``winget install -e --id UB-Mannheim.TesseractOCR``
+* ``winget install -e --id ArtifexSoftware.GhostScript``
+
+
+(Or alternately, using the `Chocolatey <https://chocolatey.org/>`_ package manager, install
+the following when running in an Administrator command prompt):
 
 * ``choco install python3``
 * ``choco install --pre tesseract``
 * ``choco install ghostscript``
 * ``choco install pngquant`` (optional)
 
-The commands above will install Python 3.x (latest version), Tesseract, Ghostscript
+Either set of commands will install the required software. At the mmoment there is no
-and pngquant. Chocolatey may also need to install the Windows Visual C++ Runtime
+single command to install Windows.
-DLLs or other Windows patches, and may require a reboot.
 
 You may then use ``pip`` to install ocrmypdf. (This can performed by a user or
 Administrator.):
 
-* ``pip install ocrmypdf``
+* ``python3 -m pip install ocrmypdf``
-
-Chocolatey automatically selects appropriate versions of these applications. Please make sure
-you are installing the 64-bit versions.
 
 OCRmyPDF will check the Windows Registry and standard locations in your Program Files
 for third party software it needs (specifically, Tesseract and Ghostscript). To
@@ -416,12 +420,12 @@ to change the PATH.
 
     As of early 2021, users have reported problems with the Microsoft Store version of
     Python and OCRmyPDF. These issues affect many other third party Python packages.
-    Please download Python from Python.org or Chocolatey instead, and do not use the
+    Please download Python from Python.org or a package manager instead of the
     Microsoft Store version.
 
 .. warning::
 
-    32-bit Windows might work, but is not supported.
+    32-bit Windows is not supported.
 
 Windows Subsystem for Linux
 ---------------------------
@@ -448,7 +452,7 @@ Cygwin64
 
 First install the the following prerequisite Cygwin packages using ``setup-x86_64.exe``::
 
-    python38 (or later)
+    python39 (or later)
     python3?-devel
     python3?-pip
     python3?-lxml
@@ -457,7 +461,7 @@ First install the the following prerequisite Cygwin packages using ``setup-x86_6
        (where 3? means match the version of python3 you installed)
 
     gcc-g++
-    ghostscript (<=9.50 or >=9.52-2 see note below)
+    ghostscript
     libexempi3
     libexempi-devel
     libffi6
@@ -468,13 +472,6 @@ First install the the following prerequisite Cygwin packages using ``setup-x86_6
     tesseract-ocr
     tesseract-ocr-devel
 
-.. note::
-
-    The Cygwin package for Ghostscript in versions 9.52 and
-    9.52-1 contained a bug that caused an exception to occur when
-    ocrmypdf invoked gs.  Make sure you have either 9.50 (or earlier)
-    or 9.52-2 (or later).
-
 Then open a Cygwin terminal (i.e. ``mintty``), run the following commands. Note
 that if you are using the version of ``pip`` that was installed with the Cygwin
 Python package, the command name will be ``pip3``.  If you have since updated
@@ -554,12 +551,15 @@ manager. ``pip`` cannot provide them.
 The following versions are required:
 
 -  Python 3.9 or newer
--  Ghostscript 9.50 or newer
+-  Ghostscript 9.55 or newer
 -  Tesseract 4.1.1 or newer
 -  jbig2enc 0.29 or newer
 -  pngquant 2.5 or newer
 -  unpaper 6.1
 
+We recommend 64-bit versions of all software. (32-bit versions are not
+supported, although on Linux, they may still work.)
+
 jbig2enc, pngquant, and unpaper are optional. If missing certain
 features are disabled. OCRmyPDF will discover them as soon as they are
 available.
@@ -665,3 +665,18 @@ To manually install the ``bash`` completion, copy
 To manually install the ``fish`` completion, copy
 ``misc/completion/ocrmypdf.fish`` to
 ``~/.config/fish/completions/ocrmypdf.fish``.
+
+Note on 32-bit support
+======================
+
+Many Python libraries no longer 32-bit binary wheels for Linux. This
+includes many of the libraries that OCRmyPDF depends on, such as
+Pillow. The easiest way to express this to end users is to say we don't
+support 32-bit Linux.
+
+However, if your Linux distribution still supports 32-bit binaries, you
+can still install and use OCRmyPDF. A warning message will appear.
+In practice, OCRmyPDF may need more than 32-bit memory space to run when
+large documents are processed, so there are practical limitations to what
+users can accomplish with it. Still, for the common use case of an 32-bit
+ARM NAS or Raspberry Pi processing small documents, it should work.

docs/jbig2.rst

@@ -59,5 +59,9 @@ To turn on JBIG2 lossy mode, add the argument ``--jbig2-lossy``.
 also required. Also, a JBIG2 encoder must be installed as described in
 the previous section.
 
+You can adjust the threshold for JBIG2 compression with the
+``--jbig2-threshold``. The default is 0.85, meaning that if two symbols
+are 85% similar, they will be compressed together.
+
 *Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by
 default.*

docs/maintainers.rst

@@ -56,4 +56,12 @@ improve OCRmyPDF's compression.
 Command line completions
 ------------------------
 
-Please ensure that command line completions are installed.
+Please ensure that command line completions are installed, as described in the
+installation documentation.
+
+32-bit Linux support
+--------------------
+
+If you maintain a Linux distribution that supports 32-bit x86 or ARM, OCRmyPDF
+should continue to work as long as all of its dependencies continue to be
+available in 32-bit form. Please note we do not test on 32-bit platforms.

docs/optimizer.rst

@@ -45,8 +45,8 @@ Optimizations that always occurs
 ================================
 
 OCRmyPDF will automatically replace obsolete or inferior compression schemes
-such as RLE or LZW with superior schemes such as Deflate and converting
+such as RLE or LZW with superior schemes such as Deflate, and convert
-monochrome images to CCITT G4. Since this is harmless it always occurs and there
+monochrome images to CCITT G4. Since this is lossless, it always occurs and there
 is no way to disable it. Other non-image compressed objects are compressed as
 well.
 

docs/release_notes.rst

@@ -28,16 +28,31 @@ tagged yet.
 
 .. |OCRmyPDF PyPI| image:: https://img.shields.io/pypi/v/ocrmypdf.svg
 
+v15.0.2
+=======
+
+-  Added Python 3.12 to test matrix.
+-  Updated documentation for notes on Python 3.12, 32-bit support and some new
+   features in v15.
+
+v15.0.1
+=======
+
+-  Wheels Python tag changed to py39.
+-  Marked as a expected fail a test that fails on recent Ghostscript versions.
+-  Clarified documentation and release notes around the extent of 32-bit support.
+-  Updated installation documentation to changes in v15.
+
 v15.0.0
 =======
 
 -  Dropped support for Python 3.8.
 -  Dropped support many older dependencies - see ``pyproject.toml`` for details.
    Generally speaking, Ubuntu 22.04 is our baseline system.
--  Dropped support 32-bit Windows and Linux. You must use a 64-bit operating system,
+-  Dropped support for 32-bit Linux wheels. We strongly recommend a 64-bit operating
-   and 64-bit versions of Python, Tesseract and Ghostscript to use OCRmyPDF. Many of
+   system, and 64-bit versions of Python, Tesseract and Ghostscript to use OCRmyPDF.
-   our dependencies are dropping 32-bit support (e.g. Pillow), and we are following
+   Many of our dependencies are dropping 32-bit builds (e.g. Pillow), and we are
-   suit.
+   following suit. (Maintainers may still build 32-bit versions from source.)
 -  Changed to trusted release for PyPI publishing.
 -  pikepdf memory mapping is enabled again for improved performance, now an issue
    with pikepdf has been fixed.
@@ -52,7 +67,10 @@ v15.0.0
    each feature occupies. Typically, small areas of high resolution in PDFs are
    errors or quirks from the repeated use of assets and high resolution is not
    beneficial. :issue:`1010,1104,1004,1079,1010`
--  Ghostscript color conversion strategy is now configurable. :issue:`1143`
+-  Ghostscript color conversion strategy is now configurable using
+   ``--color-conversion-strategy``. :issue:`1143`
+-  JBIG2 threshold for optimization is now configurable using
+   ``--jbig2-threshold``. :issue:`1133`
 
 v14.4.0
 =======

pyproject.toml

@@ -77,11 +77,11 @@ namespaces = false
 [tool.setuptools_scm]
 
 [tool.distutils.bdist_wheel]
-python-tag = "py38"
+python-tag = "py39"
 
 [tool.black]
 line-length = 88
-target-version = ["py38", "py39", "py310", "py311"]
+target-version = ["py39", "py310", "py311"]
 skip-string-normalization = true
 include = '\.pyi?$'
 exclude = '''
@@ -157,7 +157,7 @@ select = [
   "I001", # isort
   "UP",   # pyupgrade
 ]
-target-version = "py38"
+target-version = "py39"
 
 [tool.ruff.isort]
 known-first-party = ["ocrmypdf"]

src/ocrmypdf/_validation.py

@@ -44,7 +44,7 @@ log = logging.getLogger(__name__)
 
 def check_platform() -> None:
     if sys.maxsize <= 2**32:  # pragma: no cover
-        log.error(
+        log.warning(
             "You are running OCRmyPDF in a 32-bit (x86) Python interpreter. "
             "This is not supported. 32-bit does not have enough address space "
             "to process large files. "

tests/test_metadata.py

@@ -394,9 +394,12 @@ def test_prevent_gs_invalid_xml(resources, outdir):
     assert contents.find(b'\x00', xmp_start, xmp_end) == -1
 
 
-@pytest.mark.skipif(
+@pytest.mark.xfail(
-    ghostscript.version() >= Version('10.2.0'),
+    ghostscript.version() >= Version('10.01.2'),
-    reason="Ghostscript 10.2.0+ exit with an error on invalid DocumentInfo",
+    reason=(
+        "Ghostscript now exits with an error on invalid DocumentInfo, defeating "
+        "this test.",
+    ),
 )
 def test_malformed_docinfo(caplog, resources, outdir):
     generate_pdfa_ps(outdir / 'pdfa.ps')