menekis/OCRmyPDF
Public Access
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'release/v17'

Browse Source
This commit is contained in:
James R. Barlow
2026-01-29 18:31:11 -08:00
parent e77f79ac6f c84fc56e45
commit ad30dd94f7
274 changed files with 23208 additions and 13132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.docker/Dockerfile
+3 -1
View File
@@ -55,7 +55,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen \
--extra test --extra webservice --extra watcher --no-dev \
--extra webservice --extra watcher --no-dev \
--no-install-package pyarrow
FROM base
@@ -67,6 +67,8 @@ RUN add-apt-repository -y ppa:alex-p/tesseract-ocr5
RUN apt-get update && apt-get install -y --no-install-recommends \
ghostscript \
fonts-droid-fallback \
fonts-noto-core \
fonts-noto-cjk \
jbig2dec \
pngquant \
tesseract-ocr \
.docker/Dockerfile.alpine
+2 -1
View File
@@ -39,7 +39,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
COPY . /app
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen \
--extra test --extra webservice --extra watcher --no-dev \
--extra webservice --extra watcher --no-dev \
--no-install-package pyarrow
FROM base
@@ -57,6 +57,7 @@ RUN apk add --no-cache \
tesseract-ocr-data-osd \
tesseract-ocr-data-por \
tesseract-ocr-data-spa \
font-noto \
ttf-droid \
unpaper \
&& rm -rf /var/cache/apk/*
.gitattributes
+1
View File
@@ -13,5 +13,6 @@
*.jpg binary
*.bin binary
*.afdesign binary
*.ttf binary
.git_archival.txt export-subst
.github/workflows/build.yml
+14 -9
View File
@@ -22,11 +22,11 @@ jobs:
strategy:
matrix:
os: [ubuntu-22.04, ubuntu-24.04]
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.11", "3.12", "3.13", "3.14"]
include:
- os: ubuntu-22.04
tesseract_ppa: "ppa"
python: "3.10"
python: "3.11"
env:
OS: ${{ matrix.os }}
@@ -57,6 +57,8 @@ jobs:
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
curl \
fonts-noto-core \
fonts-noto-cjk \
ghostscript \
jbig2dec \
img2pdf \
@@ -74,7 +76,7 @@ jobs:
- name: Install Python packages
run: |
uv sync --extra test --no-dev
uv sync --group test
- name: Report versions
run: |
@@ -102,7 +104,7 @@ jobs:
strategy:
matrix:
os: [macos-latest]
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.11", "3.12", "3.13", "3.14"]
env:
OS: ${{ matrix.os }}
@@ -123,7 +125,9 @@ jobs:
jbig2enc \
openjpeg \
pngquant \
tesseract
poppler \
tesseract \
verapdf
- name: Install uv
uses: astral-sh/setup-uv@v7
@@ -137,7 +141,7 @@ jobs:
- name: Install Python packages
run: |
uv sync --extra test --no-dev
uv sync --group test
- name: Report versions
run: |
@@ -164,7 +168,7 @@ jobs:
strategy:
matrix:
os: [windows-latest]
python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
python: ["3.11", "3.12", "3.13", "3.14"]
env:
OS: ${{ matrix.os }}
@@ -187,12 +191,13 @@ jobs:
- name: Install system packages
run: |
choco install --yes --no-progress --pre tesseract
choco install --yes --no-progress tesseract
choco install --yes --no-progress --ignore-checksums ghostscript --version 9.56.1
choco install --yes --no-progress poppler --version=25.11.0
- name: Install Python packages
run: |
uv sync --extra test --no-dev
uv sync --group test
- name: Test
run: |
.gitignore
+1
View File
@@ -48,3 +48,4 @@ src/ocrmypdf/_version.py
.idea/
.aider*
CLAUDE.md
.pre-commit-config.yaml
+5 -10
View File
@@ -10,17 +10,12 @@ repos:
- id: check-toml
- id: check-yaml
- id: debug-statements
- repo: https://github.com/charliermarsh/ruff-pre-commit
rev: "v0.0.261"
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: "v0.14.11"
hooks:
- id: ruff
files: "src/.*\\.pyi?$"
args: [--fix, --exit-non-zero-on-fix]
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
language_version: python
- id: ruff-check
args: [--fix]
- id: ruff-format
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.2.0
hooks:
.readthedocs.yaml
+1 -1
View File
@@ -15,7 +15,7 @@ sphinx:
build:
os: ubuntu-22.04
tools:
python: "3.10"
python: "3.11"
python:
install:
README.md
+12 -2
View File
@@ -126,8 +126,8 @@ Please report issues on our [GitHub issues](https://github.com/ocrmypdf/OCRmyPDF
## Feature demo
```bash
# Add an OCR layer and convert to PDF/A
ocrmypdf input.pdf output.pdf
# Add an OCR layer and require PDF/A
ocrmypdf --output-type pdfa input.pdf output.pdf
# Convert an image to single page PDF
ocrmypdf input.jpg output.pdf
@@ -151,6 +151,16 @@ For more features, see the [documentation](https://ocrmypdf.readthedocs.io/en/la
In addition to the required Python version, OCRmyPDF requires external program installations of Ghostscript and Tesseract OCR. OCRmyPDF is pure Python, and runs on pretty much everything: Linux, macOS, Windows and FreeBSD.
## Plugins
OCRmyPDF provides a plugin interface allowing its capabilities to be extended or replaced. Here are some plugins we are aware of:
- [OCRmyPDF-AppleOCR](https://github.com/mkyt/ocrmypdf-AppleOCR): replaces the standard Tesseract OCR engine with Apple Vision Framework. Requires macOS.
- [OCRmyPDF-EasyOCR](https://github.com/ocrmypdf/OCRmyPDF-EasyOCR): replaces the standard Tesseract OCR engine with EasyOCR, a newer OCR engine based on PyTorch. GPU strongly recommended.
- [OCRmyPDF-PaddleOCR](https://github.com/clefru/ocrmypdf-paddleocr): replaces the standard Tesseract OCR engine with PaddleOCR, a powerful GPU accelerated OCR engine.
[paperless-ngx](https://docs.paperless-ngx.com/) provides integration of OCRmyPDF into a searchable document management system.
## Press & Media
- [Going paperless with OCRmyPDF](https://medium.com/@ikirichenko/going-paperless-with-ocrmypdf-e2f36143f46a)
REUSE.toml
+2 -8
View File
@@ -167,15 +167,9 @@ SPDX-FileCopyrightText = [
SPDX-License-Identifier = "Zlib"
[[annotations]]
path = "src/ocrmypdf/data/pdf.ttf"
path = "src/ocrmypdf/data/Occulta.ttf"
precedence = "aggregate"
SPDX-FileCopyrightText = [
"(C) 2014 Ray Smith",
"(C) 2015 Ken Sharp",
"(C) 2016 James R. Barlow",
"(C) 2016 Jeff Breidenbach",
"(C) 2017 Zdenko Podobný",
]
SPDX-FileCopyrightText = ["(C) 2026 James R. Barlow"]
SPDX-License-Identifier = "Apache-2.0"
[[annotations]]
docs/advanced.md
+176 -23
View File
@@ -58,6 +58,38 @@ disk space.
OCRmyPDF provides many features to control the behavior of the OCR
engine, Tesseract.
### OCR processing mode
:::{versionadded} 17.0.0
The `--mode` (`-m`) argument consolidates OCR processing options.
:::
OCRmyPDF provides a unified `--mode` argument to control how pages with
existing text are handled:
| Mode | Behavior | Legacy equivalent |
|------|----------|-------------------|
| `default` | Error if text is found | (no flag) |
| `force` | Rasterize all content and run OCR | `--force-ocr` |
| `skip` | Skip pages with existing text | `--skip-text` |
| `redo` | Re-OCR pages, stripping old OCR layer | `--redo-ocr` |
```bash
# Skip pages that already have text
ocrmypdf --mode skip input.pdf output.pdf
# or equivalently:
ocrmypdf -m skip input.pdf output.pdf
# Force OCR on all pages (rasterizes everything)
ocrmypdf --mode force input.pdf output.pdf
# Re-do OCR, replacing old invisible text
ocrmypdf --mode redo input.pdf output.pdf
```
The legacy flags (`--force-ocr`, `--skip-text`, `--redo-ocr`) remain as
silent aliases for backward compatibility.
### When OCR is skipped
If a page in a PDF seems to have text, by default OCRmyPDF will exit
@@ -65,13 +97,13 @@ without modifying the PDF. This is to ensure that PDFs that were
previously OCRed or were "born digital" rather than scanned are not
processed.
If `--skip-text` is issued, then no image processing or OCR will be
If `--mode skip` (or `--skip-text`) is issued, then no image processing or OCR will be
performed on pages that already have text. The page will be copied to
the output. This may be useful for documents that contain both "born
digital" and scanned content, or to use OCRmyPDF to normalize and
convert to PDF/A regardless of their contents.
If `--redo-ocr` is issued, then a detailed text analysis is performed.
If `--mode redo` (or `--redo-ocr`) is issued, then a detailed text analysis is performed.
Text is categorized as either visible or invisible. Invisible text (OCR)
is stripped out. Then an image of each page is created with visible text
masked out. The page image is sent for OCR, and any additional text is
@@ -82,7 +114,7 @@ technically printable or visible in some way, perhaps by drawing it and
then painting over it. OCRmyPDF cannot distinguish this type of OCR
text from real text, so it will not be "redone".
If `--force-ocr` is issued, then all pages will be rasterized to
If `--mode force` (or `--force-ocr`) is issued, then all pages will be rasterized to
images, discarding any hidden OCR text, rasterizing any printable
text, and flattening form fields or interactive objects into their visual
representation. This is useful for redoing OCR, for fixing OCR text
@@ -257,44 +289,85 @@ Their use may interfere with `--rotate-pages` and other features.
It is currently not possible to use advanced Tesseract OCR features, such as creating
OCR information, when using Tesseract through OCRmyPDF.
## Changing the PDF renderer
## Choosing a PDF rasterizer
:::{versionadded} 17.0.0
:::
rasterizing
: Converting a PDF to an image for display.
: Converting a PDF page to an image for OCR processing.
OCRmyPDF supports two PDF rasterizers:
| Rasterizer | Package | Advantages | Disadvantages |
|------------|---------|------------|---------------|
| pypdfium2 | Python package | Faster, fewer version issues | Requires pypdfium2 package |
| Ghostscript | System binary | More widely packaged | Version consistency issues, restrictive AGPLv3 |
The `--rasterizer` argument controls which rasterizer is used:
```bash
# Automatic selection (default) - prefers pypdfium when available
ocrmypdf --rasterizer auto input.pdf output.pdf
# Force pypdfium2
ocrmypdf --rasterizer pypdfium input.pdf output.pdf
# Force Ghostscript
ocrmypdf --rasterizer ghostscript input.pdf output.pdf
```
pypdfium2 is a Python binding for pdfium, the PDF rendering library used
by Google Chrome and Chromium. It generally produces output identical to
Ghostscript but with better performance.
:::{note}
If pypdfium2 is not installed and `--rasterizer pypdfium` is requested,
OCRmyPDF will exit with an error. Install it with: `pip install pypdfium2`
:::
## Changing the PDF renderer
rendering
: Creating a new PDF from other data (such as an existing PDF).
OCRmyPDF has these PDF renderers: `sandwich` and `hocr`. The
:::{versionchanged} 17.0.0
The fpdf2 renderer is now the default, replacing the legacy hOCR renderer.
:::
OCRmyPDF uses PDF renderers to create the invisible text layer. The
renderer may be selected using `--pdf-renderer`. The default is
`auto` which lets OCRmyPDF select the renderer to use. Currently,
`auto` always selects `hocr`.
`auto` which selects `fpdf2`.
### The `hocr` renderer
### The `fpdf2` renderer (default)
:::{versionchanged} 16.0.0
:::{versionadded} 17.0.0
:::
The fpdf2 renderer creates text layers using the fpdf2 library. It provides:
- Full multilingual support including RTL languages (Arabic, Hebrew, Persian)
- Accurate text positioning aligned with OCR bounding boxes
- Improved "Occulta" glyphless font handling:
- Zero-width markers are properly handled
- Double-width CJK characters are properly sized
- Direct OcrElement tree input (no hOCR intermediate format required)
The fpdf2 renderer is the recommended choice for all installations.
:::{note}
The fpdf2 renderer may be slightly slower than the legacy hocrtransform
renderer for some workloads. This is an area of ongoing optimization.
:::
In both renderers, a text-only layer is rendered and sandwiched (overlaid)
on to either the original PDF page, or newly rasterized version of the
original PDF page (when `--force-ocr` is used). In this way, loss
original PDF page (when `--mode force` is used). In this way, loss
of PDF information is generally avoided. (You may need to disable PDF/A
conversion and optimization to eliminate all lossy transformations.)
The current approach used by the new hOCR renderer is a re-implementation
of Tesseract's PDF renderer, using the same Glyphless font and general
ideas, but fixing many technical issues that impeded it. The new hocr
provides better text placement accuracy, avoids issues with word
segmentation, and provides better positioning of skewed text.
Using the experimental API, it is also possible to edit the OCR output
from Tesseract, using any tool that is capable of editing hOCR files.
Older versions of this renderer did not support non-Latin languages, but
it is now universal.
### The `sandwich` renderer
The `sandwich` renderer uses Tesseract's text-only PDF feature,
@@ -310,6 +383,11 @@ When image preprocessing features like `--deskew` are used, the
original PDF will be rendered as a full page and the OCR layer will be
placed on top.
### Legacy renderer options
The `hocr` and `hocrdebug` renderer options are deprecated and
automatically redirect to `fpdf2`. They will be removed in a future version.
## Rendering and rasterizing options
:::{versionadded} 14.3.0
@@ -341,6 +419,81 @@ curves. In this case, you may want to use a different color conversion
strategy. The `--color-conversion-strategy` option allows you to select a
different strategy, such as `RGB`.
## PDF/A output modes
:::{versionchanged} 17.0.0
The default `--output-type` is now `auto` instead of `pdfa`.
:::
OCRmyPDF can produce PDF/A compliant output for long-term archival. The
`--output-type` argument controls PDF/A conversion:
| Output type | Behavior |
|-------------|----------|
| `auto` | Best-effort PDF/A without requiring Ghostscript (default) |
| `pdfa` | PDF/A-2b via Ghostscript |
| `pdfa-1` | PDF/A-1b via Ghostscript |
| `pdfa-2` | PDF/A-2b via Ghostscript (same as `pdfa`) |
| `pdfa-3` | PDF/A-3b via Ghostscript |
| `pdf` | Standard PDF, no PDF/A conversion |
| `none` | No output file (useful with `--sidecar`) |
### Speculative PDF/A conversion
:::{versionadded} 17.0.0
:::
When `--output-type auto` is used (the default), OCRmyPDF attempts a
fast "speculative" PDF/A conversion that avoids Ghostscript when possible:
1. OCRmyPDF adds an sRGB ICC profile and PDF/A XMP metadata using pikepdf
2. If verapdf is available, it validates the result
3. If validation passes, Ghostscript is skipped entirely
4. If validation fails or verapdf is unavailable, falls back to Ghostscript
This approach is faster and avoids some Ghostscript limitations (such as
image transcoding), but only works for PDFs that are already "mostly"
PDF/A compliant.
### PDF/A conversion flow
The following diagram illustrates the PDF/A conversion decision tree:
```{mermaid}
flowchart TD
A[Start] --> B{--output-type?}
B -->|pdf| C[Output standard PDF]
B -->|pdfa/pdfa-N| D[Use Ghostscript]
B -->|auto| E[Attempt speculative conversion]
E --> F["Add sRGB ICC + XMP metadata (pikepdf)"]
F --> G{verapdf available?}
G -->|No| H{Ghostscript available?}
G -->|Yes| I[Validate with verapdf]
I --> J{Validation passed?}
J -->|Yes| K[Output PDF/A - Ghostscript skipped]
J -->|No| H
H -->|Yes| D
H -->|No| L[Output standard PDF + WARNING]
D --> M[Ghostscript PDF/A conversion]
M --> N[Output PDF/A]
style K fill:#90EE90
style N fill:#90EE90
style L fill:#FFB6C1
```
:::{warning}
**Breaking change:** If neither Ghostscript nor verapdf is installed,
`--output-type auto` will produce a standard PDF instead of PDF/A.
This is a change from previous versions where Ghostscript was required
and PDF/A was always produced.
:::
## Return code policy
OCRmyPDF writes all messages to `stderr`. `stdout` is reserved for
docs/api.md
+62 -4
View File
@@ -13,8 +13,49 @@ subprocess call anyway, as this provides isolation of its activities.
## Example
OCRmyPDF provides one high-level function to run its main engine from an
application. The parameters are symmetric to the command line arguments
and largely have the same functions.
application.
```{versionchanged} 17.0
The {func}`ocrmypdf.ocr` function now accepts an {class}`~ocrmypdf.OcrOptions`
object as its first argument, providing a cleaner API with full type hints
and validation. The previous positional argument style remains supported.
```
### Modern API (recommended)
The recommended way to call {func}`ocrmypdf.ocr` is to construct an
{class}`~ocrmypdf.OcrOptions` object with all settings, then pass it
as the sole argument:
```python
import ocrmypdf
from ocrmypdf import OcrOptions
if __name__ == '__main__': # To ensure correct behavior on Windows and macOS
options = OcrOptions(
input_file='input.pdf',
output_file='output.pdf',
deskew=True,
languages=['eng'],
)
ocrmypdf.ocr(options)
```
{class}`~ocrmypdf.OcrOptions` is a Pydantic model that provides:
- Full type hints and IDE autocompletion
- Validation of option values at construction time
- Clear documentation of all available options
```{versionadded} 17.0
The {class}`~ocrmypdf.OcrOptions` class is now exported from the top-level
`ocrmypdf` module.
```
### Legacy API
For compatibility with OCRmyPDF < v17, the traditional calling style
with positional arguments is still fully supported:
```python
import ocrmypdf
@@ -23,7 +64,7 @@ if __name__ == '__main__': # To ensure correct behavior on Windows and macOS
ocrmypdf.ocr('input.pdf', 'output.pdf', deskew=True)
```
With some exceptions, all of the command line arguments are available
With this style, all of the command line arguments are available
and may be passed as equivalent keywords.
A few differences are that `verbose` and `quiet` are not available.
@@ -51,9 +92,12 @@ OCRmyPDF fails for any reason. For example:
```python
from multiprocessing import Process
import ocrmypdf
from ocrmypdf import OcrOptions
def ocrmypdf_process():
ocrmypdf.ocr('input.pdf', 'output.pdf')
options = OcrOptions(input_file='input.pdf', output_file='output.pdf')
ocrmypdf.ocr(options)
def call_ocrmypdf_from_my_app():
p = Process(target=ocrmypdf_process)
@@ -117,3 +161,17 @@ handler. OCRmyPDF will clean up its temporary files and worker processes
automatically when an exception occurs.
When OCRmyPDF succeeds conditionally, it returns an integer exit code.
### Plugin Development Changes
```{versionchanged} 16.13
Plugin hooks now receive {class}`~ocrmypdf.OcrOptions` objects instead of
`argparse.Namespace`.
```
- {class}`~ocrmypdf.OcrOptions` provides the same attribute access as `Namespace` (duck-typing compatible)
- Plugin developers should update type hints: `from ocrmypdf import OcrOptions`
- Built-in plugins no longer modify options in-place for better immutability
Most existing plugins will continue working without modification due to the
duck-typing compatibility between {class}`~ocrmypdf.OcrOptions` and `Namespace`.
docs/apiref.md
+7
View File
@@ -13,6 +13,13 @@ should be mainly of interest to plugin developers.
:members:
```
## ocrmypdf._options
```{eval-rst}
.. automodule:: ocrmypdf._options
:members: OcrOptions
```
## ocrmypdf.exceptions
```{eval-rst}
docs/batch.md
+4
View File
@@ -117,6 +117,10 @@ tend to give better performance. watcher.py works on all platforms.
Users may need to customize the script to meet their requirements.
:::{code} bash
# Using uv (recommended)
uv sync --extra watcher
# Or using pip
pip3 install ocrmypdf[watcher]
env OCR_INPUT_DIRECTORY=/mnt/input-pdfs \
docs/conf.py
+4 -2
View File
@@ -25,10 +25,11 @@
# sys.path.insert(0, os.path.abspath('.'))
# -- General configuration ------------------------------------------------
from __future__ import annotations
needs_sphinx = '8'
import datetime
import datetime as dt
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
@@ -41,6 +42,7 @@ extensions = [
'sphinx.ext.napoleon',
'sphinx.ext.imgconverter', # PDF docs needs this for SVG to PNG conversion
'sphinx_issues',
'sphinxcontrib.mermaid',
]
myst_enable_extensions = ['colon_fence', 'attrs_block', 'attrs_inline', 'substitution']
@@ -62,7 +64,7 @@ master_doc = 'index'
# General information about the project.
project = 'ocrmypdf'
year = str(datetime.date.today().year)
year = str(dt.date.today().year)
copyright = (
f'{year}, James R. Barlow. '
+ 'Licensed under Creative Commons Attribution-ShareAlike 4.0'
docs/cookbook.md
+73 -6
View File
@@ -215,15 +215,22 @@ ocrmypdf --deskew --clean --rotate-pages input.pdf output.pdf
Don\'t actually OCR my PDF
--------------------------
If you set `--tesseract-timeout 0` OCRmyPDF will apply its image
processing without performing OCR (by causing OCR to time out). This
works if all you want to is to apply image processing or PDF/A
If you set `--ocr-engine none` OCRmyPDF will apply its image processing without
performing OCR. This works if all you want to is to apply image processing or PDF/A
conversion.
```bash
ocrmypdf --tesseract-timeout 0 --remove-background input.pdf output.pdf
ocrmypdf --ocr-engine none --deskew --output-type pdfa input.pdf output.pdf
```
:::{versionchanged} v17.0.0
Prior to this version, `--tesseract-timeout 0` was recommended as an idiom
to turn off OCR. This is not longer recommended, as we move away from
Tesseract OCR as the primary OCR engine.
:::
:::{versionchanged} v14.1.0
Prior to this version, `--tesseract-timeout 0` would prevent other uses
@@ -238,7 +245,7 @@ This is getting ridiculous, but OCRmyPDF can complete strip all textual
information from a PDF and reconstruct it as a \"bag of images\" PDF.
```bash
ocrmypdf --tesseract-timeout 0 --force-ocr input.pdf output.pdf
ocrmypdf --ocr-engine none --force-ocr input.pdf output.pdf
```
Why would you want to do this? Perhaps you have a PDF where OCR fails to
@@ -250,9 +257,69 @@ This command also removes OCR generated by third party tools.
You can also optimize all images without performing any OCR:
```bash
ocrmypdf --tesseract-timeout 0 --optimize 3 --skip-text input.pdf output.pdf
ocrmypdf --ocr-engine none --optimize 3 --skip-text input.pdf output.pdf
```
## Using v17 features
### Select a rasterizer
:::{versionadded} 17.0.0
:::
OCRmyPDF can use pypdfium2 or Ghostscript to rasterize PDF pages. pypdfium2
is generally faster and is preferred when available.
```bash
# Automatic selection (default) - prefers pypdfium when available
ocrmypdf --rasterizer auto input.pdf output.pdf
# Explicitly use pypdfium2 (requires pip install pypdfium2)
ocrmypdf --rasterizer pypdfium input.pdf output.pdf
# Explicitly use Ghostscript
ocrmypdf --rasterizer ghostscript input.pdf output.pdf
```
### PDF/A without Ghostscript
:::{versionadded} 17.0.0
:::
With verapdf installed, OCRmyPDF can produce PDF/A without using Ghostscript
for conversion. This is faster and avoids some Ghostscript limitations.
```bash
# Uses speculative conversion with verapdf validation (default)
ocrmypdf --output-type auto input.pdf output.pdf
# Explicitly request Ghostscript-based PDF/A conversion
ocrmypdf --output-type pdfa input.pdf output.pdf
```
### Using --mode instead of legacy flags
:::{versionadded} 17.0.0
:::
The `--mode` (`-m`) flag consolidates OCR behavior options:
```bash
# Instead of --skip-text
ocrmypdf --mode skip input.pdf output.pdf
# Instead of --force-ocr
ocrmypdf --mode force input.pdf output.pdf
# Instead of --redo-ocr
ocrmypdf --mode redo input.pdf output.pdf
# Short form
ocrmypdf -m skip input.pdf output.pdf
```
The legacy flags continue to work as aliases.
### Process only certain pages
You can ask OCRmyPDF to only apply [image processing](#image-processing)
docs/installation.md
+96 -13
View File
@@ -487,7 +487,7 @@ You can then run OCRmyPDF in the Windows command prompt or Powershell, prefixing
First install the the following prerequisite Cygwin packages using `setup-x86_64.exe`:
```
python310 (or later)
python311 (or later)
python3?-devel
python3?-pip
python3?-lxml
@@ -599,21 +599,45 @@ OCRmyPDF currently requires these external programs and libraries to be
installed, and must be satisfied using the operating system package
manager. `pip` cannot provide them.
:::{versionchanged} 17.0.0
Ghostscript is now optional. pypdfium2 can be used for PDF rasterization,
and verapdf can validate speculative PDF/A conversion.
:::
The following versions are required:
- Python 3.10 or newer
- Ghostscript 9.54 or newer
- Python 3.11 or newer
- Tesseract 4.1.1 or newer
- jbig2enc 0.29 or newer
- pngquant 2.5 or newer
- unpaper 6.1
- One of: Ghostscript 9.54+ **or** pypdfium2 (Python package)
- One of: Ghostscript 9.54+ **or** verapdf (for PDF/A output)
- fpdf2 2.8 or newer (Python package)
- jbig2enc 0.29 or newer (optional)
- pngquant 2.5 or newer (optional)
- unpaper 6.1 (optional)
:::{note}
For the best user experience, install both Ghostscript and pypdfium2.
pypdfium2 is faster for rasterization, while Ghostscript provides
broader compatibility and is required for certain PDF/A conversions.
:::
We recommend 64-bit versions of all software. (32-bit versions are not
supported, although on Linux, they may still work.)
jbig2enc, pngquant, and unpaper are optional. If missing certain
features are disabled. OCRmyPDF will discover them as soon as they are
available.
**fpdf2** is a required dependency that provides the text layer
rendering engine. It replaces the legacy hOCR-based renderer with improved
multilingual support. Install with: `pip install fpdf2`
**pypdfium2**, if present, provides fast PDF page rasterization using
the pdfium library (the same library used by Google Chrome). It is
preferred over Ghostscript when available due to better performance.
Install with: `pip install pypdfium2`
**verapdf**, if present, enables fast speculative PDF/A conversion.
OCRmyPDF attempts to create PDF/A by adding metadata and ICC profiles
using pikepdf, then validates with verapdf. If validation passes,
Ghostscript is skipped entirely. See your distribution's package manager
or visit [verapdf.org](https://verapdf.org/).
**jbig2enc**, if present, will be used to optimize the encoding of
monochrome images. This can significantly reduce the file size of the
@@ -623,6 +647,12 @@ available for Ubuntu or Debian due to lingering concerns about patent
issues, but can easily be built from source. To add JBIG2 encoding, see
{ref}`jbig2`.
:::{warning}
Lossy JBIG2 encoding (`--jbig2-lossy`) has been removed in v17.0.0 due to
well-documented risks of character substitution errors. Only lossless
JBIG2 compression is now supported.
:::
**pngquant**, if present, is optionally used to optimize the encoding of
PNG-style images in PDFs (actually, any that are that losslessly
encoded) by lossily quantizing to a smaller color palette. It is only
@@ -638,7 +668,7 @@ unfortunately, the `pip install` command cannot satisfy all of them.
## Installing HEAD revision from sources
If you have `git` and Python 3.10 or newer installed, you can install
If you have `git` and Python 3.11 or newer installed, you can install
from source. When the `pip` installer runs, it will alert you if
dependencies are missing.
@@ -686,18 +716,71 @@ need to be installed. The script requires specific versions of the
dependencies. Older version than the ones mentioned in the release notes
are likely not to be compatible to OCRmyPDF.
## Optional Features
OCRmyPDF provides optional features and development tools. We recommend using `uv` as your package manager.
### Installing User Features
User features are available as optional dependencies. Install them with `uv` (recommended) or `pip`:
```bash
# Using uv (recommended)
uv sync --extra watcher # File watching service
uv sync --extra webservice # Streamlit web UI
uv sync --extra watcher --extra webservice # Multiple features
# Using pip (also works)
pip install ocrmypdf[watcher]
pip install ocrmypdf[webservice]
pip install ocrmypdf[watcher,webservice]
```
### Development Tools (uv only)
Development tools use dependency groups and require `uv`:
```bash
# Testing infrastructure
uv sync --group test
# Documentation building
uv sync --group docs
# Enhanced Streamlit development
uv sync --group streamlit-dev
# All development groups
uv sync
```
:::{note}
**User features** (`watcher`, `webservice`) work with both `uv` and `pip`.
**Developer tools** (`test`, `docs`, `streamlit-dev`) require `uv` and use dependency groups (PEP 735).
:::
**Why use uv?**
- Modern, fast Python package manager
- Required for development (testing, docs)
- Better dependency resolution
- Consistent across all platforms
Install uv: `pip install uv` or visit https://docs.astral.sh/uv/
### For development
To install all of the development and test requirements:
```bash
git clone -b main https://github.com/ocrmypdf/OCRmyPDF.git
python -m venv .venv
source .venv/bin/activate
cd OCRmyPDF
pip install -e .[test]
pip install uv # Install uv if not already installed
uv sync --group test
```
Note: Development requires `uv`. The old `pip install -e .[test]` method is no longer supported.
To add JBIG2 encoding, see {ref}`jbig2`.
## Shell completions
docs/introduction.md
+25 -5
View File
@@ -77,10 +77,17 @@ straightforward, and any PDF viewer can handle PDF/A files.
OCRmyPDF analyzes each page of a PDF to determine the required colorspace
and resolution (DPI) for capturing all the information on that page without
losing content. It uses
[Ghostscript](http://ghostscript.com/) to rasterize each page and subsequently
performs OCR on the rasterized image to generate an OCR "layer." This layer
is then integrated back into the original PDF.
losing content. It uses a PDF rasterizer (pypdfium2 or
[Ghostscript](http://ghostscript.com/)) to convert each page to an image and
subsequently performs OCR on the rasterized image to generate an OCR "layer."
This layer is then integrated back into the original PDF.
:::{versionchanged} 17.0.0
OCRmyPDF now supports pypdfium2 as an alternative rasterizer to Ghostscript.
pypdfium2 is a Python binding for pdfium, the PDF rendering library used by
Google Chrome. The `--rasterizer auto` setting (default) prefers pypdfium2
when available.
:::
While it is possible to use a program like Ghostscript or ImageMagick to
obtain an image and then run that image through Tesseract OCR, this process
@@ -156,7 +163,16 @@ These limitations are inherent to any software relying on Tesseract:
the text and its bounding box. As such, the generated PDF does not
contain any information about the document's structure.
Ghostscript also imposes some limitations:
### Ghostscript considerations
:::{versionchanged} 17.0.0
Ghostscript is no longer strictly required. OCRmyPDF can use pypdfium2
for rasterization and verapdf for PDF/A validation.
:::
While Ghostscript remains a capable and feature-rich tool with a long history,
recent releases have introduced some compatibility challenges that OCRmyPDF
v17 addresses through alternative codepaths. When Ghostscript is used:
- PDFs containing JPEG 2000-encoded content may be converted to JPEG
encoding, which may introduce compression artifacts, if Ghostscript
@@ -173,6 +189,10 @@ Ghostscript also imposes some limitations:
- Ghostscript's PDF/A conversion may remove or deactivate
hyperlinks and other active content.
When pypdfium2 and verapdf are available, many of these limitations can be
avoided by using the speculative PDF/A conversion path (enabled by default
with `--output-type auto`).
You can use `--output-type pdf` to disable PDF/A conversion and produce
a standard, non-archival PDF.
docs/jbig2.md
+15 -27
View File
@@ -43,33 +43,21 @@ be required depending on your system.
[sudo] apt install autotools-dev automake libtool libleptonica-dev pkg-config
:::
{#jbig2-lossy}
## JBIG2 Compression
## Lossy mode JBIG2
OCRmyPDF uses JBIG2 lossless compression for bitonal (black and white)
images. This provides excellent compression ratios compared to the older
CCITT G4 standard, while preserving the exact pixel content of the
original image.
OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially
dangerous feature. Users should [review the technical concerns with
JBIG2 in lossy mode](https://en.wikipedia.org/wiki/JBIG2#Disadvantages)
and decide if this feature is acceptable for their use case. In general,
this mode should not be used for archival purposes, should not be used
when the original document is not available or will be destroyed, and
should not be used when numbers present in the document are important,
because there is a risk of 6/8 and 8/6 substitution errors.
You can adjust the threshold for JBIG2 compression with
`--jbig2-threshold`. The default is 0.85.
JBIG2 lossy mode does achieve higher compression ratios than any other
monochrome (bitonal) compression technology; for large text documents
the savings are considerable. JBIG2 lossless still gives great
compression ratios and is a major improvement over the older CCITT G4
standard.
To turn on JBIG2 lossy mode, add the argument `--jbig2-lossy`.
`--optimize {1,2,3}` are necessary for the argument to take effect also
required. Also, a JBIG2 encoder must be installed as described in the
previous section.
You can adjust the threshold for JBIG2 compression with the
`--jbig2-threshold`. The default is 0.85, meaning that if two symbols
are 85% similar, they will be compressed together.
*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by
default.*
:::{note}
Previous versions of OCRmyPDF supported a lossy JBIG2 mode
(`--jbig2-lossy`). This feature has been removed due to the well-known
risk of character substitution errors (e.g., 6/8 confusion). See
[JBIG2 disadvantages](https://en.wikipedia.org/wiki/JBIG2#Disadvantages)
for more information on why lossy JBIG2 is problematic. The `--jbig2-lossy`
and `--jbig2-page-group-size` arguments are now ignored with a warning.
:::
docs/maintainers.md
+114 -6
View File
@@ -20,12 +20,48 @@ with much stiffer build requirements. If you want to use OCRmyPDF on
some novel platform or distribution, first make sure you can package
pikepdf.
### Non-Python dependencies
### Core dependencies
Note that we have non-Python dependencies. In particular, OCRmyPDF
requires Ghostscript and Tesseract OCR to be installed and needs to be
able to locate their binaries on the system PATH. On Windows, OCRmyPDF
will also check the registry for their locations.
:::{versionchanged} 17.0.0
Ghostscript is no longer strictly required. OCRmyPDF now supports alternative
codepaths for both PDF rasterization and PDF/A conversion.
:::
OCRmyPDF has the following runtime dependencies:
**For PDF rasterization** (converting PDF pages to images for OCR):
- `pypdfium2` (Python package) - OR -
- `ghostscript` (system binary)
- Recommendation: Install both for best compatibility
**For PDF/A conversion**:
- `verapdf` (system binary) with pikepdf's speculative conversion - OR -
- `ghostscript` (system binary)
- Recommendation: Install both for best compatibility
**For OCR**:
- `tesseract-ocr` (system binary) - Required for MVP
**For text rendering** (expressing OCR results in PDF):
- `fpdf2` (Python package) - Required for text layer rendering
- `uharfbuzz` (Python package) - Required for text layer rendering
- `font-noto` (system package) - Recommended for text layer rendering
**Other dependencies**:
- `unpaper` (system binary) - Optional, enables `--clean` and `--clean-final`
- `pngquant` (system binary) - Optional, enables `--optimize 2` and `--optimize 3`
- `jbig2enc` (system binary) - Optional, improves compression of monochrome images
While Ghostscript remains a capable and feature-rich tool with a long history,
recent releases have introduced some compatibility challenges that OCRmyPDF v17
addresses through alternative codepaths. For the best user experience, packagers
should install both Ghostscript and the alternative tools (pypdfium2, verapdf)
when available.
On Windows, OCRmyPDF will also check the registry for Tesseract and Ghostscript
locations.
Tesseract OCR relies on SIMD for performance and only has proper support
for this on ARM and x86\_64. Performance may be poor on other processor
@@ -48,7 +84,79 @@ override versioning for some reason.
OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some
distributions have shied away from packaging JBIG2 because it contains
patented algorithms, but all patents have expired since 2017. If
possible, consider packaging it too to improve OCRmyPDF\'s compression.
possible, consider packaging it too to improve OCRmyPDF's compression.
:::{note}
Lossy JBIG2 encoding has been removed in v17.0.0 due to well-documented
risks of character substitution errors. Previously we provided this feature
on a "caveat emptor" basis but in the interest of focusing and eliminating
risks, we decided to remove this option. Now, only lossless JBIG2 compression
is supported.
:::
### Dependency matrix for packagers
:::{versionadded} 17.0.0
:::
The following table summarizes the dependency options introduced in v17.0.0:
| Feature | Option 1 | Option 2 | Notes |
|---------|----------|----------|-------|
| PDF rasterization | pypdfium2 (Python) | ghostscript (binary) | pypdfium2 preferred when available |
| PDF/A conversion | verapdf + pikepdf | ghostscript | verapdf validates speculative conversion |
| Text rendering | fpdf2 (Python) | - | Required, replaces legacy hOCR renderer |
| OCR | tesseract-ocr | `--ocr-engine none` | Can be skipped entirely |
**Minimum viable installation:**
- tesseract-ocr + (pypdfium2 OR ghostscript) + fpdf2
**Recommended installation:**
- tesseract-ocr + pypdfium2 + ghostscript + verapdf + fpdf2 + unpaper + pngquant + jbig2enc
:::{warning}
If Ghostscript is not installed and verapdf is not available, PDF/A output
cannot be produced. The output will be a standard PDF instead. This is a
breaking change for rare configurations that previously relied on PDF/A
output without Ghostscript alternatives.
:::
**Sample debian/control dependency specification**
```
Depends:
fonts-noto,
fpdf2 (>= 2.8),
ghostscript (>= 9.55), # Not strictly required, but best user experience
icc-profiles-free,
img2pdf,
python3-coloredlogs,
python3-deprecation,
python3-pdfminer (>= 20181108+dfsg-3),
python3-pikepdf (>= 8.14.0),
python3-pil,
python3-pluggy,
python3-reportlab,
python3-rich,
python3-uharfbuzz, # Not currently in Debian
tesseract-ocr (>= 5.0.0),
zlib1g,
${misc:Depends},
${python3:Depends},
Recommends:
cyclopts, # Not currently in Debian
jbig2
paddleocr, # Not currently in Debian
pngquant,
pypdfium2, # Not currently in Debian
unpaper,
verapdf, # Not currently in Debian
Suggests:
ocrmypdf-doc,
python-watchdog,
```
### Command line completions
docs/optimizer.md
-7
View File
@@ -28,9 +28,6 @@ header-rows: 1
- Enables lossless optimizations, such as transcoding images to more
efficient formats. Also compress other uncompressed objects in the
PDF and enables the more efficient "object streams" within the PDF.
(If ``--jbig2-lossy`` is issued, then lossy JBIG2 optimization is used.
The decision to use lossy JBIG2 is separate from standard optimization
settings.)
* - ``--optimize 2``
- ``-O2``
- All of the above, and enables lossy optimizations and color quantization.
@@ -105,7 +102,3 @@ quality image may be suitable for storage after OCR.
It is not possible to optimize all image types. Uncommon image types may
be skipped by the optimizer.
OCRmyPDF provides `lossy mode JBIG2 <jbig2-lossy>`{.interpreted-text
role="ref"} as an advanced feature that additional requires the argument
`--jbig2-lossy`.
docs/plugins.md
+166
View File
@@ -164,6 +164,77 @@ chaining operations.
.. autofunction:: ocrmypdf.pluginspec.check_options
```
### Plugin option models
Plugins can define their own option models using Pydantic. This allows plugins to:
- Define type-safe option structures with validation
- Add CLI arguments that map to their option model fields
- Access options via nested namespaces (e.g., `options.tesseract.timeout`)
```{eval-rst}
.. autofunction:: ocrmypdf.pluginspec.register_options
```
Plugin options can be accessed in two ways:
1. **Flat access** (backward compatible): `options.tesseract_timeout`
2. **Nested access**: `options.tesseract.timeout`
Both access patterns are equivalent and return the same values.
:::{note}
**Plugin Interface Change**: Starting in OCRmyPDF v17.0.0, plugin hooks receive
`OcrOptions` objects instead of `argparse.Namespace` objects. Most plugins will
continue working due to duck-typing compatibility, but plugin developers should
update their type hints accordingly.
:::
### Migration guide for plugin developers
:::{versionadded} 17.0.0
:::
**Update imports:**
```python
from ocrmypdf._options import OcrOptions
```
**Update type hints:**
```python
# Before (v16 and earlier)
def check_options(options: argparse.Namespace) -> None:
...
# After (v17+)
def check_options(options: OcrOptions) -> None:
...
```
**Attribute access unchanged:**
```python
# These work exactly as before
options.languages
options.output_type
options.tesseract_timeout
```
**Remove in-place modifications:**
```python
# Before (v16 pattern - no longer recommended)
def check_options(options):
options.some_computed_value = compute_value(options)
# After (v17 pattern - compute at point of use)
def some_function(options):
computed = compute_value(options)
use_computed(computed)
```
### Execution and progress reporting
```{eval-rst}
@@ -248,3 +319,98 @@ chaining operations.
```{eval-rst}
.. autofunction:: ocrmypdf.pluginspec.is_optimization_enabled
```
### Working with OcrElement trees
:::{versionadded} 17.0.0
:::
OCRmyPDF v17 introduces the `OcrElement` dataclass for representing OCR
output in an engine-agnostic format. This enables plugins to work with
OCR results without parsing hOCR XML.
**Key classes:**
```python
from ocrmypdf import OcrElement, OcrClass, BoundingBox
# OcrElement - represents any OCR structural unit
page = OcrElement(
ocr_class=OcrClass.PAGE,
bbox=BoundingBox(0, 0, 612, 792),
children=[...]
)
# BoundingBox - axis-aligned bounding box (left, top, right, bottom)
bbox = BoundingBox(left=100, top=50, right=300, bottom=80)
# OcrClass - constants for element types
OcrClass.PAGE # "ocr_page"
OcrClass.LINE # "ocr_line"
OcrClass.WORD # "ocrx_word"
OcrClass.PARAGRAPH # "ocr_par"
```
**Navigating the tree:**
```python
# Get all words in a page
words = page.words # Returns list[OcrElement]
# Get all lines
lines = page.lines
# Get combined text
text = page.get_text_recursive()
# Iterate by class
for para in page.paragraphs:
print(para.get_text_recursive())
```
**OCR engine plugins:**
Plugins implementing custom OCR engines can now output `OcrElement` trees
directly via the `generate_ocr()` method, bypassing hOCR entirely:
```python
from pathlib import Path
from ocrmypdf.pluginspec import OcrEngine
from ocrmypdf import OcrElement, OcrClass, BoundingBox
class MyOcrEngine(OcrEngine):
def generate_ocr(
self,
input_file: Path,
options,
context,
) -> OcrElement:
# Perform OCR and return OcrElement tree directly
# No need to generate hOCR XML
return OcrElement(
ocr_class=OcrClass.PAGE,
bbox=BoundingBox(0, 0, width, height),
dpi=300,
children=[
OcrElement(
ocr_class=OcrClass.LINE,
bbox=BoundingBox(100, 50, 500, 80),
children=[
OcrElement(
ocr_class=OcrClass.WORD,
bbox=BoundingBox(100, 50, 200, 80),
text="Hello",
),
# ... more words
]
),
# ... more lines
]
)
def supports_generate_ocr(self) -> bool:
return True # Indicate this engine uses generate_ocr()
```
This approach is simpler than generating hOCR and allows modern OCR
engines to integrate more naturally with OCRmyPDF.
docs/release_notes.md
+96
View File
@@ -25,6 +25,102 @@ about a forthcoming release that has not been tagged yet. A release is only
official when it's tagged and posted to PyPI.
:::
## v17.0.0 (unreleased)
**Breaking changes**
- **Plugin interface migration**: Plugin hooks now receive `OcrOptions` objects instead of
`argparse.Namespace` objects. Most plugins will continue working due to duck-typing
compatibility, but plugin developers should update their type hints from `Namespace`
to `OcrOptions`.
- Built-in plugins no longer modify options in-place, improving immutability and
code clarity.
- **Lossy JBIG2 removed**: The `--jbig2-lossy` and `--jbig2-page-group-size` options have been
removed due to well-documented risks of character substitution errors. These options are now
deprecated and will emit warnings if used. Only lossless JBIG2 compression is supported.
- **PDF/A output behavior change**: If neither Ghostscript nor verapdf is installed,
`--output-type auto` (the new default) will produce a standard PDF instead of PDF/A. This is
a change from previous versions where Ghostscript was required and PDF/A was always produced.
This configuration is rare but users should be aware of the change.
**New features**
- **pypdfium2 rasterizer**: Added optional pypdfium2-based PDF rasterization plugin as an
alternative to Ghostscript for page rendering. Use `--rasterizer pypdfium` to enable
(requires `pip install pypdfium2`). The default `--rasterizer auto` prefers pypdfium when
available and falls back to Ghostscript.
- **Pluggable OCR engines**: New `--ocr-engine` option allows selecting OCR engines:
- `auto` (default): Uses Tesseract
- `tesseract`: Explicit Tesseract selection
- `none`: Skip OCR entirely for PDF processing-only workflows
This prepares the foundation for future third-party OCR engine plugins.
- **Smart PDF/A conversion**: New `--output-type auto` (now the default) produces best-effort
PDF/A output without requiring Ghostscript when the verapdf validator is available. Falls back
to traditional Ghostscript conversion when needed.
- **verapdf integration**: Added optional verapdf validation for fast PDF/A conversion. When
available, OCRmyPDF attempts speculative PDF/A conversion using pikepdf, validates with verapdf,
and skips Ghostscript if validation passes.
- **Optional Ghostscript**: As a consequence of the changes above, Ghostscript is no longer a required dependency. It is optional.
- **fpdf2 text renderer**: Replaced legacy hOCR text renderer with new fpdf2-based implementation,
providing better multilingual support and more accurate text positioning.
- **Improved Occulta glyphless font**: The new Occulta font provides better handling of
zero-width markers and double-width CJK characters for accurate text layer positioning.
- **Expanded multilingual font support**: Added FontProvider infrastructure with language-aware
font selection for Devanagari (Hindi, Sanskrit, Marathi, Nepali), CJK (Chinese, Japanese,
Korean), Arabic script, and many other scripts. System font discovery reduces package size.
- **Simplified mode selection**: New `--mode` (`-m`) argument consolidates processing options:
- `default`: Error if text is found (standard behavior)
- `force`: Rasterize all content and run OCR (replaces `--force-ocr`)
- `skip`: Skip pages with existing text (replaces `--skip-text`)
- `redo`: Re-OCR pages, stripping old text layer (replaces `--redo-ocr`)
Legacy flags remain as silent aliases for backward compatibility.
**API improvements**
- Centralized validation logic in the `OcrOptions` Pydantic model
- Removed scattered option mutation throughout the codebase
- Better type safety for plugin development
- Simplified plugin option handling
- New `OcrElement`, `OcrClass`, and `BoundingBox` exports for OCR engine plugin developers
- Extended `OcrEngine` ABC with `generate_ocr()` method for direct OCR tree output, eliding the need to translate a modern engine's output to hOCR or directly write to PDF.
**Bug fixes**
- Fixed double-compression of already-deflated JPEGs.
- Fixed tesseract_cache plugin to properly handle cache misses.
- Fixed handling of PDF page boxes (ArtBox, BleedBox) which were not being processed correctly.
- Added thread safety lock to pypdfium plugin for concurrent operations.
- Improved pdfminer.six compatibility with explicit word spacing.
**Documentation**
- Updated cookbook to replace deprecated `--tesseract-timeout 0` with `--ocr-engine none`.
- Added comprehensive plugin documentation for new OCR engine framework.
**Dependency changes**
- Requires: one of `pypdfium2` or `ghostscript` for PDF rasterization (PDF to image)
- Preferred: both
- Requires: one of `verapdf` or `ghostscript` for PDF/A generation
- Preferred: both
- Recommended: `pypdfium2` for PDF rasterization (new dependency)
- Recommended: `ghostscript` (used to be Required)
- Recommended: Noto fonts for improved OCR text positioning
- Optional: `verapdf` for fast PDF/A validation (new dependency)
- Requires: `fpdf2` for text layer rendering (new dependency)
- Recommended: replace `typer` with `cyclopts` in misc scripts (new dependency)
- See docs/maintainers.md for details.
**Migration guide for plugin developers**
- Update imports: `from ocrmypdf._options import OcrOptions`
- Update type hints: `def check_options(options: OcrOptions)` instead of `options: Namespace`
- Attribute access remains unchanged: `options.languages`, `options.output_type`, etc.
- Remove any in-place option modifications - compute values at point of use instead
- Most existing plugins will continue working without changes due to duck-typing
## v16.13.0
- Added detection and repair for Ghostscript 10.6 JPEG corruption. When GS 10.6
misc/_webservice.py
+42 -41
View File
@@ -96,8 +96,9 @@ with st.expander("Optimization after OCR"):
png_quality = st.slider(
"PNG quality", min_value=0, max_value=100, value=75, key="png_quality"
)
jbig2_lossy = st.checkbox("JBIG2 lossy (dangerous)", value=False, key="jbig2_lossy")
jbig2_threshold = st.number_input("JBIG2 threshold", value=0, key="jbig2_threshold")
jbig2_threshold = st.number_input(
"JBIG2 threshold", value=0.85, key="jbig2_threshold"
)
with st.expander("Advanced options"):
jobs = st.slider(
@@ -189,51 +190,51 @@ if uploaded:
args.append(f"--jpeg-quality={jpeg_quality}")
if optimize > '0' and png_quality:
args.append(f"--png-quality={png_quality}")
if jbig2_lossy:
args.append("--jbig2-lossy")
if jbig2_threshold:
args.append(f"--jbig2-threshold={jbig2_threshold}")
if jobs:
args.append(f"--jobs={jobs}")
input_file = NamedTemporaryFile(delete=True, suffix=f"_{uploaded.name}")
input_file.write(uploaded.getvalue())
input_file.flush()
input_file.seek(0)
args.append(str(input_file.name))
output_file = NamedTemporaryFile(delete=True, suffix=".pdf")
args.append(str(output_file.name))
with NamedTemporaryFile(delete=True, suffix=f"_{uploaded.name}") as input_file:
input_file.write(uploaded.getvalue())
input_file.flush()
input_file.seek(0)
args.append(str(input_file.name))
with NamedTemporaryFile(delete=True, suffix=".pdf") as output_file:
args.append(str(output_file.name))
st.session_state['running'] = (
'run_button' in st.session_state and st.session_state.run_button
)
if st.button(
"Run OCRmyPDF",
disabled=st.session_state.get("running", False),
key='run_button',
):
st.session_state['running'] = True
args = [sys.executable, '-u', '-m', "ocrmypdf"] + args
st.session_state['running'] = (
'run_button' in st.session_state and st.session_state.run_button
)
if st.button(
"Run OCRmyPDF",
disabled=st.session_state.get("running", False),
key='run_button',
):
st.session_state['running'] = True
args = [sys.executable, '-u', '-m', "ocrmypdf"] + args
proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
with st.container(border=True):
while proc.poll() is None:
line = proc.stderr.readline()
if line:
st.html("<code>" + line.decode().strip() + "</code>")
proc = subprocess.Popen(
args, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)
with st.container(border=True):
while proc.poll() is None:
line = proc.stderr.readline()
if line:
st.html("<code>" + line.decode().strip() + "</code>")
if proc.returncode != 0:
st.error(f"ocrmypdf failed with exit code {proc.returncode}")
st.session_state['running'] = False
st.stop()
if proc.returncode != 0:
st.error(f"ocrmypdf failed with exit code {proc.returncode}")
st.session_state['running'] = False
st.stop()
if Path(output_file.name).stat().st_size == 0:
st.error("No output PDF file was generated")
st.stop()
if Path(output_file.name).stat().st_size == 0:
st.error("No output PDF file was generated")
st.stop()
st.download_button(
label="Download output PDF",
data=output_file.read(),
file_name=uploaded.name,
mime="application/pdf",
)
st.session_state['running'] = False
st.download_button(
label="Download output PDF",
data=output_file.read(),
file_name=uploaded.name,
mime="application/pdf",
)
st.session_state['running'] = False
misc/batch.py
+1 -4
View File
@@ -39,10 +39,7 @@ script_dir = Path(__file__).parent
# set archive_dir to a path for backup original documents. Leave empty if not required.
archive_dir = "/pdfbak"
if len(sys.argv) > 1:
start_dir = Path(sys.argv[1])
else:
start_dir = Path(".")
start_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(".")
if len(sys.argv) > 2:
log_file = Path(sys.argv[2])
misc/bisect_pdf.py
+1
View File
@@ -3,6 +3,7 @@
# SPDX-License-Identifier: MIT
"""Helper script for bisecting PDFs to find a page with an issue."""
from __future__ import annotations
import sys
misc/completion/ocrmypdf.bash
+69 -8
View File
@@ -21,14 +21,13 @@ __ocrmypdf_arguments()
--subject (set metadata)
--keywords (set metadata)
--rotate-pages (rotate pages to correct orientation)
--remove-background (attempt to remove background from pages)
--deskew (fix small horizontal alignment skew)
--clean (clean document images before OCR)
--clean-final (clean document images and keep result)
--unpaper-args (a quoted string of arguments to pass to unpaper)
--oversample (oversample images to this DPI)
--remove-vectors (don\'t send vector objects to OCR)
--threshold (threshold images before OCR)
--mode (processing mode for pages with existing text)
--force-ocr (OCR documents that already have printable text)
--skip-text (skip OCR on any pages that already contain text)
--redo-ocr (redo OCR on any pages that seem to have OCR already)
@@ -42,9 +41,12 @@ __ocrmypdf_arguments()
--pages (apply OCR to only the specified pages)
--max-image-mpixels (image decompression bomb threshold)
--pdf-renderer (select PDF renderer options)
--ocr-engine (OCR engine to use)
--rasterizer (PDF page rasterizer)
--rotate-pages-threshold (page rotation confidence)
--pdfa-image-compression (set PDF/A image compression options)
--fast-web-view (if file size if above this amount in MB linearize PDF)
--continue-on-soft-render-error (continue after recoverable render errors)
--plugin (name of plugin to import)
--keep-temporary-files (keep temporary files (debug)
--tesseract-config (set custom tesseract config file)
@@ -52,6 +54,10 @@ __ocrmypdf_arguments()
--tesseract-oem (set tesseract --oem)
--tesseract-thresholding (set tesseract image thresholding)
--tesseract-timeout (maximum number of seconds to wait for OCR)
--tesseract-non-ocr-timeout (maximum seconds for non-OCR operations)
--tesseract-downsample-large-images (downsample large images before OCR)
--no-tesseract-downsample-large-images (do not downsample large images)
--tesseract-downsample-above (downsample images larger than this pixel size)
--user-words (specify location of user words file)
--user-patterns (specify location of user patterns file)
--no-progress-bar (disable the progress bar)
@@ -68,7 +74,8 @@ __ocrmypdf_arguments()
__ocrmypdf_output-type()
{
local choices="pdfa (output a PDF/A (default))
local choices="auto (best-effort PDF/A without Ghostscript (default))
pdfa (output a PDF/A-2b)
pdf (output a standard PDF)
pdfa-1 (output a PDF/A-1b)
pdfa-2 (output a PDF/A-2b)
@@ -114,10 +121,11 @@ __ocrmypdf_optimize()
__ocrmypdf_pdf-renderer()
{
local choices="auto (auto select PDF renderer)
hocr (use hOCR renderer)
hocrdebug (uses hOCR renderer in debug mode, showing recognized text)
sandwich (use sandwich renderer)"
local choices="auto (auto select PDF renderer, uses fpdf2)
fpdf2 (use fpdf2 renderer with full language support)
sandwich (use sandwich renderer)
hocr (use hOCR renderer - deprecated)
hocrdebug (uses hOCR renderer in debug mode - deprecated)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
@@ -210,6 +218,46 @@ UseDeviceIndependentColor (convert with device independent color)"
fi
}
__ocrmypdf_mode()
{
local choices="default (error if text is found)
force (rasterize all content and run OCR)
skip (skip pages with existing text)
redo (re-OCR pages, replacing old invisible text)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_ocr-engine()
{
local choices="auto (select best available engine)
tesseract (use Tesseract OCR)
none (skip OCR entirely)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_rasterizer()
{
local choices="auto (prefer pypdfium, fall back to Ghostscript)
ghostscript (use Ghostscript rasterizer)
pypdfium (use pypdfium rasterizer - faster)"
COMPREPLY=( $( compgen -W "$choices" -- "$cur") )
# Remove description if only one completion exists
if [[ ${#COMPREPLY[*]} -eq 1 ]]; then
COMPREPLY=( ${COMPREPLY[0]%% *} )
fi
}
__ocrmypdf_check_previous()
{
case $prev in
@@ -241,6 +289,18 @@ __ocrmypdf_check_previous()
__ocrmypdf_pdf-renderer
return 0
;;
-m|--mode)
__ocrmypdf_mode
return 0
;;
--ocr-engine)
__ocrmypdf_ocr-engine
return 0
;;
--rasterizer)
__ocrmypdf_rasterizer
return 0
;;
--pdfa-image-compression)
__ocrmypdf_pdfa-image-compression
return 0
@@ -260,7 +320,8 @@ __ocrmypdf_check_previous()
--title|--author|--subject|--keywords|--unpaper-args|--pages|--plugin|\
--jpeg-quality|--png-quality|--image-dpi|--oversample|--skip-big|--max-image-mpixels|\
--tesseract-timeout|--rotate-pages-threshold|--fast-web-view)
--tesseract-timeout|--tesseract-non-ocr-timeout|--tesseract-downsample-above|\
--rotate-pages-threshold|--fast-web-view)
# argument required but no completions available
return 0
;;
misc/completion/ocrmypdf.fish
+34 -4
View File
@@ -11,8 +11,16 @@ complete -c ocrmypdf -s r -l rotate-pages -d "rotate pages to correct orientatio
complete -c ocrmypdf -s d -l deskew -d "fix small horizontal alignment skew"
complete -c ocrmypdf -s c -l clean -d "clean document images before OCR"
complete -c ocrmypdf -s i -l clean-final -d "clean document images and keep result"
complete -c ocrmypdf -x -l unpaper-args -d "quoted string of arguments to pass to unpaper"
complete -c ocrmypdf -l remove-vectors -d "don't send vector objects to OCR"
function __fish_ocrmypdf_mode
echo -e "default\t"(_ "error if text is found")
echo -e "force\t"(_ "rasterize all content and run OCR")
echo -e "skip\t"(_ "skip pages with existing text")
echo -e "redo\t"(_ "re-OCR pages, replacing old invisible text")
end
complete -c ocrmypdf -x -s m -l mode -a '(__fish_ocrmypdf_mode)' -d "processing mode for pages with existing text"
complete -c ocrmypdf -s f -l force-ocr -d "OCR documents that already have printable text"
complete -c ocrmypdf -s s -l skip-text -d "skip OCR on any pages that already contain text"
complete -c ocrmypdf -l redo-ocr -d "redo OCR on any pages that seem to have OCR already"
@@ -32,7 +40,8 @@ complete -c ocrmypdf -x -s l -l language -a '(__fish_ocrmypdf_languages)' -d lan
complete -c ocrmypdf -x -l image-dpi -d "assume this DPI if input image DPI is unknown"
function __fish_ocrmypdf_output_type
echo -e "pdfa\t"(_ "output a PDF/A (default)")
echo -e "auto\t"(_ "best-effort PDF/A without requiring Ghostscript (default)")
echo -e "pdfa\t"(_ "output a PDF/A-2b")
echo -e "pdf\t"(_ "output a standard PDF")
echo -e "pdfa-1\t"(_ "output a PDF/A-1b")
echo -e "pdfa-2\t"(_ "output a PDF/A-2b")
@@ -42,13 +51,28 @@ end
complete -c ocrmypdf -x -l output-type -a '(__fish_ocrmypdf_output_type)' -d "select PDF output options"
function __fish_ocrmypdf_pdf_renderer
echo -e "auto\t"(_ "auto select PDF renderer")
echo -e "hocr\t"(_ "use hOCR renderer")
echo -e "hocrdebug\t"(_ "uses hOCR renderer in debug mode, showing recognized text")
echo -e "auto\t"(_ "auto select PDF renderer (default, uses fpdf2)")
echo -e "fpdf2\t"(_ "use fpdf2 renderer with full language support")
echo -e "sandwich\t"(_ "use sandwich renderer")
echo -e "hocr\t"(_ "use hOCR renderer (deprecated)")
echo -e "hocrdebug\t"(_ "uses hOCR renderer in debug mode (deprecated)")
end
complete -c ocrmypdf -x -l pdf-renderer -a '(__fish_ocrmypdf_pdf_renderer)' -d "select PDF renderer options"
function __fish_ocrmypdf_ocr_engine
echo -e "auto\t"(_ "select best available engine (default)")
echo -e "tesseract\t"(_ "use Tesseract OCR")
echo -e "none\t"(_ "skip OCR entirely")
end
complete -c ocrmypdf -x -l ocr-engine -a '(__fish_ocrmypdf_ocr_engine)' -d "OCR engine to use"
function __fish_ocrmypdf_rasterizer
echo -e "auto\t"(_ "prefer pypdfium, fall back to Ghostscript (default)")
echo -e "ghostscript\t"(_ "use Ghostscript rasterizer")
echo -e "pypdfium\t"(_ "use pypdfium rasterizer (faster)")
end
complete -c ocrmypdf -x -l rasterizer -a '(__fish_ocrmypdf_rasterizer)' -d "PDF page rasterizer"
function __fish_ocrmypdf_optimize
echo -e "0\t"(_ "do not optimize")
echo -e "1\t"(_ "do safe, lossless optimizations (default)")
@@ -124,11 +148,17 @@ end
complete -c ocrmypdf -x -l tesseract-thresholding -a '(__fish_ocrmypdf_tesseract_thresholding)' -d "set tesseract thresholding method (needs Tesseract 5.x)"
complete -c ocrmypdf -x -l tesseract-timeout -d "maximum number of seconds to wait for OCR"
complete -c ocrmypdf -x -l tesseract-non-ocr-timeout -d "maximum seconds to wait for non-OCR operations"
complete -c ocrmypdf -l tesseract-downsample-large-images -d "downsample large images before OCR"
complete -c ocrmypdf -l no-tesseract-downsample-large-images -d "do not downsample large images"
complete -c ocrmypdf -x -l tesseract-downsample-above -d "downsample images larger than this pixel size"
complete -c ocrmypdf -x -l rotate-pages-threshold -d "page rotation confidence"
complete -c ocrmypdf -r -l user-words -d "specify location of user words file"
complete -c ocrmypdf -r -l user-patterns -d "specify location of user patterns file"
complete -c ocrmypdf -x -l fast-web-view -d "if file size if above this amount in MB, linearize PDF"
complete -c ocrmypdf -l continue-on-soft-render-error -d "continue processing after recoverable render errors"
complete -c ocrmypdf -r -l plugin -d "name of plugin to import"
function __fish_ocrmypdf_color_conversion_strategy
echo -e "LeaveColorUnchanged\t"(_ "do not convert color spaces (default)")
misc/ocrmypdf_compare.py
+1 -1
View File
@@ -108,7 +108,7 @@ def main():
doc1 = pymupdf.open(os.path.join(d, "output1.pdf"))
doc2 = pymupdf.open(os.path.join(d, "output2.pdf"))
for i, page1_2 in enumerate(zip(doc1, doc2)):
for i, page1_2 in enumerate(zip(doc1, doc2, strict=False)):
st.write(f"Page {i+1}")
page1, page2 = page1_2
col1, col2 = st.columns(2)
misc/pdf_compare.py
+1 -1
View File
@@ -62,7 +62,7 @@ def main():
with st.expander("Text"):
doc1 = pymupdf.open(os.path.join(d, "1.pdf"))
doc2 = pymupdf.open(os.path.join(d, "2.pdf"))
for i, page1_2 in enumerate(zip(doc1, doc2)):
for i, page1_2 in enumerate(zip(doc1, doc2, strict=False)):
st.write(f"Page {i+1}")
page1, page2 = page1_2
col1, col2 = st.columns(2)
misc/pdf_text_diff.py
+29 -18
View File
@@ -5,34 +5,45 @@
from __future__ import annotations
from pathlib import Path
from subprocess import run
from tempfile import NamedTemporaryFile
from typing import Annotated
import typer
import cyclopts
app = cyclopts.App()
@app.default
def main(
pdf1: Annotated[typer.FileBinaryRead, typer.Argument()],
pdf2: Annotated[typer.FileBinaryRead, typer.Argument()],
engine: Annotated[str, typer.Option()] = 'pdftotext',
pdf1: Annotated[Path, cyclopts.Parameter()],
pdf2: Annotated[Path, cyclopts.Parameter()],
*,
engine: Annotated[str, cyclopts.Parameter()] = 'pdftotext',
):
"""Compare text in PDFs."""
with open(pdf1, 'rb') as f1, open(pdf2, 'rb') as f2:
text1 = run(
['pdftotext', '-layout', '-', '-'],
stdin=f1,
capture_output=True,
check=True,
)
text2 = run(
['pdftotext', '-layout', '-', '-'],
stdin=f2,
capture_output=True,
check=True,
)
text1 = run(
['pdftotext', '-layout', '-', '-'], stdin=pdf1, capture_output=True, check=True
)
text2 = run(
['pdftotext', '-layout', '-', '-'], stdin=pdf2, capture_output=True, check=True
)
with NamedTemporaryFile() as f1, NamedTemporaryFile() as f2:
f1.write(text1.stdout)
f1.flush()
f2.write(text2.stdout)
f2.flush()
with NamedTemporaryFile() as t1, NamedTemporaryFile() as t2:
t1.write(text1.stdout)
t1.flush()
t2.write(text2.stdout)
t2.flush()
diff = run(
['diff', '--color=always', '--side-by-side', f1.name, f2.name],
['diff', '--color=always', '--side-by-side', t1.name, t2.name],
capture_output=True,
)
run(['less', '-R'], input=diff.stdout, check=True)
@@ -43,4 +54,4 @@ def main(
if __name__ == '__main__':
typer.run(main)
app()
misc/watcher.py
+45 -62
View File
@@ -7,18 +7,18 @@
from __future__ import annotations
import datetime as dt
import json
import logging
import shutil
import sys
import time
from datetime import datetime
from enum import Enum
from pathlib import Path
from typing import Annotated, Any
import cyclopts
import pikepdf
import typer
from dotenv import load_dotenv
from watchdog.events import PatternMatchingEventHandler
from watchdog.observers import Observer
@@ -30,7 +30,7 @@ load_dotenv()
# pylint: disable=logging-format-interpolation
app = typer.Typer(name="ocrmypdf-watcher")
app = cyclopts.App(name="ocrmypdf-watcher")
log = logging.getLogger('ocrmypdf-watcher')
@@ -48,7 +48,7 @@ class LoggingLevelEnum(str, Enum):
def get_output_path(root: Path, basename: str, output_dir_year_month: bool) -> Path:
assert '/' not in basename, "basename must not contain '/'"
if output_dir_year_month:
today = datetime.today()
today = dt.datetime.today()
output_directory_year_month = root / str(today.year) / f'{today.month:02d}'
if not output_directory_year_month.exists():
output_directory_year_month.mkdir(parents=True, exist_ok=True)
@@ -114,9 +114,11 @@ def execute_ocrmypdf(
f'kwargs: {ocrmypdf_kwargs}'
)
exit_code = ocrmypdf.ocr(
input_file=file_path,
output_file=output_path,
**ocrmypdf_kwargs,
ocrmypdf.OcrOptions(
input_file=file_path,
output_file=output_path,
**ocrmypdf_kwargs,
)
)
if exit_code == 0:
if on_success_delete:
@@ -138,7 +140,7 @@ class HandleObserverEvent(PatternMatchingEventHandler):
ignore_patterns=None,
ignore_directories=False,
case_sensitive=False,
settings={},
settings=None,
):
super().__init__(
patterns=patterns,
@@ -146,117 +148,101 @@ class HandleObserverEvent(PatternMatchingEventHandler):
ignore_directories=ignore_directories,
case_sensitive=case_sensitive,
)
self._settings = settings
self._settings = settings if settings else {}
def on_any_event(self, event):
if event.event_type in ['created']:
execute_ocrmypdf(file_path=Path(event.src_path), **self._settings)
@app.command()
@app.default
def main(
input_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_INPUT_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
readable=True,
resolve_path=True,
cyclopts.Parameter(
env_var='OCR_INPUT_DIRECTORY',
),
] = '/input',
] = Path('/input'),
output_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_OUTPUT_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
writable=True,
resolve_path=True,
cyclopts.Parameter(
env_var='OCR_OUTPUT_DIRECTORY',
),
] = '/output',
] = Path('/output'),
archive_dir: Annotated[
Path,
typer.Argument(
envvar='OCR_ARCHIVE_DIRECTORY',
exists=True,
file_okay=False,
dir_okay=True,
writable=True,
resolve_path=True,
cyclopts.Parameter(
env_var='OCR_ARCHIVE_DIRECTORY',
),
] = '/processed',
] = Path('/processed'),
*,
output_dir_year_month: Annotated[
bool,
typer.Option(
envvar='OCR_OUTPUT_DIRECTORY_YEAR_MONTH',
cyclopts.Parameter(
env_var='OCR_OUTPUT_DIRECTORY_YEAR_MONTH',
help='Create a subdirectory in the output directory for each year/month',
),
] = False,
on_success_delete: Annotated[
bool,
typer.Option(
envvar='OCR_ON_SUCCESS_DELETE',
cyclopts.Parameter(
env_var='OCR_ON_SUCCESS_DELETE',
help='Delete the input file after successful OCR',
),
] = False,
on_success_archive: Annotated[
bool,
typer.Option(
envvar='OCR_ON_SUCCESS_ARCHIVE',
cyclopts.Parameter(
env_var='OCR_ON_SUCCESS_ARCHIVE',
help='Archive the input file after successful OCR',
),
] = False,
deskew: Annotated[
bool,
typer.Option(
envvar='OCR_DESKEW',
cyclopts.Parameter(
env_var='OCR_DESKEW',
help='Deskew the input file before OCR',
),
] = False,
ocr_json_settings: Annotated[
str,
typer.Option(
envvar='OCR_JSON_SETTINGS',
str | None,
cyclopts.Parameter(
env_var='OCR_JSON_SETTINGS',
help='JSON settings to pass to OCRmyPDF (JSON string or file path)',
),
] = None,
poll_new_file_seconds: Annotated[
int,
typer.Option(
envvar='OCR_POLL_NEW_FILE_SECONDS',
cyclopts.Parameter(
env_var='OCR_POLL_NEW_FILE_SECONDS',
help='Seconds to wait before polling a new file',
min=0,
),
] = 1,
use_polling: Annotated[
bool,
typer.Option(
envvar='OCR_USE_POLLING',
cyclopts.Parameter(
env_var='OCR_USE_POLLING',
help='Use polling instead of filesystem events',
),
] = False,
retries_loading_file: Annotated[
int,
typer.Option(
envvar='OCR_RETRIES_LOADING_FILE',
cyclopts.Parameter(
env_var='OCR_RETRIES_LOADING_FILE',
help='Number of times to retry loading a file before giving up',
min=0,
),
] = 5,
loglevel: Annotated[
LoggingLevelEnum,
typer.Option(
envvar='OCR_LOGLEVEL',
cyclopts.Parameter(
env_var='OCR_LOGLEVEL',
help='Logging level',
),
] = LoggingLevelEnum.INFO,
patterns: Annotated[
str,
typer.Option(
envvar='OCR_PATTERNS',
cyclopts.Parameter(
env_var='OCR_PATTERNS',
help='File patterns to watch',
),
] = '*.pdf,*.PDF',
@@ -316,13 +302,10 @@ def main(
'output_dir_year_month': output_dir_year_month,
},
)
if use_polling:
observer = PollingObserver()
else:
observer = Observer()
observer = PollingObserver() if use_polling else Observer()
observer.schedule(handler, input_dir, recursive=True)
observer.start()
typer.echo(f"Watching {input_dir} for new PDFs. Press Ctrl+C to exit.")
print(f"Watching {input_dir} for new PDFs. Press Ctrl+C to exit.")
try:
while True:
time.sleep(30)
misc/webservice.py
+3 -1
View File
@@ -4,6 +4,8 @@
"""Run the OCRmyPDF web service."""
from __future__ import annotations
import os
import sys
@@ -13,7 +15,7 @@ except ImportError:
raise ImportError(
'You need to install streamlit in the Python environment '
'to run the web service.\n'
)
) from None
if __name__ == '__main__':
os.execvp(
pyproject.toml
+56 -29
View File
@@ -10,9 +10,10 @@ dynamic = ["version"]
description = "OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched"
readme = "README.md"
license = "MPL-2.0"
requires-python = ">=3.10"
requires-python = ">=3.11"
dependencies = [
"deprecation>=2.1.0",
"fpdf2>=2.8.0",
"img2pdf>=0.5",
"packaging>=20",
"pdfminer.six>=20220319",
@@ -20,7 +21,10 @@ dependencies = [
"pikepdf>=10",
"Pillow>=10.0.1",
"pluggy>=1",
"pydantic>=2.12.5",
"pypdfium2>=5.0.0",
"rich>=13",
"uharfbuzz>=0.53.2",
]
authors = [{ name = "James R. Barlow", email = "james@purplerock.ca" }]
classifiers = [
@@ -48,20 +52,8 @@ Tracker = "https://github.com/ocrmypdf/OCRmyPDF/issues"
Changelog = "https://github.com/ocrmypdf/OCRmyPDF/docs/release_notes.md"
[project.optional-dependencies]
docs = ["myst-parser>=4.0.1", "sphinx", "sphinx-issues", "sphinx-rtd-theme"]
extended_test = ["PyMuPDF>=1.19.1"]
test = [
"coverage[toml]>=6.2",
"hypothesis>=6.36.0",
"pytest>=6.2.5",
"pytest-cov>=3.0.0",
"pytest-xdist>=2.5.0",
"python-xmp-toolkit==2.0.1", # also requires apt-get install libexempi3
"reportlab>=3.6.8",
"types-Pillow",
"types-humanfriendly",
]
watcher = ["watchdog>=1.0.2", "typer-slim[standard]", "python-dotenv"]
# User-installable features - use `uv sync --extra <name>` or `pip install ocrmypdf[name]`
watcher = ["watchdog>=1.0.2", "cyclopts>=3", "python-dotenv"]
webservice = ["streamlit>=1.41.0"]
[project.scripts]
@@ -74,7 +66,7 @@ source = "vcs"
version-file = "src/ocrmypdf/_version.py"
[tool.distutils.bdist_wheel]
python-tag = "py310"
python-tag = "py311"
[tool.coverage.run]
branch = true
@@ -125,27 +117,43 @@ module = [
ignore_missing_imports = true
[tool.ruff]
target-version = "py310"
target-version = "py311"
exclude = ["src/ocrmypdf/_version.py"] # Autogenerated
[tool.ruff.lint]
"select" = [
"D", # pydocstyle
"E", # pycodestyle
"W", # pycodestyle
"F", # pyflakes
"I001", # isort
"UP", # pyupgrade
"D", # pydocstyle
"E", # pycodestyle
"W", # pycodestyle
"F", # pyflakes
"I", # isort
"UP", # pyupgrade
"SIM", # simplify
"B", # flake8-bugbear
"ICN", # flake8-import-conventions
]
ignore = [
"B028", # warning with no explicit stacklevel
# rule is key in dict instead of key in dict.keys(); but pikepdf semantics differ
"SIM118",
]
[tool.ruff.lint.isort]
known-first-party = ["ocrmypdf"]
required-imports = ["from __future__ import annotations"]
[tool.ruff.lint.flake8-import-conventions]
# Prohibit explicit imports from the 'datetime' module
banned-from = ["datetime"]
# Optionally, suggest an alias for 'import datetime' (e.g., as dt)
extend-aliases = { "datetime" = "dt" }
[tool.ruff.lint.pydocstyle]
convention = "google"
[tool.ruff.lint.per-file-ignores]
"docs/conf.py" = ["D100", "D101", "D105"]
"tests/*.py" = ["D100", "D101", "D102", "D103", "D105"]
"tests/*.py" = ["D100", "D101", "D102", "D103", "D105", "E501"]
"misc/*.py" = ["D103", "D101", "D102"]
"src/ocrmypdf/builtin_plugins/*.py" = ["D103", "D102", "D105"]
@@ -153,10 +161,29 @@ convention = "google"
quote-style = "preserve"
[dependency-groups]
dev = [
"mypy>=1.13.0",
# Developer-only tools - use `uv sync --group <name>`
dev = ["mypy>=1.13.0", "ipykernel>=6.29.5", "reportlab>=4.4.4"]
test = [
# Core testing framework
"coverage[toml]>=6.2",
"hypothesis>=6.36.0",
"pytest>=6.2.5",
"pytest-cov>=3.0.0",
"pytest-xdist>=2.5.0",
# Test dependencies
"python-xmp-toolkit==2.0.1", # also requires apt-get install libexempi3
"reportlab>=3.6.8",
# Type stubs for testing
"types-Pillow",
"types-humanfriendly",
# Extended test capabilities (merged from extended_test)
"pymupdf>=1.24.14",
"streamlit-pdf-viewer>=0.0.19",
"streamlit>=1.40.2",
"ipykernel>=6.29.5",
]
docs = [
"myst-parser>=4.0.1",
"sphinx",
"sphinx-issues",
"sphinx-rtd-theme",
"sphinxcontrib-mermaid",
]
streamlit-dev = ["streamlit>=1.40.2", "streamlit-pdf-viewer>=0.0.19"]
scripts/generate_glyphless_font.py
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env python3
# SPDX-FileCopyrightText: 2024 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Generate the Occulta glyphless font for OCRmyPDF.
Occulta (Latin for "hidden") is a glyphless font designed for invisible text layers
in searchable PDFs. It has proper Unicode cmap coverage using format 13 (many-to-one)
for efficient mapping of all BMP codepoints to a small set of width-specific glyphs.
Features:
- Full BMP coverage (U+0000 to U+FFFF)
- Width-aware glyphs for proper text selection:
- Zero-width for combining marks and invisible characters
- Regular width (500 units) for Latin, Greek, Cyrillic, Arabic, Hebrew, etc.
- Double width (1000 units) for CJK and fullwidth characters
- Uses cmap format 13 (many-to-one) for ~12KB size vs ~780KB with format 12
- Compatible with fpdf2 and other modern PDF libraries
Usage:
python scripts/generate_glyphless_font.py
Output:
src/ocrmypdf/data/Occulta.ttf
"""
from __future__ import annotations
import unicodedata
from pathlib import Path
from fontTools.fontBuilder import FontBuilder
from fontTools.ttLib import TTFont
from fontTools.ttLib.tables._c_m_a_p import CmapSubtable
from fontTools.ttLib.tables._g_l_y_f import Glyph
# Output path relative to this script
OUTPUT_PATH = Path(__file__).parent.parent / "src" / "ocrmypdf" / "data" / "Occulta.ttf"
# Font metrics (units per em = 1000)
UNITS_PER_EM = 1000
ASCENT = 800
DESCENT = -200
# Glyph definitions: (name, advance_width, left_side_bearing)
GLYPHS = [
(".notdef", 500, 0), # Required, used for unmapped characters
("space", 500, 0), # U+0020 SPACE
("nbspace", 500, 0), # U+00A0 NO-BREAK SPACE
("blank0", 0, 0), # Zero-width (combining marks, ZWNJ, ZWJ, BOM)
("blank1", 500, 0), # Regular width (most scripts)
("blank2", 1000, 0), # Double width (CJK, fullwidth)
]
# Explicit zero-width character codepoints
ZERO_WIDTH_CHARS = frozenset(
[
0x200B, # ZERO WIDTH SPACE
0x200C, # ZERO WIDTH NON-JOINER
0x200D, # ZERO WIDTH JOINER
0xFEFF, # ZERO WIDTH NO-BREAK SPACE (BOM)
0x200E, # LEFT-TO-RIGHT MARK
0x200F, # RIGHT-TO-LEFT MARK
0x202A, # LEFT-TO-RIGHT EMBEDDING
0x202B, # RIGHT-TO-LEFT EMBEDDING
0x202C, # POP DIRECTIONAL FORMATTING
0x202D, # LEFT-TO-RIGHT OVERRIDE
0x202E, # RIGHT-TO-LEFT OVERRIDE
0x2060, # WORD JOINER
0x2061, # FUNCTION APPLICATION
0x2062, # INVISIBLE TIMES
0x2063, # INVISIBLE SEPARATOR
0x2064, # INVISIBLE PLUS
]
)
def classify_codepoint(codepoint: int) -> str:
"""Classify a Unicode codepoint into one of our glyph categories.
Args:
codepoint: Unicode codepoint (0x0000 to 0xFFFF)
Returns:
Glyph name to map this codepoint to
"""
# Special cases first
if codepoint == 0x0020:
return "space"
if codepoint == 0x00A0:
return "nbspace"
if codepoint in ZERO_WIDTH_CHARS:
return "blank0"
# Use Unicode properties for the rest
char = chr(codepoint)
try:
category = unicodedata.category(char)
east_asian_width = unicodedata.east_asian_width(char)
# Combining marks are zero-width
if category.startswith("M"):
return "blank0"
# Wide and Fullwidth characters are double-width
if east_asian_width in ("W", "F"):
return "blank2"
# Everything else is regular width
return "blank1"
except (ValueError, TypeError):
# Fallback for any edge cases
return "blank1"
def build_cmap() -> dict[int, str]:
"""Build the Unicode to glyph name mapping for the entire BMP.
Returns:
Dictionary mapping codepoints to glyph names
"""
return {cp: classify_codepoint(cp) for cp in range(0x10000)}
def create_font() -> TTFont:
"""Create the Occulta glyphless font.
Returns:
TTFont object ready to be saved
"""
glyph_names = [g[0] for g in GLYPHS]
# Start building the font
fb = FontBuilder(UNITS_PER_EM, isTTF=True)
fb.setupGlyphOrder(glyph_names)
# Create empty (invisible) glyphs
glyphs = {}
for name, _, _ in GLYPHS:
glyph = Glyph()
glyph.numberOfContours = 0
glyphs[name] = glyph
fb.setupGlyf(glyphs)
# Set up horizontal metrics
metrics = {name: (width, lsb) for name, width, lsb in GLYPHS}
fb.setupHorizontalMetrics(metrics)
# Minimal cmap to satisfy FontBuilder (we'll replace it later)
fb.setupCharacterMap({0x0020: "space", 0x00A0: "nbspace"})
# Set up other required tables
fb.setupHorizontalHeader(ascent=ASCENT, descent=DESCENT)
fb.setupOS2(
sTypoAscender=ASCENT,
sTypoDescender=DESCENT,
sTypoLineGap=0,
usWinAscent=UNITS_PER_EM,
usWinDescent=abs(DESCENT),
sxHeight=500,
sCapHeight=700,
)
import time
# Use current time for font timestamps
now = int(time.time())
fb.setupHead(unitsPerEm=UNITS_PER_EM, created=now, modified=now)
fb.setupPost()
fb.setupNameTable(
{
"familyName": "Occulta",
"styleName": "Regular",
"uniqueFontIdentifier": "OCRmyPDF;Occulta-Regular;2026",
"fullName": "Occulta Regular",
"version": "Version 2.0",
"psName": "Occulta-Regular",
}
)
# Build the font
font = fb.font
# Now replace the cmap with format 13 for efficient many-to-one mapping
char_to_glyph = build_cmap()
cmap13 = CmapSubtable.newSubtable(13)
cmap13.platformID = 3 # Windows
cmap13.platEncID = 10 # Unicode full repertoire
cmap13.language = 0
cmap13.cmap = char_to_glyph
font["cmap"].tables = [cmap13]
return font
def main() -> None:
"""Generate the Occulta font and save it."""
print("Generating Occulta glyphless font...")
font = create_font()
# Create output directory if needed
OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)
# Save the font
font.save(str(OUTPUT_PATH))
font.close()
# Report statistics
size = OUTPUT_PATH.stat().st_size
print(f"Saved to: {OUTPUT_PATH}")
print(f"Size: {size:,} bytes")
# Verify cmap
font = TTFont(str(OUTPUT_PATH))
for table in font["cmap"].tables:
print(
f"cmap: Platform {table.platformID}, "
f"Encoding {table.platEncID}, "
f"Format {table.format}, "
f"{len(table.cmap)} mappings"
)
font.close()
print("Done!")
if __name__ == "__main__":
main()
snapcraft.yaml
+8 -13
View File
@@ -5,7 +5,7 @@
name: ocrmypdf
title: OCRmyPDF
base: core22
base: core24
version: git
summary: OCRmyPDF adds a searchable text layer to scanned PDF files
description: OCRmyPDF packaged for snap
@@ -14,12 +14,13 @@ confinement: strict
icon: docs/images/logo-square-256.svg
license: MPL-2.0
architectures: [amd64]
platforms:
amd64:
environment:
TESSDATA_PREFIX: $SNAP/usr/share/tesseract-ocr/4.00/tessdata
GS_LIB: $SNAP/usr/share/ghostscript/9.55.0/Resource/Init
GS_FONTPATH: $SNAP/usr/share/ghostscript/9.55.0/Resource/Font
TESSDATA_PREFIX: $SNAP/usr/share/tesseract-ocr/5/tessdata
GS_LIB: $SNAP/usr/share/ghostscript/10.02.1/Resource/Init
GS_FONTPATH: $SNAP/usr/share/ghostscript/10.02.1/Resource/Font
LD_LIBRARY_PATH: $SNAP/usr/lib/x86_64-linux-gnu
apps:
@@ -84,11 +85,5 @@ parts:
- wheel
override-build: |
pip3 install --user dephell[full]
$HOME/.local/bin/dephell deps convert \
--from-path pyproject.toml \
--from-format pyproject \
--to-path setup.py \
--to-format setuppy
snapcraftctl build
ln -sf ../usr/lib/libsnapcraft-preload.so $SNAPCRAFT_PART_INSTALL/lib/libsnapcraft-preload.so
craftctl default
ln -sf ../usr/lib/libsnapcraft-preload.so $CRAFT_PART_INSTALL/lib/libsnapcraft-preload.so
src/ocrmypdf/__init__.py
+14
View File
@@ -11,6 +11,7 @@ from ocrmypdf import helpers, hocrtransform, pdfa, pdfinfo
from ocrmypdf._concurrent import Executor
from ocrmypdf._defaults import PROGRAM_NAME
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipelines._common import (
configure_debug_logging,
)
@@ -34,6 +35,13 @@ from ocrmypdf.exceptions import (
TesseractConfigError,
UnsupportedImageFormatError,
)
from ocrmypdf.models.ocr_element import (
Baseline,
BoundingBox,
FontInfo,
OcrClass,
OcrElement,
)
from ocrmypdf.pluginspec import OcrEngine, OrientationConfidence
hookimpl = _HookimplMarker('ocrmypdf')
@@ -41,6 +49,8 @@ hookimpl = _HookimplMarker('ocrmypdf')
__all__ = [
'__version__',
'BadArgsError',
'Baseline',
'BoundingBox',
'configure_debug_logging',
'configure_logging',
'DpiError',
@@ -48,13 +58,17 @@ __all__ = [
'Executor',
'ExitCode',
'ExitCodeException',
'FontInfo',
'helpers',
'hocrtransform',
'hookimpl',
'InputFileError',
'MissingDependencyError',
'ocr',
'OcrClass',
'OcrElement',
'OcrEngine',
'OcrOptions',
'OrientationConfidence',
'OutputFileAccessError',
'PageContext',
src/ocrmypdf/__main__.py
+2 -2
View File
@@ -15,9 +15,9 @@ from contextlib import suppress
from ocrmypdf import __version__
from ocrmypdf._pipelines.ocr import run_pipeline_cli
from ocrmypdf._plugin_manager import get_parser_options_plugins
from ocrmypdf._validation import check_options
from ocrmypdf.api import Verbosity, configure_logging
from ocrmypdf.cli import get_options_and_plugins
from ocrmypdf.exceptions import (
BadArgsError,
ExitCode,
@@ -39,7 +39,7 @@ def sigbus(*args):
def run(args=None):
"""Run the ocrmypdf command line interface."""
_parser, options, plugin_manager = get_parser_options_plugins(args=args)
options, plugin_manager = get_options_and_plugins(args=args)
with suppress(AttributeError, PermissionError):
os.nice(5)
src/ocrmypdf/_concurrent.py
+3 -3
View File
@@ -15,7 +15,7 @@ from ocrmypdf._progressbar import NullProgressBar, ProgressBar
T = TypeVar('T')
def _task_noop(*_args, **_kwargs):
def _task_noop(*_args, **_kwargs) -> None:
return
@@ -101,8 +101,8 @@ class Executor(ABC):
def setup_executor(plugin_manager) -> Executor:
pbar_class = plugin_manager.hook.get_progressbar_class()
return plugin_manager.hook.get_executor(progressbar_class=pbar_class)
pbar_class = plugin_manager.get_progressbar_class()
return plugin_manager.get_executor(progressbar_class=pbar_class)
class SerialExecutor(Executor):
src/ocrmypdf/_defaults.py
+2
View File
@@ -2,6 +2,8 @@
# SPDX-License-Identifier: MPL-2.0
# Enforce English hegemony
from __future__ import annotations
DEFAULT_LANGUAGE = 'eng'
# Default rotation threshold
src/ocrmypdf/_exec/ghostscript.py
+19 -3
View File
@@ -9,7 +9,6 @@ import logging
import os
import re
from collections import deque
from io import BytesIO
from os import fspath
from pathlib import Path
from subprocess import PIPE, CalledProcessError
@@ -23,6 +22,7 @@ from ocrmypdf.exceptions import (
SubprocessOutputError,
)
from ocrmypdf.helpers import Resolution
from ocrmypdf.pluginspec import GhostscriptRasterDevice
from ocrmypdf.subprocess import get_version, run, run_polling_stderr
COLOR_CONVERSION_STRATEGIES = frozenset(
@@ -99,15 +99,30 @@ def rasterize_pdf(
input_file: os.PathLike,
output_file: os.PathLike,
*,
raster_device: str,
raster_device: GhostscriptRasterDevice,
raster_dpi: Resolution,
pageno: int = 1,
page_dpi: Resolution | None = None,
rotation: int | None = None,
filter_vector: bool = False,
stop_on_error: bool = False,
use_cropbox: bool = False,
):
"""Rasterize one page of a PDF at resolution raster_dpi in canvas units."""
"""Rasterize one page of a PDF at resolution raster_dpi in canvas units.
Args:
input_file: The PDF file to rasterize.
output_file: The file to write the rasterized PDF to.
raster_device: The Ghostscript raster device to use to rasterize the PDF.
raster_dpi: Resolution in dots per inch at which to rasterize page.
pageno: Page number to rasterize (beginning at page 1).
page_dpi: Resolution, overriding output image DPI.
rotation: Cardinal angle, clockwise, to rotate page.
filter_vector: If True, remove vector graphics objects.
stop_on_error: If True, stop rasterizing on the first error.
use_cropbox: If True, rasterize the CropBox instead of MediaBox.
Default is False (use MediaBox).
"""
raster_dpi = raster_dpi.round(6)
if not page_dpi:
page_dpi = raster_dpi
@@ -124,6 +139,7 @@ def rasterize_pdf(
f'-dLastPage={pageno}',
f'-r{raster_dpi.x:f}x{raster_dpi.y:f}',
]
+ (['-dUseCropBox'] if use_cropbox else [])
+ (['-dFILTERVECTOR'] if filter_vector else [])
+ (['-dPDFSTOPONERROR'] if stop_on_error else [])
+ [
src/ocrmypdf/_exec/jbig2enc.py
-18
View File
@@ -31,24 +31,6 @@ def available():
return True
def convert_group(cwd, infiles, out_prefix, threshold):
args = [
'jbig2',
'-b',
out_prefix,
'--symbol-mode', # symbol mode (lossy)
'-t',
str(threshold), # threshold
# '-r', # refinement mode (lossless symbol mode, currently disabled in
# jbig2)
'--pdf',
]
args.extend(infiles)
proc = run(args, cwd=cwd, stdout=PIPE, stderr=PIPE)
proc.check_returncode()
return proc
def convert_single(cwd, infile, outfile, threshold):
args = ['jbig2', '--pdf', '-t', str(threshold), infile]
with open(outfile, 'wb') as fstdout:
src/ocrmypdf/_exec/tesseract.py
+79 -19
View File
@@ -6,8 +6,10 @@
from __future__ import annotations
import logging
import os
import re
from contextlib import suppress
from enum import IntEnum
from math import pi
from os import fspath
from pathlib import Path
@@ -26,11 +28,30 @@ from ocrmypdf.subprocess import get_version, run
log = logging.getLogger(__name__)
def _tesseract_env(omp_thread_limit: int | None) -> dict[str, str] | None:
"""Create environment dict with OMP_THREAD_LIMIT set for Tesseract subprocesses."""
if omp_thread_limit is None:
return None
env = os.environ.copy()
env['OMP_THREAD_LIMIT'] = str(omp_thread_limit)
return env
class ThresholdingMethod(IntEnum):
"""Tesseract thresholding methods for image binarization."""
AUTO = 0
OTSU = 0 # Alias for AUTO - uses Tesseract's default (legacy Otsu)
ADAPTIVE_OTSU = 1
SAUVOLA = 2
# Legacy dictionary for backward compatibility
TESSERACT_THRESHOLDING_METHODS: dict[str, int] = {
'auto': 0,
'otsu': 0,
'adaptive-otsu': 1,
'sauvola': 2,
'auto': ThresholdingMethod.AUTO,
'otsu': ThresholdingMethod.OTSU,
'adaptive-otsu': ThresholdingMethod.ADAPTIVE_OTSU,
'sauvola': ThresholdingMethod.SAUVOLA,
}
@@ -155,7 +176,10 @@ def _parse_tesseract_output(binary_output: bytes) -> dict[str, str]:
def get_orientation(
input_file: Path, engine_mode: int | None, timeout: float
input_file: Path,
engine_mode: int | None,
timeout: float,
omp_thread_limit: int | None = None,
) -> OrientationConfidence:
args_tesseract = tess_base_args(['osd'], engine_mode) + [
'--psm',
@@ -165,15 +189,24 @@ def get_orientation(
]
try:
p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
p = run(
args_tesseract,
stdout=PIPE,
stderr=STDOUT,
timeout=timeout,
check=True,
env=_tesseract_env(omp_thread_limit),
)
except TimeoutExpired:
return OrientationConfidence(angle=0, confidence=0.0)
except CalledProcessError as e:
tesseract_log_output(e.stdout)
tesseract_log_output(e.stderr)
# Check both stdout (e.output) and stderr for known non-fatal messages
all_output = (e.output or b'') + (e.stderr or b'')
if (
b'Too few characters. Skipping this page' in e.output
or b'Image too large' in e.output
b'Too few characters. Skipping this page' in all_output
or b'Image too large' in all_output
):
return OrientationConfidence(0, 0)
raise SubprocessOutputError() from e
@@ -199,7 +232,11 @@ def _is_empty_page_error(exc):
def get_deskew(
input_file: Path, languages: list[str], engine_mode: int | None, timeout: float
input_file: Path,
languages: list[str],
engine_mode: int | None,
timeout: float,
omp_thread_limit: int | None = None,
) -> float:
"""Gets angle to deskew this page, in degrees."""
args_tesseract = tess_base_args(languages, engine_mode) + [
@@ -210,7 +247,14 @@ def get_deskew(
]
try:
p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
p = run(
args_tesseract,
stdout=PIPE,
stderr=STDOUT,
timeout=timeout,
check=True,
env=_tesseract_env(omp_thread_limit),
)
except TimeoutExpired:
return 0.0
except CalledProcessError as e:
@@ -243,9 +287,9 @@ def tesseract_log_output(stream: bytes) -> None:
lines = text.splitlines()
for line in lines:
if line.startswith("Tesseract Open Source"):
continue
elif line.startswith("Warning in pixReadMem"):
if line.startswith(
("Tesseract Open Source", "Warning in pixReadMem")
):
continue
elif 'diacritics' in line:
tlog.warning("lots of diacritics - possibly poor OCR")
@@ -294,9 +338,10 @@ def generate_hocr(
tessconfig: list[str],
timeout: float,
pagesegmode: int,
thresholding: int,
thresholding: ThresholdingMethod,
user_words,
user_patterns,
omp_thread_limit: int | None = None,
) -> None:
"""Generate a hOCR file, which must be converted to PDF."""
prefix = output_hocr.with_suffix('')
@@ -306,7 +351,7 @@ def generate_hocr(
if pagesegmode is not None:
args_tesseract.extend(['--psm', str(pagesegmode)])
if thresholding != 0 and has_thresholding():
if thresholding != ThresholdingMethod.AUTO and has_thresholding():
args_tesseract.extend(['-c', f'thresholding_method={thresholding}'])
if user_words:
@@ -320,7 +365,14 @@ def generate_hocr(
args_tesseract.extend([fspath(input_file), fspath(prefix), 'hocr', 'txt'])
args_tesseract.extend(tessconfig)
try:
p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
p = run(
args_tesseract,
stdout=PIPE,
stderr=STDOUT,
timeout=timeout,
check=True,
env=_tesseract_env(omp_thread_limit),
)
stdout = p.stdout
except TimeoutExpired:
# Generate a HOCR file with no recognized text if tesseract times out
@@ -360,9 +412,10 @@ def generate_pdf(
tessconfig: list[str],
timeout: float,
pagesegmode: int,
thresholding: int,
thresholding: ThresholdingMethod,
user_words,
user_patterns,
omp_thread_limit: int | None = None,
) -> None:
"""Generate a PDF using Tesseract's internal PDF generator.
@@ -376,7 +429,7 @@ def generate_pdf(
args_tesseract.extend(['-c', 'textonly_pdf=1'])
if thresholding != 0 and has_thresholding():
if thresholding != ThresholdingMethod.AUTO and has_thresholding():
args_tesseract.extend(['-c', f'thresholding_method={thresholding}'])
if user_words:
@@ -393,7 +446,14 @@ def generate_pdf(
args_tesseract.extend([fspath(input_file), fspath(prefix), 'pdf', 'txt'])
args_tesseract.extend(tessconfig)
try:
p = run(args_tesseract, stdout=PIPE, stderr=STDOUT, timeout=timeout, check=True)
p = run(
args_tesseract,
stdout=PIPE,
stderr=STDOUT,
timeout=timeout,
check=True,
env=_tesseract_env(omp_thread_limit),
)
stdout = p.stdout
with suppress(FileNotFoundError):
prefix.with_suffix('.txt').replace(output_text)
src/ocrmypdf/_exec/verapdf.py
+108
View File
@@ -0,0 +1,108 @@
# SPDX-FileCopyrightText: 2024 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Interface to verapdf executable."""
from __future__ import annotations
import json
import logging
from pathlib import Path
from subprocess import PIPE
from typing import NamedTuple
from packaging.version import Version
from ocrmypdf.exceptions import MissingDependencyError
from ocrmypdf.subprocess import get_version, run
log = logging.getLogger(__name__)
class ValidationResult(NamedTuple):
"""Result of PDF/A validation."""
valid: bool
failed_rules: int
message: str
def version() -> Version:
"""Get verapdf version."""
return Version(get_version('verapdf', regex=r'veraPDF (\d+(\.\d+)*)'))
def available() -> bool:
"""Check if verapdf is available."""
try:
version()
except MissingDependencyError:
return False
return True
def output_type_to_flavour(output_type: str) -> str:
"""Map OCRmyPDF output_type to verapdf flavour.
Args:
output_type: One of 'pdfa', 'pdfa-1', 'pdfa-2', 'pdfa-3'
Returns:
verapdf flavour string like '1b', '2b', '3b'
"""
mapping = {
'pdfa': '2b',
'pdfa-1': '1b',
'pdfa-2': '2b',
'pdfa-3': '3b',
}
return mapping.get(output_type, '2b')
def validate(input_file: Path, flavour: str) -> ValidationResult:
"""Validate a PDF against a PDF/A profile.
Args:
input_file: Path to PDF file to validate
flavour: verapdf flavour (1a, 1b, 2a, 2b, 2u, 3a, 3b, 3u)
Returns:
ValidationResult with validation status
"""
args = [
'verapdf',
'--format',
'json',
'--flavour',
flavour,
str(input_file),
]
try:
proc = run(args, stdout=PIPE, stderr=PIPE, check=False)
except FileNotFoundError as e:
raise MissingDependencyError('verapdf') from e
try:
result = json.loads(proc.stdout)
jobs = result.get('report', {}).get('jobs', [])
if not jobs:
return ValidationResult(False, -1, 'No validation jobs in result')
validation_results = jobs[0].get('validationResult', [])
if not validation_results:
return ValidationResult(False, -1, 'No validation result in output')
validation_result = validation_results[0]
details = validation_result.get('details', {})
failed_rules = details.get('failedRules', 0)
if failed_rules == 0:
return ValidationResult(True, 0, 'PDF/A validation passed')
else:
return ValidationResult(
False,
failed_rules,
f'PDF/A validation failed with {failed_rules} rule violations',
)
except (json.JSONDecodeError, KeyError, TypeError) as e:
log.debug('Failed to parse verapdf output: %s', e)
return ValidationResult(False, -1, f'Failed to parse verapdf output: {e}')
src/ocrmypdf/_graft.py
+463 -195
View File
@@ -7,30 +7,155 @@ from __future__ import annotations
import logging
from contextlib import suppress
from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from ocrmypdf.hocrtransform import OcrElement
from pikepdf import (
Dictionary,
Matrix,
Name,
Operator,
Page,
Pdf,
PdfError,
Stream,
parse_content_stream,
unparse_content_stream,
)
from ocrmypdf._jobcontext import PdfContext
from ocrmypdf._options import ProcessingMode
from ocrmypdf._pipeline import VECTOR_PAGE_DPI
class RenderMode(Enum):
"""Controls where the OCR text layer is placed relative to page content.
ON_TOP: Text layer renders above page content (reserved for future use).
UNDERNEATH: Text layer renders below page content (current default behavior).
"""
ON_TOP = 0
UNDERNEATH = 1
@dataclass
class Fpdf2PageInfo:
"""Information needed to render and graft an fpdf2 page."""
pageno: int
hocr_path: Path
dpi: float
autorotate_correction: int
emplaced_page: bool
@dataclass
class Fpdf2ParsedPage:
"""Parsed page data ready for fpdf2 rendering."""
pageno: int
ocr_tree: OcrElement
dpi: float
autorotate_correction: int
emplaced_page: bool
# Alias for backward compatibility with plan documentation
Fpdf2DirectPage = Fpdf2ParsedPage
def _compute_text_misalignment(
content_rotation: int, autorotate_correction: int, emplaced_page: bool
) -> int:
"""Compute rotation needed to align text layer with page content.
Args:
content_rotation: Original page /Rotate value (degrees).
autorotate_correction: Rotation applied during rasterization (degrees).
emplaced_page: Whether the page content was replaced with rasterized image.
Returns:
Rotation in degrees to apply to text layer to align with content.
"""
if emplaced_page:
# New image is upright after autorotation was applied
content_rotation = autorotate_correction
text_rotation = autorotate_correction
return (text_rotation - content_rotation) % 360
def _compute_page_rotation(
content_rotation: int, autorotate_correction: int, emplaced_page: bool
) -> int:
"""Compute final page /Rotate value after grafting.
Args:
content_rotation: Original page /Rotate value (degrees).
autorotate_correction: Rotation applied during rasterization (degrees).
emplaced_page: Whether the page content was replaced with rasterized image.
Returns:
Final /Rotate value for the page.
"""
if emplaced_page:
content_rotation = autorotate_correction
return (content_rotation - autorotate_correction) % 360
def _build_text_layer_ctm(
text_width: float,
text_height: float,
page_width: float,
page_height: float,
page_origin_x: float,
page_origin_y: float,
text_rotation: int,
):
"""Build transformation matrix to align text layer with page content.
Args:
text_width: Width of text layer mediabox.
text_height: Height of text layer mediabox.
page_width: Width of target page mediabox.
page_height: Height of target page mediabox.
page_origin_x: X origin of target page mediabox.
page_origin_y: Y origin of target page mediabox.
text_rotation: Rotation in degrees (clockwise) to apply to text layer.
Returns:
pikepdf.Matrix transformation matrix, or None if no rotation needed.
"""
if text_rotation == 0:
return None
from pikepdf import Matrix
wt, ht = text_width, text_height
# Center text, rotate, scale to fit page, then position at page origin
translate = Matrix().translated(-wt / 2, -ht / 2)
untranslate = Matrix().translated(page_width / 2, page_height / 2)
corner = Matrix().translated(page_origin_x, page_origin_y)
# Negate rotation because input is clockwise angle
rotate = Matrix().rotated(-text_rotation % 360)
# Swap dimensions if 90 or 270 degree rotation
if text_rotation in (90, 270):
wt, ht = ht, wt
# Scale to fit page dimensions
scale_x = page_width / wt if wt else 1.0
scale_y = page_height / ht if ht else 1.0
scale = Matrix().scaled(scale_x, scale_y)
return translate @ rotate @ scale @ untranslate @ corner
log = logging.getLogger(__name__)
MAX_REPLACE_PAGES = 100
@@ -41,22 +166,6 @@ def _ensure_dictionary(obj: Dictionary | Stream, name: Name):
return obj[name]
def _update_resources(
*,
obj: Dictionary | Stream,
font: Dictionary | None,
font_key: Name | None,
):
"""Update this obj's fonts with a reference to the Glyphless font.
obj can be a page or Form XObject.
"""
resources = _ensure_dictionary(obj, Name.Resources)
fonts = _ensure_dictionary(resources, Name.Font)
if font_key is not None and font_key not in fonts:
fonts[font_key] = font
def strip_invisible_text(pdf: Pdf, page: Page):
stream = []
in_text_obj = False
@@ -72,12 +181,9 @@ def strip_invisible_text(pdf: Pdf, page: Page):
render_mode_stack.append(render_mode)
if operator == Operator('Q'):
try:
# IndexError is raised if stack is empty; try to carry on
with suppress(IndexError):
render_mode = render_mode_stack.pop()
except IndexError:
# Stack underflow: content stream is malformed
# but try to carry on
pass
if not in_text_obj:
if operator == Operator('BT'):
@@ -105,27 +211,45 @@ class OcrGrafter:
self.path_base = context.origin
self.pdf_base = Pdf.open(self.path_base)
self.font: Dictionary | None = None
self.font_key: Name | None = None
self.pdfinfo = context.pdfinfo
self.output_file = context.get_path('graft_layers.pdf')
self.emplacements = 1
self.interim_count = 0
self.render_mode = RenderMode.UNDERNEATH
# Check renderer type
pdf_renderer = context.options.pdf_renderer
self.use_sandwich_renderer = pdf_renderer == 'sandwich'
# For fpdf2: accumulate pages before rendering
self.fpdf2_hocr_pages: list[Fpdf2PageInfo] = []
self.fpdf2_parsed_pages: list[Fpdf2ParsedPage] = []
def graft_page(
self,
*,
pageno: int,
image: Path | None,
textpdf: Path | None,
ocr_output: Path | None,
ocr_tree: OcrElement | None,
autorotate_correction: int,
):
if textpdf and not self.font:
self.font, self.font_key = self._find_font(textpdf)
"""Graft OCR output onto a page of the base PDF.
Args:
pageno: Zero-based page number.
image: Path to the visible page image PDF, or None if not replacing.
ocr_output: Path to OCR output file. For fpdf2 renderer this is an
hOCR file; for sandwich renderer this is a text-only PDF.
ocr_tree: OCR tree for fpdf2 renderer.
autorotate_correction: Orientation correction in degrees (0, 90, 180, 270).
"""
if ocr_output and ocr_tree:
raise ValueError(
'Cannot specify both ocr_output and ocr_tree for fpdf2 renderer'
)
# Handle image emplacement first
emplaced_page = False
content_rotation = self.pdfinfo[pageno].rotation
path_image = Path(image).resolve() if image else None
@@ -144,195 +268,339 @@ class OcrGrafter:
del self.pdf_base.pages[-1]
emplaced_page = True
# Calculate if the text is misaligned compared to the content
if emplaced_page:
content_rotation = autorotate_correction
text_rotation = autorotate_correction
text_misaligned = (text_rotation - content_rotation) % 360
log.debug(
f"Text rotation: (text, autorotate, content) -> text misalignment = "
f"({text_rotation}, {autorotate_correction}, {content_rotation}) -> "
f"{text_misaligned}"
)
if textpdf and self.font:
if self.font_key is None:
raise ValueError("Font key is not set")
# Graft the text layer onto this page, whether new or old, possibly
# rotating the text layer by the amount is misaligned.
strip_old = self.context.options.redo_ocr
self._graft_text_layer(
page_num=pageno + 1,
textpdf=textpdf,
font=self.font,
font_key=self.font_key,
text_rotation=text_misaligned,
strip_old_text=strip_old,
)
# Correct the overall page rotation if needed, now that the text and content
# are aligned
page_rotation = (content_rotation - autorotate_correction) % 360
self.pdf_base.pages[pageno].Rotate = page_rotation
log.debug(
f"Page rotation: (content, auto) -> page = "
f"({content_rotation}, {autorotate_correction}) -> {page_rotation}"
)
if self.emplacements % MAX_REPLACE_PAGES == 0:
self.save_and_reload()
def save_and_reload(self) -> None:
"""Save and reload the Pdf.
This will keep a lid on our memory usage for very large files. Attach
the font to page 1 even if page 1 doesn't use it, so we have a way to get it
back.
"""
page0 = self.pdf_base.pages[0]
_update_resources(obj=page0.obj, font=self.font, font_key=self.font_key)
# We cannot read and write the same file, that will corrupt it
# but we don't to keep more copies than we need to. Delete intermediates.
# {interim_count} is the opened file we were updating
# {interim_count - 1} can be deleted
# {interim_count + 1} is the new file will produce and open
old_file = self.output_file.with_suffix(f'.working{self.interim_count - 1}.pdf')
if not self.context.options.keep_temporary_files:
with suppress(FileNotFoundError):
old_file.unlink()
next_file = self.output_file.with_suffix(
f'.working{self.interim_count + 1}.pdf'
)
self.pdf_base.save(next_file)
self.pdf_base.close()
self.pdf_base = Pdf.open(next_file)
self.font, self.font_key = None, None # Ensure we reacquire this information
self.interim_count += 1
if self.use_sandwich_renderer:
# Sandwich renderer: graft pre-rendered PDF immediately
if ocr_output:
text_misaligned = _compute_text_misalignment(
content_rotation, autorotate_correction, emplaced_page
)
self._graft_sandwich_text_layer(
pageno=pageno,
textpdf=ocr_output,
text_rotation=text_misaligned,
)
page_rotation = _compute_page_rotation(
content_rotation, autorotate_correction, emplaced_page
)
self.pdf_base.pages[pageno].Rotate = page_rotation
else:
# fpdf2 renderer: accumulate page info for batch rendering.
# The hOCR coordinates are in the corrected (upright) coordinate system.
# We store autorotate_correction and emplaced_page to set the final
# page /Rotate tag after grafting.
if ocr_tree:
self.fpdf2_parsed_pages.append(
Fpdf2ParsedPage(
ocr_tree=ocr_tree,
pageno=pageno,
autorotate_correction=autorotate_correction,
emplaced_page=emplaced_page,
dpi=self.pdfinfo[pageno].dpi.to_scalar(),
)
)
if ocr_output:
self.fpdf2_hocr_pages.append(
Fpdf2PageInfo(
hocr_path=ocr_output,
pageno=pageno,
autorotate_correction=autorotate_correction,
emplaced_page=emplaced_page,
dpi=self.pdfinfo[pageno].dpi.to_scalar(),
)
)
def finalize(self):
# Can have hocr OR parsed pages OR neither (no OCR), but not both
assert not (
self.fpdf2_hocr_pages and self.fpdf2_parsed_pages
), "Can't have both hocr and ocrtree pages"
if self.fpdf2_hocr_pages:
# Render all pages with fpdf2, then graft
parsed_pages = self._parse_hocr_pages()
self.fpdf2_parsed_pages = parsed_pages
if self.fpdf2_parsed_pages:
self._render_and_graft_fpdf2_pages()
self.pdf_base.save(self.output_file)
self.pdf_base.close()
return self.output_file
def _find_font(self, text: Path) -> tuple[Dictionary | None, Name | None]:
"""Copy a font from the filename text into pdf_base."""
font, font_key = None, None
possible_font_names = ('/f-0-0', '/F1')
try:
with Pdf.open(text) as pdf_text:
try:
pdf_text_fonts = pdf_text.pages[0].Resources.get(
Name.Font, Dictionary()
)
except (AttributeError, IndexError, KeyError):
return None, None
if not isinstance(pdf_text_fonts, Dictionary):
log.warning("Page fonts are not stored in a dictionary")
return None, None
pdf_text_font = None
for f in possible_font_names:
pdf_text_font = pdf_text_fonts.get(f, None)
if pdf_text_font is not None:
font_key = Name(f)
break
if pdf_text_font:
font = self.pdf_base.copy_foreign(pdf_text_font)
if not isinstance(font, Dictionary):
log.warning("Font is not a dictionary")
font, font_key = None, None
return font, font_key
except (FileNotFoundError, PdfError):
# PdfError occurs if a 0-length file is written e.g. due to OCR timeout
return None, None
def _parse_hocr_pages(self):
"""Render all pages to multi-page PDF with shared fonts, then graft."""
from ocrmypdf.hocrtransform.hocr_parser import HocrParser
def _graft_text_layer(
log.info(
"Parsing %d pages with HocrParser",
len(self.fpdf2_hocr_pages),
)
# Parse all hOCR files and collect OcrElements
pages_data: list[Fpdf2ParsedPage] = []
for page_info in self.fpdf2_hocr_pages:
if page_info.hocr_path.stat().st_size == 0:
continue # Skip empty pages
# Parse hOCR to OcrElement
parser = HocrParser(page_info.hocr_path)
ocr_tree = parser.parse()
# Use DPI from hOCR (scan_res) which reflects actual rasterization DPI.
# Fall back to pdfinfo DPI or VECTOR_PAGE_DPI for vector-only pages.
effective_dpi = ocr_tree.dpi or page_info.dpi or float(VECTOR_PAGE_DPI)
pages_data.append(
Fpdf2ParsedPage(
pageno=page_info.pageno,
ocr_tree=ocr_tree,
dpi=effective_dpi,
autorotate_correction=page_info.autorotate_correction,
emplaced_page=page_info.emplaced_page,
)
)
return pages_data
def _render_and_graft_fpdf2_pages(self):
font_dir = Path(__file__).parent / "data"
# Render all pages to single PDF
multi_page_pdf_path = self.context.get_path('fpdf2_multipage.pdf')
from ocrmypdf.font import MultiFontManager
from ocrmypdf.fpdf_renderer import Fpdf2MultiPageRenderer
multi_font_manager = MultiFontManager(font_dir)
# Build renderer input as (pageno, ocr_tree, dpi) tuples
renderer_pages_data = [
(parsed.pageno, parsed.ocr_tree, parsed.dpi)
for parsed in self.fpdf2_parsed_pages
]
renderer = Fpdf2MultiPageRenderer(
pages_data=renderer_pages_data,
multi_font_manager=multi_font_manager,
invisible_text=True,
)
renderer.render(multi_page_pdf_path)
# Now graft each page from the multi-page PDF
with Pdf.open(multi_page_pdf_path) as pdf_text:
for idx, parsed in enumerate(self.fpdf2_parsed_pages):
# Copy page from multi-page PDF
text_page = pdf_text.pages[idx]
content_rotation = self.pdfinfo[parsed.pageno].rotation
text_misaligned = _compute_text_misalignment(
content_rotation,
parsed.autorotate_correction,
parsed.emplaced_page,
)
self._graft_fpdf2_text_layer(parsed.pageno, text_page, text_misaligned)
page_rotation = _compute_page_rotation(
content_rotation,
parsed.autorotate_correction,
parsed.emplaced_page,
)
self.pdf_base.pages[parsed.pageno].Rotate = page_rotation
# Clean up multi-page PDF if not keeping temp files
if not self.context.options.keep_temporary_files:
with suppress(FileNotFoundError):
multi_page_pdf_path.unlink()
def _graft_fpdf2_text_layer(self, pageno: int, text_page: Page, text_rotation: int):
"""Graft a single text page onto the base PDF.
Similar to existing _graft_text_layer but works with
already-rendered pikepdf Page instead of file path.
Args:
pageno: Zero-based page number.
text_page: The text-only PDF page to graft.
text_rotation: Rotation to apply to align text with content (degrees).
"""
from pikepdf import Array
base_page = self.pdf_base.pages[pageno]
# Extract content stream from text_page
text_contents = text_page.Contents.read_bytes()
# Get the mediabox from the text page
mediabox = Array([float(x) for x in text_page.mediabox]) # type: ignore[misc]
wt = float(mediabox[2]) - float(mediabox[0])
ht = float(mediabox[3]) - float(mediabox[1])
# Get base page mediabox
base_mediabox = base_page.mediabox
wp = float(base_mediabox[2]) - float(base_mediabox[0])
hp = float(base_mediabox[3]) - float(base_mediabox[1])
# Create Form XObject from text page content
base_resources = _ensure_dictionary(base_page.obj, Name.Resources)
base_xobjs = _ensure_dictionary(base_resources, Name.XObject)
text_xobj_name = Name.random(prefix="OCR-")
xobj = self.pdf_base.make_stream(text_contents)
base_xobjs[text_xobj_name] = xobj
xobj.Type = Name.XObject
xobj.Subtype = Name.Form
xobj.FormType = 1
xobj.BBox = mediabox
# Copy resources from text page's Resources to xobj
# We need to handle this carefully since text_page is from a foreign PDF
if hasattr(text_page, 'Resources') and text_page.Resources:
# Create empty Resources dictionary for xobj
xobj_resources = _ensure_dictionary(xobj, Name.Resources)
# Copy fonts if they exist
if Name.Font in text_page.Resources:
xobj_fonts = _ensure_dictionary(xobj_resources, Name.Font)
text_fonts = text_page.Resources[Name.Font]
# Copy each font from the foreign PDF
for font_name, font_obj in text_fonts.items():
xobj_fonts[font_name] = self.pdf_base.copy_foreign(font_obj)
# Copy ExtGState (graphics state) if it exists - needed for transparency
if Name.ExtGState in text_page.Resources:
xobj_extstates = _ensure_dictionary(xobj_resources, Name.ExtGState)
text_extstates = text_page.Resources[Name.ExtGState]
# Copy each graphics state from the foreign PDF
for gs_name, gs_obj in text_extstates.items():
xobj_extstates[gs_name] = self.pdf_base.copy_foreign(gs_obj)
# Build transformation matrix for rotation and scaling
ctm = _build_text_layer_ctm(
wt,
ht,
wp,
hp,
float(base_mediabox[0]),
float(base_mediabox[1]),
text_rotation,
)
if ctm is not None:
pdf_draw_xobj = (
(b'q %s cm\n' % ctm.encode()) + (b'%s Do\n' % text_xobj_name) + b'Q\n'
)
else:
pdf_draw_xobj = b'q\n' + (b'%s Do\n' % text_xobj_name) + b'\nQ\n'
new_text_layer = Stream(self.pdf_base, pdf_draw_xobj)
# Strip old invisible text if redo mode is enabled
if self.context.options.mode == ProcessingMode.redo:
strip_invisible_text(self.pdf_base, base_page)
# Add text layer to base page
base_page.contents_coalesce()
base_page.contents_add(
new_text_layer, prepend=self.render_mode == RenderMode.UNDERNEATH
)
base_page.contents_coalesce()
def _graft_sandwich_text_layer(
self,
*,
page_num: int,
pageno: int,
textpdf: Path,
font: Dictionary,
font_key: Name,
text_rotation: int,
strip_old_text: bool,
):
"""Insert the text layer from text page 0 on to pdf_base at page_num."""
# pylint: disable=invalid-name
"""Graft a pre-rendered text-only PDF onto the base PDF.
log.debug("Grafting")
This is used by the sandwich renderer which generates PDFs directly
from Tesseract rather than going through hOCR.
"""
from pikepdf import PdfError
log.debug("Grafting sandwich text layer")
if Path(textpdf).stat().st_size == 0:
return
# This is a pointer indicating a specific page in the base file
with Pdf.open(textpdf) as pdf_text:
pdf_text_contents = pdf_text.pages[0].Contents.read_bytes()
try:
with Pdf.open(textpdf) as pdf_text:
pdf_text_contents = pdf_text.pages[0].Contents.read_bytes()
base_page = self.pdf_base.pages.p(page_num)
base_page = self.pdf_base.pages[pageno]
# The text page always will be oriented up by this stage but the original
# content may have a rotation applied. Wrap the text stream with a rotation
# so it will be oriented the same way as the rest of the page content.
# (Previous versions OCRmyPDF rotated the content layer to match the text.)
mediabox = pdf_text.pages[0].mediabox
wt, ht = mediabox[2] - mediabox[0], mediabox[3] - mediabox[1]
# Get font from the text PDF
pdf_text_fonts = pdf_text.pages[0].Resources.get(
Name.Font, Dictionary()
)
font = None
font_key = None
for f in ('/f-0-0', '/F1'):
pdf_text_font = pdf_text_fonts.get(f, None)
if pdf_text_font is not None:
font_key = Name(f)
font = self.pdf_base.copy_foreign(pdf_text_font)
break
mediabox = base_page.mediabox
wp, hp = mediabox[2] - mediabox[0], mediabox[3] - mediabox[1]
# Get mediabox dimensions for rotation calculations
mediabox = pdf_text.pages[0].mediabox
wt = float(mediabox[2]) - float(mediabox[0])
ht = float(mediabox[3]) - float(mediabox[1])
translate = Matrix().translated(-wt / 2, -ht / 2)
untranslate = Matrix().translated(wp / 2, hp / 2)
corner = Matrix().translated(mediabox[0], mediabox[1])
# -rotation because the input is a clockwise angle and this formula
# uses CCW
text_rotation = -text_rotation % 360
rotate = Matrix().rotated(text_rotation)
base_mediabox = base_page.mediabox
wp = float(base_mediabox[2]) - float(base_mediabox[0])
hp = float(base_mediabox[3]) - float(base_mediabox[1])
# Because of rounding of DPI, we might get a text layer that is not
# identically sized to the target page. Scale to adjust. Normally this
# is within 0.998.
if text_rotation in (90, 270):
wt, ht = ht, wt
scale_x = wp / wt
scale_y = hp / ht
# Build transformation matrix for rotation and scaling
ctm = _build_text_layer_ctm(
wt,
ht,
wp,
hp,
float(base_mediabox[0]),
float(base_mediabox[1]),
text_rotation,
)
log.debug("Grafting with ctm %r", ctm)
# log.debug('%r', scale_x, scale_y)
scale = Matrix().scaled(scale_x, scale_y)
# Create Form XObject
base_resources = _ensure_dictionary(base_page.obj, Name.Resources)
base_xobjs = _ensure_dictionary(base_resources, Name.XObject)
text_xobj_name = Name.random(prefix="OCR-")
xobj = self.pdf_base.make_stream(pdf_text_contents)
base_xobjs[text_xobj_name] = xobj
xobj.Type = Name.XObject
xobj.Subtype = Name.Form
xobj.FormType = 1
xobj.BBox = base_mediabox
# Translate the text so it is centered at (0, 0), rotate it there, adjust
# for a size different between initial and text PDF, then untranslate, and
# finally move the lower left corner to match the mediabox.
ctm = translate @ rotate @ scale @ untranslate @ corner
log.debug("Grafting with ctm %r", ctm)
# Add font to xobj resources
if font_key is not None and font is not None:
xobj_resources = _ensure_dictionary(xobj, Name.Resources)
xobj_fonts = _ensure_dictionary(xobj_resources, Name.Font)
if font_key not in xobj_fonts:
xobj_fonts[font_key] = font
base_resources = _ensure_dictionary(base_page.obj, Name.Resources)
base_xobjs = _ensure_dictionary(base_resources, Name.XObject)
text_xobj_name = Name.random(prefix="OCR-")
xobj = self.pdf_base.make_stream(pdf_text_contents)
base_xobjs[text_xobj_name] = xobj
xobj.Type = Name.XObject
xobj.Subtype = Name.Form
xobj.FormType = 1
xobj.BBox = mediabox
_update_resources(obj=xobj, font=font, font_key=font_key)
if ctm is not None:
pdf_draw_xobj = (
(b'q %s cm\n' % ctm.encode())
+ (b'%s Do\n' % text_xobj_name)
+ b'\nQ\n'
)
else:
pdf_draw_xobj = b'q\n' + (b'%s Do\n' % text_xobj_name) + b'\nQ\n'
new_text_layer = Stream(self.pdf_base, pdf_draw_xobj)
pdf_draw_xobj = (
(b'q %s cm\n' % ctm.encode()) + (b'%s Do\n' % text_xobj_name) + b'\nQ\n'
)
new_text_layer = Stream(self.pdf_base, pdf_draw_xobj)
if self.context.options.mode == ProcessingMode.redo:
strip_invisible_text(self.pdf_base, base_page)
base_page.contents_coalesce()
base_page.contents_add(
new_text_layer, prepend=self.render_mode == RenderMode.UNDERNEATH
)
base_page.contents_coalesce()
if strip_old_text:
strip_invisible_text(self.pdf_base, base_page)
base_page.contents_coalesce()
if self.render_mode == RenderMode.ON_TOP:
# Add q/Q to ensure content we append is drawn correctly
# Strictly speaking this needs to trace the whole q/Q stack in case
# stack is not balanced.
original = base_page.Contents.read_bytes()
base_page.Contents.write(b'q\n' + original + b'\nQ\n')
base_page.contents_add(
new_text_layer, prepend=self.render_mode == RenderMode.UNDERNEATH
)
base_page.contents_coalesce()
_update_resources(obj=base_page.obj, font=font, font_key=font_key)
# Add font to page resources
if font_key is not None and font is not None:
page_resources = _ensure_dictionary(base_page.obj, Name.Resources)
page_fonts = _ensure_dictionary(page_resources, Name.Font)
if font_key not in page_fonts:
page_fonts[font_key] = font
except (FileNotFoundError, PdfError):
# PdfError occurs if a 0-length file is written e.g. due to OCR timeout
pass
src/ocrmypdf/_jobcontext.py
+36 -15
View File
@@ -5,29 +5,31 @@
from __future__ import annotations
import os
from argparse import Namespace
from collections.abc import Iterator
from copy import copy
from pathlib import Path
from typing import TYPE_CHECKING
from pluggy import PluginManager
from ocrmypdf._options import OcrOptions
from ocrmypdf.pdfinfo import PdfInfo
from ocrmypdf.pdfinfo.info import PageInfo
if TYPE_CHECKING:
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
class PdfContext:
"""Holds the context for a particular run of the pipeline."""
options: Namespace #: The specified options for processing this PDF.
options: OcrOptions #: The specified options for processing this PDF.
origin: Path #: The filename of the original input file.
pdfinfo: PdfInfo #: Detailed data for this PDF.
plugin_manager: PluginManager #: PluginManager for processing the current PDF.
plugin_manager: (
OcrmypdfPluginManager #: PluginManager for processing the current PDF.
)
def __init__(
self,
options: Namespace,
options: OcrOptions,
work_folder: Path,
origin: Path,
pdfinfo: PdfInfo,
@@ -65,21 +67,27 @@ class PageContext:
Must be pickle-able, so stores only intrinsic/simple data elements or those
capable of their serializing themselves via ``__getstate__``.
Note: Uses OcrOptions with JSON serialization for multiprocessing compatibility.
"""
options: Namespace #: The specified options for processing this PDF.
origin: Path #: The filename of the original input file.
pageno: int #: This page number (zero-based).
pageinfo: PageInfo #: Information on this page.
plugin_manager: PluginManager #: PluginManager for processing the current PDF.
plugin_manager: (
OcrmypdfPluginManager #: PluginManager for processing the current PDF.
)
def __init__(self, pdf_context: PdfContext, pageno):
self.work_folder = pdf_context.work_folder
self.origin = pdf_context.origin
# Store OcrOptions directly instead of Namespace
self.options = pdf_context.options
self.pageno = pageno
self.pageinfo = pdf_context.pdfinfo[pageno]
self.plugin_manager = pdf_context.plugin_manager
# Ensure no reference to PdfContext which contains OcrOptions
self._pdf_context = None
def get_path(self, name: str) -> Path:
"""Generate a ``Path`` for a file that is part of processing this page.
@@ -92,9 +100,22 @@ class PageContext:
def __getstate__(self):
state = self.__dict__.copy()
state['options'] = copy(self.options)
if not isinstance(state['options'].input_file, str | bytes | os.PathLike):
state['options'].input_file = 'stream'
if not isinstance(state['options'].output_file, str | bytes | os.PathLike):
state['options'].output_file = 'stream'
options_json = self.options.model_dump_json_safe()
state['options_json'] = options_json
# Remove the OcrOptions object to avoid pickle issues
del state['options']
# Remove any potential references to Pydantic objects
state.pop('_pdf_context', None)
return state
def __setstate__(self, state):
self.__dict__.update(state)
# Reconstruct OcrOptions from JSON if available
if 'options_json' in state:
from ocrmypdf._options import OcrOptions
self.options = OcrOptions.model_validate_json_safe(state['options_json'])
# Otherwise, we have a fallback Namespace (shouldn't happen in normal operation)
# Leave it as-is for compatibility
src/ocrmypdf/_metadata.py
+9 -11
View File
@@ -5,9 +5,9 @@
from __future__ import annotations
import datetime as dt
import logging
import os
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
@@ -15,7 +15,6 @@ from pikepdf import Dictionary, Name, Pdf
from pikepdf import __version__ as PIKEPDF_VERSION
from pikepdf.models.metadata import PdfMetadata, encode_pdf_date
from ocrmypdf._annots import remove_broken_goto_annotations
from ocrmypdf._defaults import PROGRAM_NAME
from ocrmypdf._jobcontext import PdfContext
from ocrmypdf._version import __version__ as OCRMYPF_VERSION
@@ -48,11 +47,13 @@ def get_docinfo(base_pdf: Pdf, context: PdfContext) -> dict[str, str]:
if options.subject:
pdfmark['/Subject'] = options.subject
creator_tag = context.plugin_manager.hook.get_ocr_engine().creator_tag(options)
creator_tag = context.plugin_manager.get_ocr_engine(options=options).creator_tag(
options
)
pdfmark['/Creator'] = f'{PROGRAM_NAME} {OCRMYPF_VERSION} / {creator_tag}'
pdfmark['/Producer'] = f'pikepdf {PIKEPDF_VERSION}'
pdfmark['/ModDate'] = encode_pdf_date(datetime.now(timezone.utc))
pdfmark['/ModDate'] = encode_pdf_date(dt.datetime.now(dt.UTC))
return pdfmark
@@ -99,9 +100,7 @@ def should_linearize(working_file: Path, context: PdfContext) -> bool:
For smaller files, linearization is not worth the effort.
"""
filesize = os.stat(working_file).st_size
if filesize > (context.options.fast_web_view * 1_000_000):
return True
return False
return filesize > (context.options.fast_web_view * 1_000_000)
def _fix_metadata(meta_original: PdfMetadata, meta_pdf: PdfMetadata):
@@ -109,12 +108,11 @@ def _fix_metadata(meta_original: PdfMetadata, meta_pdf: PdfMetadata):
# ensure consistency with Ghostscript.
if 'xmp:CreateDate' not in meta_pdf:
meta_pdf['xmp:CreateDate'] = meta_pdf.get('xmp:ModifyDate', '')
if meta_pdf.get('dc:title') == 'Untitled':
if meta_pdf.get('dc:title') == 'Untitled' and ('dc:title' not in meta_original):
# Ghostscript likes to set title to Untitled if omitted from input.
# Reverse this, because PDF/A TechNote 0003:Metadata in PDF/A-1
# and the XMP Spec do not make this recommendation.
if 'dc:title' not in meta_original:
del meta_pdf['dc:title']
del meta_pdf['dc:title']
def _unset_empty_metadata(meta: PdfMetadata, options):
@@ -187,7 +185,7 @@ def metadata_fixup(
output_file = context.get_path('metafix.pdf')
options = context.options
pbar_class = context.plugin_manager.hook.get_progressbar_class()
pbar_class = context.plugin_manager.get_progressbar_class()
with (
Pdf.open(context.origin) as original,
Pdf.open(working_file) as pdf,
src/ocrmypdf/_options.py
+613
View File
@@ -0,0 +1,613 @@
# SPDX-FileCopyrightText: 2024 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Internal options model for OCRmyPDF."""
from __future__ import annotations
import json
import logging
import os
import unicodedata
from collections.abc import Sequence
from enum import StrEnum
from io import IOBase
from pathlib import Path
from typing import Any, BinaryIO
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
from ocrmypdf._defaults import DEFAULT_LANGUAGE, DEFAULT_ROTATE_PAGES_THRESHOLD
from ocrmypdf.exceptions import BadArgsError
from ocrmypdf.helpers import monotonic
# Import plugin option models - these will be available after plugins are loaded
# We'll use forward references and handle imports dynamically
log = logging.getLogger(__name__)
# Module-level registry for plugin option models
# This is populated by setup_plugin_infrastructure() after plugins are loaded
_plugin_option_models: dict[str, type] = {}
PathOrIO = BinaryIO | IOBase | Path | str | bytes
class ProcessingMode(StrEnum):
"""OCR processing mode for handling pages with existing text.
This enum controls how OCRmyPDF handles pages that already contain text:
- ``default``: Error if text is found (standard OCR behavior)
- ``force``: Rasterize all content and run OCR regardless of existing text
- ``skip``: Skip OCR on pages that already have text
- ``redo``: Re-OCR pages, stripping old invisible text layer
"""
default = 'default'
force = 'force'
skip = 'skip'
redo = 'redo'
def _pages_from_ranges(ranges: str) -> set[int]:
"""Convert page range string to set of page numbers."""
pages: list[int] = []
page_groups = ranges.replace(' ', '').split(',')
for group in page_groups:
if not group:
continue
try:
start, end = group.split('-')
except ValueError:
pages.append(int(group) - 1)
else:
try:
new_pages = list(range(int(start) - 1, int(end)))
if not new_pages:
raise BadArgsError(
f"invalid page subrange '{start}-{end}'"
) from None
pages.extend(new_pages)
except ValueError:
raise BadArgsError(f"invalid page subrange '{group}'") from None
if not pages:
raise BadArgsError(
f"The string of page ranges '{ranges}' did not contain any recognizable "
f"page ranges."
)
if not monotonic(pages):
log.warning(
"List of pages to process contains duplicate pages, or pages that are "
"out of order"
)
if any(page < 0 for page in pages):
raise BadArgsError("pages refers to a page number less than 1")
log.debug("OCRing only these pages: %s", pages)
return set(pages)
class OcrOptions(BaseModel):
"""Internal options model that can masquerade as argparse.Namespace.
This model provides proper typing and validation while maintaining
compatibility with existing code that expects argparse.Namespace behavior.
"""
# I/O options
input_file: PathOrIO
output_file: PathOrIO
sidecar: PathOrIO | None = None
output_folder: Path | None = None
work_folder: Path | None = None
# Core OCR options
languages: list[str] = Field(default_factory=lambda: [DEFAULT_LANGUAGE])
output_type: str = 'auto'
mode: ProcessingMode = ProcessingMode.default
# Backward compatibility properties for force_ocr, skip_text, redo_ocr
@property
def force_ocr(self) -> bool:
"""Backward compatibility alias for mode == ProcessingMode.force."""
return self.mode == ProcessingMode.force
@property
def skip_text(self) -> bool:
"""Backward compatibility alias for mode == ProcessingMode.skip."""
return self.mode == ProcessingMode.skip
@property
def redo_ocr(self) -> bool:
"""Backward compatibility alias for mode == ProcessingMode.redo."""
return self.mode == ProcessingMode.redo
# Job control
jobs: int | None = None
use_threads: bool = True
progress_bar: bool = True
quiet: bool = False
verbose: int = 0
keep_temporary_files: bool = False
# Image processing
image_dpi: int | None = None
deskew: bool = False
clean: bool = False
clean_final: bool = False
rotate_pages: bool = False
remove_background: bool = False
remove_vectors: bool = False
oversample: int = 0
unpaper_args: str | list[str] | None = (
None # Can be string or list after validation
)
# OCR behavior
skip_big: float | None = None
pages: str | set[int] | None = None # Can be string or set after validation
invalidate_digital_signatures: bool = False
# Metadata
title: str | None = None
author: str | None = None
subject: str | None = None
keywords: str | None = None
# Optimization
optimize: int = 1
jpg_quality: int | None = None
png_quality: int | None = None
jbig2_threshold: float = 0.85
# Compatibility alias for plugins that expect jpeg_quality
@property
def jpeg_quality(self):
"""Compatibility alias for jpg_quality."""
return self.jpg_quality
@jpeg_quality.setter
def jpeg_quality(self, value):
"""Compatibility alias for jpg_quality."""
self.jpg_quality = value
# Advanced options
max_image_mpixels: float = 250.0
pdf_renderer: str = 'auto'
ocr_engine: str = 'auto'
rasterizer: str = 'auto'
rotate_pages_threshold: float = DEFAULT_ROTATE_PAGES_THRESHOLD
user_words: os.PathLike | None = None
user_patterns: os.PathLike | None = None
fast_web_view: float = 1.0
continue_on_soft_render_error: bool | None = None
# Tesseract options - also accessible via options.tesseract.<field>
tesseract_config: list[str] = []
tesseract_pagesegmode: int | None = None
tesseract_oem: int | None = None
tesseract_thresholding: int | None = None
tesseract_timeout: float = 0.0
tesseract_non_ocr_timeout: float | None = None
tesseract_downsample_above: int = 32767
tesseract_downsample_large_images: bool | None = None
# Ghostscript options - also accessible via options.ghostscript.<field>
pdfa_image_compression: str | None = None
color_conversion_strategy: str = "LeaveColorUnchanged"
# Optimize/JBIG2 options - also accessible via options.optimize.<field>
jbig2_threshold: float = 0.85
# Plugin system
plugins: Sequence[Path | str] | None = None
# Store any extra attributes (for plugins and dynamic options)
extra_attrs: dict[str, Any] = Field(
default_factory=dict, exclude=True, alias='_extra_attrs'
)
@field_validator('languages')
@classmethod
def validate_languages(cls, v):
"""Ensure languages list is not empty."""
if not v:
return [DEFAULT_LANGUAGE]
return v
@field_validator('output_type')
@classmethod
def validate_output_type(cls, v):
"""Validate output type is one of the allowed values."""
valid_types = {'auto', 'pdfa', 'pdf', 'pdfa-1', 'pdfa-2', 'pdfa-3', 'none'}
if v not in valid_types:
raise ValueError(f"output_type must be one of {valid_types}")
return v
@field_validator('pdf_renderer')
@classmethod
def validate_pdf_renderer(cls, v):
"""Validate PDF renderer is one of the allowed values."""
valid_renderers = {'auto', 'sandwich', 'fpdf2'}
# Legacy hocr/hocrdebug are accepted but redirected to fpdf2
legacy_renderers = {'hocr', 'hocrdebug'}
all_accepted = valid_renderers | legacy_renderers
if v not in all_accepted:
raise ValueError(f"pdf_renderer must be one of {all_accepted}")
return v
@field_validator('rasterizer')
@classmethod
def validate_rasterizer(cls, v):
"""Validate rasterizer is one of the allowed values."""
valid_rasterizers = {'auto', 'ghostscript', 'pypdfium'}
if v not in valid_rasterizers:
raise ValueError(f"rasterizer must be one of {valid_rasterizers}")
return v
@field_validator('clean_final')
@classmethod
def validate_clean_final(cls, v, info):
"""If clean_final is True, also set clean to True."""
if v and hasattr(info, 'data') and 'clean' in info.data:
info.data['clean'] = True
return v
@field_validator('jobs')
@classmethod
def validate_jobs(cls, v):
"""Validate jobs is a reasonable number."""
if v is not None and (v < 0 or v > 256):
raise ValueError("jobs must be between 0 and 256")
return v
@field_validator('verbose')
@classmethod
def validate_verbose(cls, v):
"""Validate verbose level."""
if v < 0 or v > 2:
raise ValueError("verbose must be between 0 and 2")
return v
@field_validator('oversample')
@classmethod
def validate_oversample(cls, v):
"""Validate oversample DPI."""
if v < 0 or v > 5000:
raise ValueError("oversample must be between 0 and 5000")
return v
@field_validator('max_image_mpixels')
@classmethod
def validate_max_image_mpixels(cls, v):
"""Validate max image megapixels."""
if v < 0:
raise ValueError("max_image_mpixels must be non-negative")
return v
@field_validator('rotate_pages_threshold')
@classmethod
def validate_rotate_pages_threshold(cls, v):
"""Validate rotate pages threshold."""
if v < 0 or v > 1000:
raise ValueError("rotate_pages_threshold must be between 0 and 1000")
return v
@field_validator('title', 'author', 'keywords', 'subject')
@classmethod
def validate_metadata_unicode(cls, v):
"""Validate metadata strings don't contain unsupported Unicode characters."""
if v is None:
return v
for char in v:
if unicodedata.category(char) == 'Co' or ord(char) >= 0x10000:
hexchar = hex(ord(char))[2:].upper()
raise ValueError(
f"Metadata string contains unsupported Unicode character: "
f"{char} (U+{hexchar})"
)
return v
@field_validator('pages')
@classmethod
def validate_pages_format(cls, v):
"""Convert page ranges string to set of page numbers."""
if v is None:
return v
if isinstance(v, set):
return v # Already processed
# Convert string ranges to set of page numbers
return _pages_from_ranges(v)
@model_validator(mode='before')
@classmethod
def handle_special_cases(cls, data):
"""Handle special cases for API compatibility and legacy options."""
if isinstance(data, dict):
# For hOCR API, output_file might not be present
if 'output_folder' in data and 'output_file' not in data:
data['output_file'] = '/dev/null' # Placeholder
# Convert legacy boolean options (force_ocr, skip_text, redo_ocr) to mode
force = data.pop('force_ocr', None)
skip = data.pop('skip_text', None)
redo = data.pop('redo_ocr', None)
# Count how many legacy options are set to True
legacy_set = [
(force, ProcessingMode.force),
(skip, ProcessingMode.skip),
(redo, ProcessingMode.redo),
]
legacy_true = [(val, mode) for val, mode in legacy_set if val]
legacy_count = len(legacy_true)
# Get current mode value (may be string or enum)
current_mode = data.get('mode', ProcessingMode.default)
if isinstance(current_mode, str):
current_mode = ProcessingMode(current_mode)
mode_is_set = current_mode != ProcessingMode.default
if legacy_count > 1:
raise ValueError(
"Choose only one of --force-ocr, --skip-text, --redo-ocr."
)
if legacy_count == 1:
expected_mode = legacy_true[0][1]
if mode_is_set and current_mode != expected_mode:
legacy_flag = f"--{expected_mode.value.replace('_', '-')}-ocr"
raise ValueError(
f"Conflicting options: --mode {current_mode.value} "
f"cannot be used with {legacy_flag} or similar legacy flag."
)
# Set mode from legacy option
data['mode'] = expected_mode
return data
@model_validator(mode='after')
def validate_redo_ocr_options(self):
"""Validate options compatible with redo mode."""
if self.mode == ProcessingMode.redo and (
self.deskew or self.clean_final or self.remove_background
):
raise ValueError(
"--redo-ocr (or --mode redo) is not currently compatible with "
"--deskew, --clean-final, and --remove-background"
)
return self
@model_validator(mode='after')
def validate_output_type_compatibility(self):
"""Validate output type is compatible with output file."""
if self.output_type == 'none' and str(self.output_file) not in (
os.devnull,
'-',
):
raise ValueError(
"Since you specified `--output-type none`, the output file "
f"{self.output_file} cannot be produced. Set the output file to "
f"`-` to suppress this message."
)
return self
@property
def lossless_reconstruction(self):
"""Determine lossless_reconstruction based on other options."""
lossless = not any(
[
self.deskew,
self.clean_final,
self.mode == ProcessingMode.force,
self.remove_background,
]
)
return lossless
def model_dump_json_safe(self) -> str:
"""Serialize to JSON with special handling for non-serializable types."""
# Create a copy of the model data for serialization
data = self.model_dump()
# Handle special types that don't serialize to JSON directly
def _serialize_value(value):
if isinstance(value, Path):
return {'__type__': 'Path', 'value': str(value)}
elif (
isinstance(value, BinaryIO | IOBase)
or hasattr(value, 'read')
or hasattr(value, 'write')
):
# Stream object - replace with placeholder
return {'__type__': 'Stream', 'value': 'stream'}
elif hasattr(value, '__class__') and 'Iterator' in value.__class__.__name__:
# Handle Pydantic serialization iterators
return {'__type__': 'Stream', 'value': 'stream'}
elif isinstance(value, property):
# Handle property objects that shouldn't be serialized
return None
elif isinstance(value, list | tuple):
return [_serialize_value(item) for item in value]
elif isinstance(value, dict):
return {k: _serialize_value(v) for k, v in value.items()}
else:
return value
# Process all fields
serializable_data = {}
for key, value in data.items():
serialized_value = _serialize_value(value)
if serialized_value is not None: # Skip None values from properties
serializable_data[key] = serialized_value
# Add extra_attrs, excluding plugin cache entries (they'll be recreated lazily)
if self.extra_attrs:
filtered_extra = {
k: v
for k, v in self.extra_attrs.items()
if not k.startswith('_plugin_cache_')
}
if filtered_extra:
serializable_data['_extra_attrs'] = _serialize_value(filtered_extra)
return json.dumps(serializable_data)
@classmethod
def model_validate_json_safe(cls, json_str: str) -> OcrOptions:
"""Reconstruct from JSON with special handling for non-serializable types."""
data = json.loads(json_str)
# Handle special types during deserialization
def _deserialize_value(value):
if isinstance(value, dict) and '__type__' in value:
if value['__type__'] == 'Path':
return Path(value['value'])
elif value['__type__'] == 'Stream':
# For streams, we'll use a placeholder string
return value['value']
else:
return value['value']
elif isinstance(value, list):
return [_deserialize_value(item) for item in value]
elif isinstance(value, dict):
return {k: _deserialize_value(v) for k, v in value.items()}
else:
return value
# Process all fields
deserialized_data = {}
extra_attrs = {}
for key, value in data.items():
if key == '_extra_attrs':
extra_attrs = _deserialize_value(value)
else:
deserialized_data[key] = _deserialize_value(value)
# Create instance
instance = cls(**deserialized_data)
instance.extra_attrs = extra_attrs
return instance
model_config = ConfigDict(
extra="forbid", # Force use of extra_attrs for unknown fields
arbitrary_types_allowed=True, # Allow BinaryIO, Path, etc.
validate_assignment=True, # Validate on attribute assignment
)
@classmethod
def register_plugin_models(cls, models: dict[str, type]) -> None:
"""Register plugin option model classes for nested access.
Args:
models: Dictionary mapping namespace to model class
"""
global _plugin_option_models
_plugin_option_models.update(models)
def _get_plugin_options(self, namespace: str) -> Any:
"""Get or create a plugin options instance for the given namespace.
This method creates plugin option instances lazily from flat field values.
Args:
namespace: The plugin namespace (e.g., 'tesseract', 'optimize')
Returns:
An instance of the plugin's option model, or None if not registered
"""
# Use extra_attrs to cache plugin option instances
cache_key = f'_plugin_cache_{namespace}'
if cache_key in self.extra_attrs:
return self.extra_attrs[cache_key]
if namespace not in _plugin_option_models:
raise AttributeError(
f"Plugin namespace '{namespace}' is not registered. "
f"Ensure setup_plugin_infrastructure() was called."
)
model_class = _plugin_option_models[namespace]
def _convert_value(value):
"""Convert value to be compatible with plugin model fields."""
if isinstance(value, os.PathLike):
return os.fspath(value)
return value
# Build kwargs from flat fields
kwargs = {}
for field_name in model_class.model_fields:
# Try namespace_field pattern first (e.g., tesseract_timeout)
flat_name = f"{namespace}_{field_name}"
if flat_name in OcrOptions.model_fields:
value = getattr(self, flat_name)
if value is not None:
kwargs[field_name] = _convert_value(value)
# Also check direct field name (for fields like jbig2_lossy)
elif field_name in OcrOptions.model_fields:
value = getattr(self, field_name)
if value is not None:
kwargs[field_name] = _convert_value(value)
# Check for special mappings
elif namespace == 'optimize' and field_name == 'level':
# 'optimize' field maps to 'level' in OptimizeOptions
if 'optimize' in OcrOptions.model_fields:
value = self.optimize
if value is not None:
kwargs[field_name] = _convert_value(value)
elif namespace == 'optimize' and field_name == 'jpeg_quality':
# jpg_quality maps to jpeg_quality
if 'jpg_quality' in OcrOptions.model_fields:
value = self.jpg_quality
if value is not None:
kwargs[field_name] = _convert_value(value)
# Create and cache the plugin options instance
instance = model_class(**kwargs)
self.extra_attrs[cache_key] = instance
return instance
def __getattr__(self, name: str) -> Any:
"""Support dynamic access to plugin option namespaces.
This allows accessing plugin options like:
options.tesseract.timeout
options.optimize.level
Plugin models must be registered via register_plugin_models() for
namespace access to work. Built-in plugins register their models
during initialization.
Args:
name: Attribute name
Returns:
Plugin options instance if name is a registered namespace,
otherwise raises AttributeError
"""
# Check if this is a plugin namespace
if name.startswith('_'):
# Private attributes should not trigger plugin lookup
raise AttributeError(
f"'{type(self).__name__}' object has no attribute '{name}'"
)
# Try to get plugin options for this namespace
if name in _plugin_option_models:
return self._get_plugin_options(name)
# Check extra_attrs
if 'extra_attrs' in self.__dict__ and name in self.extra_attrs:
return self.extra_attrs[name]
raise AttributeError(
f"'{type(self).__name__}' object has no attribute '{name}'"
)
src/ocrmypdf/_pipeline.py
+237 -98
View File
@@ -15,7 +15,10 @@ from contextlib import suppress
from io import BytesIO
from pathlib import Path
from shutil import copyfileobj
from typing import Any, BinaryIO, TypeVar, cast
from typing import TYPE_CHECKING, Any, BinaryIO, TypeVar, cast
if TYPE_CHECKING:
from ocrmypdf.hocrtransform import OcrElement
import img2pdf
import pikepdf
@@ -25,6 +28,7 @@ from ocrmypdf._concurrent import Executor
from ocrmypdf._exec import unpaper
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._metadata import repair_docinfo_nuls
from ocrmypdf._options import OcrOptions, ProcessingMode
from ocrmypdf.exceptions import (
DigitalSignatureError,
DpiError,
@@ -35,12 +39,13 @@ from ocrmypdf.exceptions import (
UnsupportedImageFormatError,
)
from ocrmypdf.helpers import IMG2PDF_KWARGS, Resolution, safe_symlink
from ocrmypdf.hocrtransform import DebugRenderOptions, HocrTransform
from ocrmypdf.hocrtransform._font import Courier
from ocrmypdf.pdfa import generate_pdfa_ps
from ocrmypdf.pdfinfo import Colorspace, Encoding, PageInfo, PdfInfo
from ocrmypdf.pdfinfo.info import FloatRect
from ocrmypdf.pluginspec import OrientationConfidence
from ocrmypdf.pdfa import (
file_claims_pdfa,
generate_pdfa_ps,
speculative_pdfa_conversion,
)
from ocrmypdf.pdfinfo import Colorspace, Encoding, FloatRect, PageInfo, PdfInfo
from ocrmypdf.pluginspec import GhostscriptRasterDevice, OrientationConfidence
try:
from pi_heif import register_heif_opener
@@ -59,7 +64,7 @@ VECTOR_PAGE_DPI = 400
register_heif_opener()
def triage_image_file(input_file: Path, output_file: Path, options) -> None:
def triage_image_file(input_file: Path, output_file: Path, options: OcrOptions) -> None:
"""Triage the input image file.
If the input file is an image, check its resolution and convert it to PDF.
@@ -158,7 +163,7 @@ def _pdf_guess_version(input_file: Path, search_window=1024) -> str:
def triage(
original_filename: str, input_file: Path, output_file: Path, options
original_filename: str, input_file: Path, output_file: Path, options: OcrOptions
) -> Path:
"""Triage the input file. We can handle PDFs and images."""
try:
@@ -228,10 +233,10 @@ def validate_pdfinfo_options(context: PdfContext) -> None:
else:
raise DigitalSignatureError()
if pdfinfo.has_acroform:
if options.redo_ocr:
if options.mode == ProcessingMode.redo:
raise InputFileError(
"This PDF has a user fillable form. --redo-ocr is not "
"currently possible on such files."
"This PDF has a user fillable form. --redo-ocr (or --mode redo) "
"is not currently possible on such files."
)
else:
log.warning(
@@ -239,14 +244,14 @@ def validate_pdfinfo_options(context: PdfContext) -> None:
"Chances are it is a pure digital "
"document that does not need OCR."
)
if not options.force_ocr:
if options.mode != ProcessingMode.force:
log.info(
"Use the option --force-ocr to produce an image of the "
"form and all filled form fields. The output PDF will be "
"'flattened' and will no longer be fillable."
"Use the option --force-ocr (or --mode force) to produce an "
"image of the form and all filled form fields. The output PDF "
"will be 'flattened' and will no longer be fillable."
)
if pdfinfo.is_tagged:
if options.force_ocr or options.skip_text or options.redo_ocr:
if options.mode != ProcessingMode.default:
log.warning(
"This PDF is marked as a Tagged PDF. This often indicates "
"that the PDF was generated from an office document and does "
@@ -255,7 +260,7 @@ def validate_pdfinfo_options(context: PdfContext) -> None:
)
else:
raise TaggedPDFError()
context.plugin_manager.hook.validate(pdfinfo=pdfinfo, options=options)
context.plugin_manager.validate(pdfinfo=pdfinfo, options=options)
def _vector_page_dpi(pageinfo: PageInfo) -> int:
@@ -323,24 +328,24 @@ def is_ocr_required(page_context: PageContext) -> bool:
log.debug(f"skipped {pageinfo.pageno} as requested by --pages {options.pages}")
ocr_required = False
elif pageinfo.has_text:
if not options.force_ocr and not (options.skip_text or options.redo_ocr):
if options.mode == ProcessingMode.default:
raise PriorOcrFoundError(
"page already has text! - aborting (use --force-ocr to force OCR; "
" see also help for the arguments --skip-text and --redo-ocr"
"page already has text! - aborting (use --force-ocr or --mode force "
"to force OCR; see also help for --skip-text, --redo-ocr, and --mode)"
)
elif options.force_ocr:
elif options.mode == ProcessingMode.force:
log.info("page already has text! - rasterizing text and running OCR anyway")
ocr_required = True
elif options.redo_ocr:
elif options.mode == ProcessingMode.redo:
if pageinfo.has_corrupt_text:
log.warning(
"some text on this page cannot be mapped to characters: "
"consider using --force-ocr instead"
"consider using --force-ocr (or --mode force) instead"
)
else:
log.info("redoing OCR")
ocr_required = True
elif options.skip_text:
elif options.mode == ProcessingMode.skip:
log.info("skipping all processing on this page")
ocr_required = False
elif not pageinfo.images and not options.lossless_reconstruction:
@@ -351,14 +356,14 @@ def is_ocr_required(page_context: PageContext) -> bool:
# ahead and rasterize. If not forced, then pretend there's no text
# on the page at all so we don't lose anything.
# This could be made smarter by explicitly searching for vector art.
if options.force_ocr and options.oversample:
if options.mode == ProcessingMode.force and options.oversample:
# The user really wants to reprocess this file
log.info(
"page has no images - "
f"rasterizing at {options.oversample} DPI because "
"--force-ocr --oversample was specified"
"--force-ocr --oversample (or --mode force --oversample) was specified"
)
elif options.force_ocr:
elif options.mode == ProcessingMode.force:
# Warn the user they might not want to do this
log.warning(
"page has no images - "
@@ -371,8 +376,8 @@ def is_ocr_required(page_context: PageContext) -> bool:
log.info(
"page has no images - "
"skipping all processing on this page to avoid losing detail. "
"Use --force-ocr if you wish to perform OCR on pages that "
"have vector content."
"Use --force-ocr (or --mode force) if you wish to perform OCR on "
"pages that have vector content."
)
ocr_required = False
@@ -395,16 +400,18 @@ def rasterize_preview(input_file: Path, page_context: PageContext) -> Path:
[get_canvas_square_dpi(page_context)]
)
page_dpi = Resolution(300.0, 300.0).take_min([get_page_square_dpi(page_context)])
page_context.plugin_manager.hook.rasterize_pdf_page(
page_context.plugin_manager.rasterize_pdf_page(
input_file=input_file,
output_file=output_file,
raster_device='jpeggray',
raster_device=GhostscriptRasterDevice.JPEGGRAY,
raster_dpi=canvas_dpi,
pageno=page_context.pageinfo.pageno + 1,
page_dpi=page_dpi,
rotation=0,
filter_vector=False,
stop_on_soft_error=not page_context.options.continue_on_soft_render_error,
options=page_context.options,
use_cropbox=False,
)
return output_file
@@ -424,10 +431,7 @@ def describe_rotation(
else:
action = 'rotation appears correct'
else:
if correction != 0:
action = 'confidence too low to rotate'
else:
action = 'no change'
action = "confidence too low to rotate" if correction != 0 else "no change"
facing = ''
@@ -453,9 +457,10 @@ def get_orientation_correction(preview: Path, page_context: PageContext) -> int:
which points it (hopefully) upright. _graft.py takes care of the orienting
the image and text layers.
"""
orient_conf = page_context.plugin_manager.hook.get_ocr_engine().get_orientation(
preview, page_context.options
ocr_engine = page_context.plugin_manager.get_ocr_engine(
options=page_context.options
)
orient_conf = ocr_engine.get_orientation(preview, page_context.options)
correction = orient_conf.angle % 360
log.info(describe_rotation(page_context, orient_conf, correction))
@@ -521,7 +526,12 @@ def rasterize(
Returns:
Path: The output PNG file path.
"""
colorspaces = ['pngmono', 'pnggray', 'png256', 'png16m']
colorspaces = [
GhostscriptRasterDevice.PNGMONO,
GhostscriptRasterDevice.PNGGRAY,
GhostscriptRasterDevice.PNG256,
GhostscriptRasterDevice.PNG16M,
]
device_idx = 0
if remove_vectors is None:
@@ -538,15 +548,15 @@ def rasterize(
continue # ignore masks
if image.bpc > 1:
if image.color == Colorspace.index:
device_idx = at_least('png256')
device_idx = at_least(GhostscriptRasterDevice.PNG256)
elif image.color == Colorspace.gray:
device_idx = at_least('pnggray')
device_idx = at_least(GhostscriptRasterDevice.PNGGRAY)
else:
device_idx = at_least('png16m')
device_idx = at_least(GhostscriptRasterDevice.PNG16M)
if pageinfo.has_vector:
log.debug("Page has vector content, using png16m")
device_idx = at_least('png16m')
log.debug(f"Page has vector content, using {GhostscriptRasterDevice.PNG16M}")
device_idx = at_least(GhostscriptRasterDevice.PNG16M)
device = colorspaces[device_idx]
@@ -556,7 +566,7 @@ def rasterize(
canvas_dpi, page_dpi = calculate_raster_dpi(page_context)
page_context.plugin_manager.hook.rasterize_pdf_page(
page_context.plugin_manager.rasterize_pdf_page(
input_file=input_file,
output_file=output_file,
raster_device=device,
@@ -566,6 +576,8 @@ def rasterize(
rotation=correction,
filter_vector=remove_vectors,
stop_on_soft_error=not page_context.options.continue_on_soft_render_error,
options=page_context.options,
use_cropbox=False,
)
return output_file
@@ -594,7 +606,9 @@ def preprocess_deskew(input_file: Path, page_context: PageContext) -> Path:
output_file = page_context.get_path('pp_deskew.png')
dpi = get_page_square_dpi(page_context, calculate_image_dpi(page_context))
ocr_engine = page_context.plugin_manager.hook.get_ocr_engine()
ocr_engine = page_context.plugin_manager.get_ocr_engine(
options=page_context.options
)
deskew_angle_degrees = ocr_engine.get_deskew(input_file, page_context.options)
with Image.open(input_file) as im:
@@ -633,11 +647,11 @@ def create_ocr_image(image: Path, page_context: PageContext) -> Path:
with Image.open(image) as im:
log.debug('resolution %r', im.info['dpi'])
if not options.force_ocr:
if options.mode != ProcessingMode.force:
# Do not mask text areas when forcing OCR, because we need to OCR
# all text areas
mask = None # Exclude both visible and invisible text from OCR
if options.redo_ocr:
if options.mode == ProcessingMode.redo:
mask = True # Mask visible text, but not invisible text
draw = ImageDraw.ImageDraw(im)
@@ -659,7 +673,7 @@ def create_ocr_image(image: Path, page_context: PageContext) -> Path:
draw.rectangle(pixcoords, fill='white')
# draw.rectangle(pixcoords, outline='pink')
filter_im = page_context.plugin_manager.hook.filter_ocr_image(
filter_im = page_context.plugin_manager.filter_ocr_image(
page=page_context, image=im
)
if filter_im is not None:
@@ -677,7 +691,7 @@ def ocr_engine_hocr(input_file: Path, page_context: PageContext) -> tuple[Path,
hocr_text_out = page_context.get_path('ocr_hocr.txt')
options = page_context.options
ocr_engine = page_context.plugin_manager.hook.get_ocr_engine()
ocr_engine = page_context.plugin_manager.get_ocr_engine(options=options)
ocr_engine.generate_hocr(
input_file=input_file,
output_hocr=hocr_out,
@@ -687,6 +701,37 @@ def ocr_engine_hocr(input_file: Path, page_context: PageContext) -> tuple[Path,
return hocr_out, hocr_text_out
def ocr_engine_direct(
input_file: Path, page_context: PageContext
) -> tuple[OcrElement, Path]:
"""Run the OCR engine and return OcrElement tree directly.
This is the modern path for OCR engines that support the generate_ocr() API.
It bypasses hOCR file generation for better performance and richer data.
Args:
input_file: The image file to OCR.
page_context: The page context with options and path utilities.
Returns:
A tuple of (OcrElement tree, path to text sidecar file).
"""
text_out = page_context.get_path('ocr_direct.txt')
options = page_context.options
ocr_engine = page_context.plugin_manager.get_ocr_engine(options=options)
ocr_tree, text_content = ocr_engine.generate_ocr(
input_file=input_file,
options=options,
page_number=page_context.pageno,
)
# Write text sidecar file
text_out.write_text(text_content, encoding='utf-8')
return ocr_tree, text_out
def should_visible_page_image_use_jpg(pageinfo: PageInfo) -> bool:
"""Determines whether the visible page image should be saved as a JPEG.
@@ -764,47 +809,12 @@ def create_pdf_page_from_image(
bio.seek(0)
fix_pagepdf_boxes(bio, output_file, page_context, swap_axis=swap_axis)
output_file = page_context.plugin_manager.hook.filter_pdf_page(
output_file = page_context.plugin_manager.filter_pdf_page(
page=page_context, image_filename=image, output_pdf=output_file
)
return output_file
def render_hocr_page(hocr: Path, page_context: PageContext) -> Path:
"""Render the hOCR page to a PDF."""
options = page_context.options
output_file = page_context.get_path('ocr_hocr.pdf')
if hocr.stat().st_size == 0:
# If hOCR file is empty (skipped page marker), create an empty PDF file
output_file.touch()
return output_file
dpi = get_page_square_dpi(page_context, calculate_image_dpi(page_context))
debug_kwargs = {}
if options.pdf_renderer == 'hocrdebug':
debug_kwargs = dict(
debug_render_options=DebugRenderOptions(
render_baseline=True,
render_triangle=True,
render_line_bbox=False,
render_word_bbox=True,
render_paragraph_bbox=False,
render_space_bbox=False,
),
font=Courier(),
)
HocrTransform(
hocr_filename=hocr,
dpi=dpi.to_scalar(),
**debug_kwargs, # square
).to_pdf(
out_filename=output_file,
image_filename=None,
invisible_text=True if not debug_kwargs else False,
)
return output_file
def ocr_engine_textonly_pdf(
input_image: Path, page_context: PageContext
) -> tuple[Path, Path]:
@@ -813,7 +823,7 @@ def ocr_engine_textonly_pdf(
output_text = page_context.get_path('ocr_tess.txt')
options = page_context.options
ocr_engine = page_context.plugin_manager.hook.get_ocr_engine()
ocr_engine = page_context.plugin_manager.get_ocr_engine(options=options)
ocr_engine.generate_pdf(
input_file=input_image,
output_pdf=output_pdf,
@@ -936,15 +946,26 @@ def convert_to_pdfa(input_pdf: Path, input_ps_stub: Path, context: PdfContext) -
else:
safe_symlink(input_pdf, fix_docinfo_file)
context.plugin_manager.hook.generate_pdfa(
# Extract PDF/A part correctly
if options.output_type.startswith('pdfa'):
if options.output_type == 'pdfa':
pdfa_part = '2' # Default to PDF/A-2
else:
pdfa_part = options.output_type.split('-')[
-1
] # Extract number from pdfa-1, pdfa-2, etc.
else:
pdfa_part = '2' # Fallback
context.plugin_manager.generate_pdfa(
pdf_version=input_pdfinfo.min_version,
pdf_pages=[fix_docinfo_file],
pdfmark=input_ps_stub,
output_file=output_file,
context=context,
pdfa_part=options.output_type[-1], # is pdfa-1, pdfa-2, or pdfa-3
pdfa_part=pdfa_part,
progressbar_class=(
context.plugin_manager.hook.get_progressbar_class()
context.plugin_manager.get_progressbar_class()
if options.progress_bar
else None
),
@@ -954,15 +975,136 @@ def convert_to_pdfa(input_pdf: Path, input_ps_stub: Path, context: PdfContext) -
return output_file
def try_speculative_pdfa(input_pdf: Path, context: PdfContext) -> Path | None:
"""Try speculative PDF/A conversion with verapdf validation.
This attempts a fast PDF/A conversion by adding PDF/A structures
directly with pikepdf, then validating with verapdf. If validation
passes, returns the converted file. If it fails or verapdf is not
available, returns None to signal that Ghostscript should be used.
Args:
input_pdf: Path to the PDF to convert
context: The PDF context
Returns:
Path to valid PDF/A file, or None if speculative conversion failed
"""
from ocrmypdf._exec import verapdf
options = context.options
# Skip speculative conversion if user requested specific image compression,
# since that requires Ghostscript to apply
gs_opts = getattr(options, 'ghostscript', None)
if gs_opts is not None:
compression = getattr(gs_opts, 'pdfa_image_compression', 'auto')
if compression != 'auto':
log.debug(
'Skipping speculative PDF/A: --pdfa-image-compression=%s requires '
'Ghostscript',
compression,
)
return None
if not verapdf.available():
log.debug('verapdf not available, skipping speculative PDF/A conversion')
return None
output_file = context.get_path('speculative_pdfa.pdf')
try:
speculative_pdfa_conversion(input_pdf, output_file, options.output_type)
flavour = verapdf.output_type_to_flavour(options.output_type)
result = verapdf.validate(output_file, flavour)
if result.valid:
log.info('Speculative PDF/A conversion succeeded - skipping Ghostscript')
return output_file
else:
log.debug(
'Speculative PDF/A validation failed (%d rule violations), '
'falling back to Ghostscript',
result.failed_rules,
)
return None
except Exception as e:
log.debug('Speculative PDF/A conversion failed: %s', e)
return None
def try_auto_pdfa(input_pdf: Path, context: PdfContext) -> tuple[Path, str]:
"""Best-effort PDF/A for 'auto' output type.
This function attempts to produce PDF/A without requiring Ghostscript:
1. If verapdf is available, tries speculative conversion with validation
2. Without verapdf, passes through as PDF/A if safe (input already PDF/A
or force-ocr was used)
3. Falls back to regular PDF if neither condition is met
Args:
input_pdf: Path to the PDF to convert
context: The PDF context
Returns:
Tuple of (output_path, actual_output_type) where actual_output_type
is 'pdfa' if PDF/A was achieved, 'pdf' otherwise
"""
from ocrmypdf._exec import verapdf
# If verapdf available, try speculative conversion with validation
if verapdf.available():
result = try_speculative_pdfa(input_pdf, context)
if result is not None:
return (result, 'pdfa')
# verapdf validation failed - fall through to regular PDF
log.info(
'Auto mode: speculative PDF/A validation failed, outputting regular PDF'
)
return (input_pdf, 'pdf')
# Without verapdf, check if we can pass through as PDF/A
if _is_safe_pdfa(input_pdf, context.options):
# Pass through as-is (no modifications needed)
log.info('Auto mode: passing through as PDF/A (input already compliant)')
return (input_pdf, 'pdfa')
# Fall through to regular PDF
log.info('Auto mode: no verapdf available and input is not PDF/A, outputting PDF')
return (input_pdf, 'pdf')
def _is_safe_pdfa(input_pdf: Path, options) -> bool:
"""Check if file can be considered PDF/A without validation.
These are cases where our modifications don't break PDF/A compliance:
1. Input already claims PDF/A (we just grafted OCR text onto it)
2. We used force-ocr (we rewrote the entire PDF from scratch)
Args:
input_pdf: Path to the PDF to check
options: OCR options
Returns:
True if file can safely be considered PDF/A
"""
# Safe if input already claims PDF/A
pdfa_status = file_claims_pdfa(input_pdf)
if pdfa_status['pass']:
return True
# Safe if we rewrote the PDF with force mode
return options.mode == ProcessingMode.force
def should_linearize(working_file: Path, context: PdfContext) -> bool:
"""Determine whether the PDF should be linearized.
For smaller files, linearization is not worth the effort.
"""
filesize = os.stat(working_file).st_size
if filesize > (context.options.fast_web_view * 1_000_000):
return True
return False
return filesize > (context.options.fast_web_view * 1_000_000)
def get_pdf_save_settings(output_type: str) -> dict[str, Any]:
@@ -1016,7 +1158,7 @@ def optimize_pdf(
) -> tuple[Path, Sequence[str]]:
"""Optimize the given PDF file."""
output_file = context.get_path('optimize.pdf')
output_pdf, messages = context.plugin_manager.hook.optimize_pdf(
output_pdf, messages = context.plugin_manager.optimize_pdf(
input_pdf=input_file,
output_pdf=output_file,
context=context,
@@ -1080,10 +1222,7 @@ def merge_sidecars(txt_files: Iterable[Path | None], context: PdfContext) -> Pat
# others don't. Remove it if it exists, since we add one manually.
stream.write(txt.removesuffix('\f'))
else:
if from_ != to_:
pages = f'{from_}-{to_}'
else:
pages = f'{from_}'
pages = f"{from_}-{to_}" if from_ != to_ else f"{from_}"
stream.write(f'[OCR skipped on page(s) {pages}]')
return output_file
src/ocrmypdf/_pipelines/_common.py
+78 -20
View File
@@ -3,7 +3,6 @@
from __future__ import annotations
import argparse
import json
import logging
import logging.handlers
@@ -17,9 +16,13 @@ from concurrent.futures.thread import BrokenThreadPool
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import NamedTuple, cast
from typing import TYPE_CHECKING, NamedTuple, cast
if TYPE_CHECKING:
from ocrmypdf.hocrtransform import OcrElement
import PIL
import PIL.Image
from pikepdf import Pdf
from ocrmypdf._annots import remove_broken_goto_annotations
@@ -27,6 +30,7 @@ from ocrmypdf._concurrent import Executor, setup_executor
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._logging import PageNumberFilter
from ocrmypdf._metadata import metadata_fixup
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipeline import (
convert_to_pdfa,
create_ocr_image,
@@ -44,6 +48,8 @@ from ocrmypdf._pipeline import (
rasterize_preview,
should_linearize,
should_visible_page_image_use_jpg,
try_auto_pdfa,
try_speculative_pdfa,
)
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
from ocrmypdf._validation import (
@@ -51,7 +57,6 @@ from ocrmypdf._validation import (
)
from ocrmypdf.exceptions import ExitCode, ExitCodeException
from ocrmypdf.helpers import (
available_cpu_count,
check_pdf,
pikepdf_enable_mmap,
running_in_docker,
@@ -105,6 +110,9 @@ class PageResult(NamedTuple):
orientation_correction: int = 0
"""Orientation correction in degrees."""
ocr_tree: OcrElement | None = None
"""Direct OcrElement tree (when using generate_ocr() API)."""
class HOCRResultEncoder(json.JSONEncoder):
def default(self, obj):
@@ -115,7 +123,8 @@ class HOCRResultEncoder(json.JSONEncoder):
class HOCRResultDecoder(json.JSONDecoder):
def __init__(self, *args, **kwargs):
super().__init__(object_hook=self.dict_to_object, *args, **kwargs)
kwargs['object_hook'] = self.dict_to_object
super().__init__(*args, **kwargs)
def dict_to_object(self, d):
if 'Path' in d:
@@ -142,6 +151,9 @@ class HOCRResult:
orientation_correction: int = 0
"""Orientation correction in degrees."""
ocr_tree: OcrElement | None = None
"""Direct OcrElement tree (when using generate_ocr() API)."""
@classmethod
def from_json(cls, json_str: str) -> HOCRResult:
"""Create an instance from a dict."""
@@ -194,7 +206,7 @@ def worker_init(max_pixels: int | None) -> None:
@contextmanager
def manage_debug_log_handler(
*,
options: argparse.Namespace,
options: OcrOptions,
work_folder: Path,
):
remover = None
@@ -243,8 +255,8 @@ def manage_work_folder(*, work_folder: Path, retain: bool, print_location: bool)
def cli_exception_handler(
fn: Callable[[argparse.Namespace, OcrmypdfPluginManager], ExitCode],
options: argparse.Namespace,
fn: Callable[[OcrOptions, OcrmypdfPluginManager], ExitCode],
options: OcrOptions,
plugin_manager: OcrmypdfPluginManager,
) -> ExitCode:
"""Convert exceptions into command line error messages and exit codes.
@@ -274,6 +286,16 @@ def cli_exception_handler(
else:
log.error(type(e).__name__)
return e.exit_code
except ValueError as e:
# Convert Pydantic validation errors to BadArgsError for proper exit code
if "validation error" in str(e).lower() or "value error" in str(e).lower():
if options.verbose >= 1:
log.exception("Validation error")
else:
log.error("Invalid argument: %s", str(e))
return ExitCode.bad_args
# Re-raise other ValueErrors to be caught by the general exception handler
raise
except PIL.Image.DecompressionBombError:
log.exception(
"A decompression bomb error was encountered while executing the "
@@ -298,23 +320,33 @@ def cli_exception_handler(
def setup_pipeline(
options: argparse.Namespace,
options: OcrOptions,
plugin_manager: OcrmypdfPluginManager,
) -> Executor:
# Any changes to options will not take effect for options that are already
# bound to function parameters in the pipeline. (For example
# options.input_file, options.pdf_renderer are already bound.)
if not options.jobs:
options.jobs = available_cpu_count()
# Note: OcrOptions is immutable, so we can't modify options.jobs directly
# The jobs field should already be set correctly during OcrOptions creation
# Apply PIL max image pixels side effect
PIL.Image.MAX_IMAGE_PIXELS = int(options.max_image_mpixels * 1_000_000)
if PIL.Image.MAX_IMAGE_PIXELS == 0:
PIL.Image.MAX_IMAGE_PIXELS = None # type: ignore
pikepdf_enable_mmap()
executor = setup_executor(plugin_manager)
return executor
def do_get_pdfinfo(
pdf_path: Path, executor: Executor, options: argparse.Namespace
) -> PdfInfo:
def do_get_pdfinfo(pdf_path: Path, executor: Executor, options) -> PdfInfo:
# Handle pages field - it might be a string that needs conversion
check_pages = options.pages
if isinstance(check_pages, str):
from ocrmypdf._options import _pages_from_ranges
check_pages = _pages_from_ranges(check_pages)
return get_pdfinfo(
pdf_path,
executor=executor,
@@ -322,7 +354,7 @@ def do_get_pdfinfo(
progbar=options.progress_bar,
max_workers=options.jobs,
use_threads=options.use_threads,
check_pages=options.pages,
check_pages=check_pages,
)
@@ -425,7 +457,7 @@ def process_page(page_context: PageContext) -> tuple[Path, Path | None, int]:
visible_image_out = preprocess_out
if should_visible_page_image_use_jpg(page_context.pageinfo):
visible_image_out = create_visible_page_jpg(visible_image_out, page_context)
filtered_image = page_context.plugin_manager.hook.filter_page_image(
filtered_image = page_context.plugin_manager.filter_page_image(
page=page_context, image_filename=visible_image_out
)
if filtered_image is not None: # None if no hook is present
@@ -448,11 +480,22 @@ def postprocess(
pdf_out = fix_annots
else:
pdf_out = pdf_file
if context.options.output_type.startswith('pdfa'):
ps_stub_out = generate_postscript_stub(context)
pdf_out = convert_to_pdfa(pdf_out, ps_stub_out, context)
if context.options.output_type == 'auto':
# Best effort PDF/A - never uses Ghostscript
pdf_out, actual_type = try_auto_pdfa(pdf_out, context)
# Store actual output type for reporting
context.options.extra_attrs['_actual_output_type'] = actual_type
elif context.options.output_type.startswith('pdfa'):
# Required PDF/A - uses Ghostscript as fallback
speculative_result = try_speculative_pdfa(pdf_out, context)
if speculative_result is not None:
pdf_out = speculative_result
else:
# Fall back to Ghostscript conversion
ps_stub_out = generate_postscript_stub(context)
pdf_out = convert_to_pdfa(pdf_out, ps_stub_out, context)
optimizing = context.plugin_manager.hook.is_optimization_enabled(context=context)
optimizing = context.plugin_manager.is_optimization_enabled(context=context)
save_settings = get_pdf_save_settings(context.options.output_type)
save_settings['linearize'] = not optimizing and should_linearize(pdf_out, context)
@@ -468,7 +511,22 @@ def report_output_pdf(options, start_input_file, optimize_messages) -> ExitCode:
elif samefile(options.output_file, Path(os.devnull)):
pass # Say nothing when sending to dev null
else:
if options.output_type.startswith('pdfa'):
if options.output_type == 'auto':
# For 'auto' mode, check what we actually produced
actual_type = options.extra_attrs.get('_actual_output_type', 'pdf')
pdfa_info = file_claims_pdfa(options.output_file)
if actual_type == 'pdfa' and pdfa_info['pass']:
log.info(
"Output file is a %s (auto mode achieved PDF/A)",
pdfa_info['conformance'],
)
elif pdfa_info['pass']:
# Unexpectedly got PDF/A
log.info("Output file is a %s", pdfa_info['conformance'])
else:
# Regular PDF - this is expected for auto mode fallback
log.info("Output file is a PDF (auto mode)")
elif options.output_type.startswith('pdfa'):
pdfa_info = file_claims_pdfa(options.output_file)
if pdfa_info['pass']:
log.info("Output file is a %s (as expected)", pdfa_info['conformance'])
src/ocrmypdf/_pipelines/hocr_to_ocr_pdf.py
+11 -12
View File
@@ -6,7 +6,6 @@
from __future__ import annotations
import argparse
import logging
import logging.handlers
from collections.abc import Sequence
@@ -17,10 +16,8 @@ import PIL
from ocrmypdf._concurrent import Executor
from ocrmypdf._graft import OcrGrafter
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._pipeline import (
copy_final,
render_hocr_page,
)
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipeline import copy_final
from ocrmypdf._pipelines._common import (
HOCRResult,
do_get_pdfinfo,
@@ -34,6 +31,7 @@ from ocrmypdf._pipelines._common import (
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
from ocrmypdf._progressbar import ProgressBar
from ocrmypdf.exceptions import ExitCode
from ocrmypdf.helpers import available_cpu_count
log = logging.getLogger(__name__)
@@ -45,9 +43,8 @@ def _exec_hocrtransform_sync(page_context: PageContext) -> HOCRResult:
# No hOCR file, so no OCR was performed on this page.
return HOCRResult(pageno=page_context.pageno)
hocr_result = HOCRResult.from_json(hocr_json.read_text())
hocr_result.textpdf = render_hocr_page(
page_context.get_path('ocr_hocr.hocr'), page_context
)
# hOCR path is passed directly to the grafting phase where fpdf2 renders it
hocr_result.textpdf = page_context.get_path('ocr_hocr.hocr')
return hocr_result
@@ -55,7 +52,8 @@ def exec_hocr_to_ocr_pdf(context: PdfContext, executor: Executor) -> Sequence[st
"""Convert hOCR files to OCR PDF."""
# Run exec_page_sync on every page
options = context.options
max_workers = min(len(context.pdfinfo), options.jobs)
jobs = options.jobs or available_cpu_count()
max_workers = min(len(context.pdfinfo), jobs)
if max_workers > 1:
log.info("Continue processing %d pages concurrently", max_workers)
@@ -69,7 +67,8 @@ def exec_hocr_to_ocr_pdf(context: PdfContext, executor: Executor) -> Sequence[st
ocrgraft.graft_page(
pageno=result.pageno,
image=result.pdf_page_from_image,
textpdf=result.textpdf,
ocr_output=result.textpdf,
ocr_tree=result.ocr_tree,
autorotate_correction=result.orientation_correction,
)
pbar.update()
@@ -105,7 +104,7 @@ def exec_hocr_to_ocr_pdf(context: PdfContext, executor: Executor) -> Sequence[st
def run_hocr_to_ocr_pdf_pipeline(
options: argparse.Namespace,
options: OcrOptions,
*,
plugin_manager: OcrmypdfPluginManager,
) -> ExitCode:
@@ -119,7 +118,7 @@ def run_hocr_to_ocr_pdf_pipeline(
# Gather pdfinfo and create context
pdfinfo = do_get_pdfinfo(origin_pdf, executor, options)
context = PdfContext(options, work_folder, origin_pdf, pdfinfo, plugin_manager)
plugin_manager.hook.check_options(options=options)
plugin_manager.check_options(options=options)
optimize_messages = exec_hocr_to_ocr_pdf(context, executor)
return report_output_pdf(options, origin_pdf, optimize_messages)
src/ocrmypdf/_pipelines/ocr.py
+31 -19
View File
@@ -6,7 +6,6 @@
from __future__ import annotations
import argparse
import logging
import logging.handlers
from collections.abc import Sequence
@@ -19,13 +18,14 @@ import PIL
from ocrmypdf._concurrent import Executor
from ocrmypdf._graft import OcrGrafter
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipeline import (
copy_final,
is_ocr_required,
merge_sidecars,
ocr_engine_direct,
ocr_engine_hocr,
ocr_engine_textonly_pdf,
render_hocr_page,
triage,
validate_pdfinfo_options,
)
@@ -49,23 +49,32 @@ from ocrmypdf._validation import (
create_input_file,
)
from ocrmypdf.exceptions import ExitCode
from ocrmypdf.helpers import available_cpu_count
from ocrmypdf.models.ocr_element import OcrElement
log = logging.getLogger(__name__)
def _image_to_ocr_text(
page_context: PageContext, ocr_image_out: Path
) -> tuple[Path, Path]:
) -> tuple[Path | None, Path, OcrElement | None]:
"""Run OCR engine on image to create OCR PDF and text file."""
options = page_context.options
if options.pdf_renderer.startswith('hocr'):
hocr_out, text_out = ocr_engine_hocr(ocr_image_out, page_context)
ocr_out = render_hocr_page(hocr_out, page_context)
elif options.pdf_renderer == 'sandwich':
pdf_renderer = options.pdf_renderer
# fpdf2 is the default renderer (auto resolves to fpdf2)
if pdf_renderer in ('auto', 'fpdf2'):
# Use generate_ocr() if the engine supports it, otherwise use hOCR path
ocr_engine = page_context.plugin_manager.get_ocr_engine(options=options)
if ocr_engine and ocr_engine.supports_generate_ocr():
ocr_tree, text_out = ocr_engine_direct(ocr_image_out, page_context)
return None, text_out, ocr_tree
ocr_out, text_out = ocr_engine_hocr(ocr_image_out, page_context)
elif pdf_renderer == 'sandwich':
ocr_out, text_out = ocr_engine_textonly_pdf(ocr_image_out, page_context)
else:
raise NotImplementedError(f"pdf_renderer {options.pdf_renderer}")
return ocr_out, text_out
raise NotImplementedError(f"pdf_renderer {pdf_renderer}")
return ocr_out, text_out, None
def _exec_page_sync(page_context: PageContext) -> PageResult:
@@ -78,22 +87,24 @@ def _exec_page_sync(page_context: PageContext) -> PageResult:
ocr_image_out, pdf_page_from_image_out, orientation_correction = process_page(
page_context
)
ocr_out, text_out = _image_to_ocr_text(page_context, ocr_image_out)
ocr_out, text_out, ocr_tree = _image_to_ocr_text(page_context, ocr_image_out)
return PageResult(
pageno=page_context.pageno,
pdf_page_from_image=pdf_page_from_image_out,
ocr=ocr_out,
text=text_out,
orientation_correction=orientation_correction,
ocr_tree=ocr_tree,
)
def exec_concurrent(context: PdfContext, executor: Executor) -> Sequence[str]:
"""Execute the OCR pipeline concurrently."""
options = context.options
max_workers = min(len(context.pdfinfo), options.jobs)
jobs = options.jobs or available_cpu_count()
max_workers = min(len(context.pdfinfo), jobs)
if max_workers > 1:
log.info("Start processing %d pages concurrently", max_workers)
log.info("Starting processing with %d workers concurrently", max_workers)
sidecars: list[Path | None] = [None] * len(context.pdfinfo)
ocrgraft = OcrGrafter(context)
@@ -107,7 +118,8 @@ def exec_concurrent(context: PdfContext, executor: Executor) -> Sequence[str]:
ocrgraft.graft_page(
pageno=result.pageno,
image=result.pdf_page_from_image,
textpdf=result.ocr,
ocr_output=result.ocr,
ocr_tree=result.ocr_tree,
autorotate_correction=result.orientation_correction,
)
pbar.update(0.5)
@@ -119,7 +131,7 @@ def exec_concurrent(context: PdfContext, executor: Executor) -> Sequence[str]:
max_workers=max_workers,
progress_kwargs=dict(
total=len(context.pdfinfo),
desc='OCR' if options.tesseract_timeout > 0 else 'Image processing',
desc='OCR' if options.ocr_engine != 'none' else 'Image processing',
unit='page',
disable=not options.progress_bar,
),
@@ -150,7 +162,7 @@ def exec_concurrent(context: PdfContext, executor: Executor) -> Sequence[str]:
def _run_pipeline(
options: argparse.Namespace,
options: OcrOptions,
plugin_manager: OcrmypdfPluginManager,
) -> ExitCode:
with (
@@ -185,14 +197,14 @@ def _run_pipeline(
def run_pipeline_cli(
options: argparse.Namespace,
options: OcrOptions,
*,
plugin_manager: OcrmypdfPluginManager,
) -> ExitCode:
"""Run the OCR pipeline with command line exception handling.
Args:
options: The parsed command line options.
options: The parsed OCR options.
plugin_manager: The plugin manager to use. If not provided, one will be
created.
"""
@@ -200,14 +212,14 @@ def run_pipeline_cli(
def run_pipeline(
options: argparse.Namespace,
options: OcrOptions,
*,
plugin_manager: OcrmypdfPluginManager,
) -> ExitCode:
"""Run the OCR pipeline without command line exception handling.
Args:
options: The parsed command line options.
options: The parsed OCR options.
plugin_manager: The plugin manager to use. If not provided, one will be
created.
"""
src/ocrmypdf/_pipelines/pdf_to_hocr.py
+8 -8
View File
@@ -6,7 +6,6 @@
from __future__ import annotations
import argparse
import logging
import logging.handlers
import shutil
@@ -16,6 +15,7 @@ import PIL
from ocrmypdf._concurrent import Executor
from ocrmypdf._jobcontext import PageContext, PdfContext
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipeline import (
is_ocr_required,
ocr_engine_hocr,
@@ -31,9 +31,7 @@ from ocrmypdf._pipelines._common import (
worker_init,
)
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
from ocrmypdf._validation import (
set_lossless_reconstruction,
)
from ocrmypdf.helpers import available_cpu_count
log = logging.getLogger(__name__)
@@ -64,9 +62,10 @@ def exec_pdf_to_hocr(context: PdfContext, executor: Executor) -> None:
"""Execute the OCR pipeline concurrently and output hOCR."""
# Run exec_page_sync on every page
options = context.options
max_workers = min(len(context.pdfinfo), options.jobs)
jobs = options.jobs or available_cpu_count()
max_workers = min(len(context.pdfinfo), jobs)
if max_workers > 1:
log.info("Start processing %d pages concurrently", max_workers)
log.info("Starting processing with %d workers concurrently", max_workers)
executor(
use_threads=options.use_threads,
@@ -85,11 +84,13 @@ def exec_pdf_to_hocr(context: PdfContext, executor: Executor) -> None:
def run_hocr_pipeline(
options: argparse.Namespace,
options: OcrOptions,
*,
plugin_manager: OcrmypdfPluginManager,
) -> None:
"""Run pipeline to output hOCR."""
if options.output_folder is None:
raise ValueError("output_folder must be specified for hOCR pipeline")
with manage_work_folder(
work_folder=options.output_folder, retain=True, print_location=False
) as work_folder:
@@ -103,6 +104,5 @@ def run_hocr_pipeline(
options, work_folder, options.input_file, pdfinfo, plugin_manager
)
# Validate options are okay for this pdf
set_lossless_reconstruction(options)
validate_pdfinfo_options(context)
exec_pdf_to_hocr(context, executor)
src/ocrmypdf/_plugin_manager.py
+197 -44
View File
@@ -1,33 +1,44 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Plugin manager using pluggy."""
"""Plugin manager using pluggy with type-safe interface."""
from __future__ import annotations
import argparse
import importlib
import importlib.util
import pkgutil
import sys
from argparse import ArgumentParser
from collections.abc import Sequence
from logging import Handler
from pathlib import Path
from typing import TYPE_CHECKING
import pluggy
from pydantic import BaseModel
import ocrmypdf.builtin_plugins
from ocrmypdf import pluginspec
from ocrmypdf.cli import get_parser, plugins_only_parser
from ocrmypdf import Executor, PdfContext, pluginspec
from ocrmypdf._options import OcrOptions
from ocrmypdf._progressbar import ProgressBar
from ocrmypdf.helpers import Resolution
from ocrmypdf.pluginspec import OcrEngine
if TYPE_CHECKING:
from PIL import Image
from ocrmypdf._jobcontext import PageContext
from ocrmypdf.pdfinfo import PdfInfo
class OcrmypdfPluginManager(pluggy.PluginManager):
"""pluggy.PluginManager that can fork.
class OcrmypdfPluginManager:
"""Type-safe wrapper around pluggy.PluginManager.
Capable of reconstructing itself in child workers.
Capable of reconstructing itself in child workers via pickle.
Arguments:
setup_func: callback that initializes the plugin manager with all
standard plugins
This class provides type-safe methods for all hooks defined in pluginspec.py,
removing the need for unsafe `hook.method_name()` calls.
"""
def __init__(
@@ -37,19 +48,28 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
builtins: bool = True,
**kwargs,
):
self.__init_args = args
self.__init_kwargs = kwargs
self.__plugins = plugins
self.__builtins = builtins
super().__init__(*args, **kwargs)
self.setup_plugins()
self._init_args = args
self._init_kwargs = kwargs
self._plugins = plugins
self._builtins = builtins
self._pm = pluggy.PluginManager(*args, **kwargs)
self._setup_plugins()
@property
def pluggy(self) -> pluggy.PluginManager:
"""Access the underlying pluggy.PluginManager for advanced use cases.
This is useful for plugins that need to call methods like set_blocked()
in their initialize hook.
"""
return self._pm
def __getstate__(self):
state = dict(
init_args=self.__init_args,
plugins=self.__plugins,
builtins=self.__builtins,
init_kwargs=self.__init_kwargs,
init_args=self._init_args,
plugins=self._plugins,
builtins=self._builtins,
init_kwargs=self._init_kwargs,
)
return state
@@ -61,23 +81,23 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
**state['init_kwargs'],
)
def setup_plugins(self):
self.add_hookspecs(pluginspec)
def _setup_plugins(self):
self._pm.add_hookspecs(pluginspec)
# 1. Register builtins
if self.__builtins:
if self._builtins:
for module in sorted(
pkgutil.iter_modules(ocrmypdf.builtin_plugins.__path__)
):
name = f'ocrmypdf.builtin_plugins.{module.name}'
module = importlib.import_module(name)
self.register(module)
self._pm.register(module)
# 2. Register setuptools plugins
self.load_setuptools_entrypoints('ocrmypdf')
self._pm.load_setuptools_entrypoints('ocrmypdf')
# 3. Register plugins specified on command line
for name in self.__plugins:
for name in self._plugins:
if isinstance(name, Path) or name.endswith('.py'):
# Import by filename
module_name = Path(name).stem
@@ -88,7 +108,156 @@ class OcrmypdfPluginManager(pluggy.PluginManager):
else:
# Import by dotted module name
module = importlib.import_module(name)
self.register(module)
self._pm.register(module)
# =========================================================================
# Type-safe hook methods
# =========================================================================
# --- firstresult hooks ---
def get_logging_console(self) -> Handler | None:
"""Returns a custom logging handler for progress bar compatibility."""
return self._pm.hook.get_logging_console()
def get_executor(self, *, progressbar_class: type[ProgressBar]) -> Executor | None:
"""Returns an executor for parallel processing."""
return self._pm.hook.get_executor(progressbar_class=progressbar_class)
def get_progressbar_class(self) -> type[ProgressBar] | None:
"""Returns a progress bar class."""
return self._pm.hook.get_progressbar_class()
def rasterize_pdf_page(
self,
*,
input_file: Path,
output_file: Path,
raster_device: str,
raster_dpi: Resolution,
pageno: int,
page_dpi: Resolution | None,
rotation: int | None,
filter_vector: bool,
stop_on_soft_error: bool,
options: OcrOptions | None,
use_cropbox: bool,
) -> Path | None:
"""Rasterize one page of a PDF at specified resolution."""
return self._pm.hook.rasterize_pdf_page(
input_file=input_file,
output_file=output_file,
raster_device=raster_device,
raster_dpi=raster_dpi,
pageno=pageno,
page_dpi=page_dpi,
rotation=rotation,
filter_vector=filter_vector,
stop_on_soft_error=stop_on_soft_error,
options=options,
use_cropbox=use_cropbox,
)
def filter_ocr_image(
self, *, page: PageContext, image: Image.Image
) -> Image.Image | None:
"""Filter the image before it is sent to OCR."""
return self._pm.hook.filter_ocr_image(page=page, image=image)
def filter_page_image(
self, *, page: PageContext, image_filename: Path
) -> Path | None:
"""Filter the whole page image before it is inserted into the PDF."""
return self._pm.hook.filter_page_image(page=page, image_filename=image_filename)
def filter_pdf_page(
self, *, page: PageContext, image_filename: Path, output_pdf: Path
) -> Path | None:
"""Convert a filtered whole page image into a PDF."""
return self._pm.hook.filter_pdf_page(
page=page, image_filename=image_filename, output_pdf=output_pdf
)
def get_ocr_engine(self, *, options: OcrOptions | None = None) -> OcrEngine | None:
"""Returns an OcrEngine to use for processing.
Args:
options: OcrOptions to pass to the hook for engine selection.
"""
return self._pm.hook.get_ocr_engine(options=options)
def generate_pdfa(
self,
*,
pdf_pages: list[Path],
pdfmark: Path,
output_file: Path,
context: PdfContext,
pdf_version: str,
pdfa_part: str,
progressbar_class: type[ProgressBar] | None,
stop_on_soft_error: bool,
) -> Path | None:
"""Generate a PDF/A file."""
return self._pm.hook.generate_pdfa(
pdf_pages=pdf_pages,
pdfmark=pdfmark,
output_file=output_file,
context=context,
pdf_version=pdf_version,
pdfa_part=pdfa_part,
progressbar_class=progressbar_class,
stop_on_soft_error=stop_on_soft_error,
)
def optimize_pdf(
self,
*,
input_pdf: Path,
output_pdf: Path,
context: PdfContext,
executor: Executor,
linearize: bool,
) -> tuple[Path, Sequence[str]] | None:
"""Optimize a PDF after OCR processing."""
return self._pm.hook.optimize_pdf(
input_pdf=input_pdf,
output_pdf=output_pdf,
context=context,
executor=executor,
linearize=linearize,
)
def is_optimization_enabled(self, *, context: PdfContext) -> bool | None:
"""Returns whether optimization is enabled for given context."""
return self._pm.hook.is_optimization_enabled(context=context)
# --- non-firstresult hooks ---
def initialize(self, *, plugin_manager: pluggy.PluginManager) -> list[None]:
"""Called when plugins are first loaded.
Args:
plugin_manager: The underlying pluggy.PluginManager, allowing
plugins to call methods like set_blocked().
"""
return self._pm.hook.initialize(plugin_manager=plugin_manager)
def add_options(self, *, parser: ArgumentParser) -> list[None]:
"""Allows plugins to add command line and API arguments."""
return self._pm.hook.add_options(parser=parser)
def register_options(self) -> list[dict[str, type[BaseModel]]]:
"""Returns plugin option models keyed by namespace."""
return self._pm.hook.register_options()
def check_options(self, *, options: OcrOptions) -> list[None]:
"""Called to validate options after parsing."""
return self._pm.hook.check_options(options=options)
def validate(self, *, pdfinfo: PdfInfo, options: OcrOptions) -> list[None]:
"""Called to validate options and pdfinfo after PDF is loaded."""
return self._pm.hook.validate(pdfinfo=pdfinfo, options=options)
def get_plugin_manager(
@@ -101,20 +270,4 @@ def get_plugin_manager(
)
def get_parser_options_plugins(
args: Sequence[str],
) -> tuple[argparse.ArgumentParser, argparse.Namespace, pluggy.PluginManager]:
pre_options, _unused = plugins_only_parser.parse_known_args(args=args)
plugin_manager = get_plugin_manager(pre_options.plugins)
parser = get_parser()
plugin_manager.hook.initialize( # pylint: disable=no-member
plugin_manager=plugin_manager
)
plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
options = parser.parse_args(args=args)
return parser, options, plugin_manager
__all__ = ['OcrmypdfPluginManager', 'get_plugin_manager', 'get_parser_options_plugins']
__all__ = ['OcrmypdfPluginManager', 'get_plugin_manager']
src/ocrmypdf/_plugin_registry.py
+50
View File
@@ -0,0 +1,50 @@
# SPDX-FileCopyrightText: 2024 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Plugin option registry for dynamic model composition."""
from __future__ import annotations
import logging
from pydantic import BaseModel
log = logging.getLogger(__name__)
class PluginOptionRegistry:
"""Registry for plugin option models.
This registry collects option models from plugins during initialization.
Plugin options can be accessed via nested namespaces on OcrOptions
(e.g., options.tesseract.timeout) or via flat field names for backward
compatibility (e.g., options.tesseract_timeout).
"""
def __init__(self):
self._option_models: dict[str, type[BaseModel]] = {}
def register_option_model(
self, namespace: str, model_class: type[BaseModel]
) -> None:
"""Register a plugin's option model.
Args:
namespace: The namespace for the plugin options (e.g., 'tesseract')
model_class: The Pydantic model class for the plugin options
"""
if namespace in self._option_models:
log.warning(
f"Plugin option namespace '{namespace}' already registered, overriding"
)
self._option_models[namespace] = model_class
log.debug(
f"Registered plugin option model for namespace '{namespace}': "
f"{model_class.__name__}"
)
def get_registered_models(self) -> dict[str, type[BaseModel]]:
"""Get all registered plugin option models."""
return self._option_models.copy()
src/ocrmypdf/_progressbar.py
+14 -6
View File
@@ -48,9 +48,11 @@ class ProgressBar(Protocol):
A brief description of the current step (e.g. "Scanning contents",
"OCR", "PDF/A conversion"). OCRmyPDF updates this before each major step.
unit (str | None):
A short label for the type of work being tracked (e.g. "page", "%", "image").
A short label for the type of work being tracked
(e.g. "page", "%", "image").
disable (bool):
If ``True``, progress updates are suppressed (no output). Defaults to ``False``.
If ``True``, progress updates are suppressed (no output).
Defaults to ``False``.
**kwargs:
Future or extra parameters that OCRmyPDF might pass. Implementations
should accept and ignore unrecognized keywords gracefully.
@@ -64,7 +66,8 @@ class ProgressBar(Protocol):
from ocrmypdf import hookimpl
class ConsoleProgressBar(ProgressBar):
def __init__(self, *, total=None, desc=None, unit=None, disable=False, **kwargs):
def __init__(self, *, total=None, desc=None, unit=None, disable=False,
**kwargs):
self.total = total
self.desc = desc
self.unit = unit
@@ -73,7 +76,9 @@ class ProgressBar(Protocol):
def __enter__(self):
if not self.disable:
print(f"Starting {self.desc or 'an OCR task'} (total={self.total} {self.unit})")
print(f"Starting {self.desc or 'an OCR task'} "
f"(total={self.total} {self.unit})"
)
return self
def __exit__(self, exc_type, exc_value, traceback):
@@ -86,7 +91,7 @@ class ProgressBar(Protocol):
def update(self, n=1, *, completed=None):
if completed is not None:
# If 'completed' is given, you could set self.current = completed
# If 'completed' is given, set self.current
# but let's just read it to show usage
print(f"Absolute completion reported: {completed}")
# Otherwise, we increment by 'n'
@@ -94,7 +99,10 @@ class ProgressBar(Protocol):
if not self.disable:
if self.total:
percent = (self.current / self.total) * 100
print(f"{self.desc}: {self.current}/{self.total} ({percent:.1f}%)")
print(
f"{self.desc}: {self.current}"
f"/{self.total} ({percent:.1f}%)"
)
else:
print(f"{self.desc}: {self.current} units done")
src/ocrmypdf/_validation.py
+49 -127
View File
@@ -6,22 +6,19 @@
from __future__ import annotations
import locale
import logging
import os
import sys
import unicodedata
from argparse import Namespace
from collections.abc import Sequence
from pathlib import Path
from shutil import copyfileobj
import pikepdf
import PIL
from pluggy import PluginManager
from ocrmypdf._defaults import DEFAULT_LANGUAGE, DEFAULT_ROTATE_PAGES_THRESHOLD
from ocrmypdf._defaults import DEFAULT_ROTATE_PAGES_THRESHOLD
from ocrmypdf._exec import unpaper
from ocrmypdf._options import OcrOptions
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
from ocrmypdf.exceptions import (
BadArgsError,
InputFileError,
@@ -30,7 +27,6 @@ from ocrmypdf.exceptions import (
)
from ocrmypdf.helpers import (
is_file_writable,
monotonic,
running_in_docker,
running_in_snap,
safe_symlink,
@@ -51,13 +47,19 @@ def check_platform() -> None:
def check_options_languages(
options: Namespace, ocr_engine_languages: list[str]
options: OcrOptions, ocr_engine_languages: list[str]
) -> None:
if not options.languages:
options.languages = [DEFAULT_LANGUAGE]
system_lang = locale.getlocale()[0]
if system_lang and not system_lang.startswith('en'):
log.debug("No language specified; assuming --language %s", DEFAULT_LANGUAGE)
# Check for blocked languages first, before checking if they're installed
DENIED_LANGUAGES = {'equ', 'osd'}
blocked = DENIED_LANGUAGES & set(options.languages)
if blocked:
raise BadArgsError(
"The following languages are for Tesseract's internal use and "
"should not be issued explicitly: "
f"{', '.join(blocked)}\n"
"Remove them from the -l/--language argument."
)
if not ocr_engine_languages:
return
@@ -81,36 +83,7 @@ def check_options_languages(
raise MissingDependencyError(msg)
def check_options_output(options: Namespace) -> None:
if options.output_type == 'none' and options.output_file not in (os.devnull, '-'):
raise BadArgsError(
"Since you specified `--output-type none`, the output file "
f"{options.output_file} cannot be produced. Set the output file to "
f"`-` to suppress this message."
)
def set_lossless_reconstruction(options: Namespace) -> None:
lossless_reconstruction = False
if not any(
(
options.deskew,
options.clean_final,
options.force_ocr,
options.remove_background,
)
):
lossless_reconstruction = True
options.lossless_reconstruction = lossless_reconstruction
if not options.lossless_reconstruction and options.redo_ocr:
raise BadArgsError(
"--redo-ocr is not currently compatible with --deskew, "
"--clean-final, and --remove-background"
)
def check_options_sidecar(options: Namespace) -> None:
def check_options_sidecar(options: OcrOptions) -> None:
if options.sidecar == '\0':
if options.output_file == '-':
raise BadArgsError("--sidecar filename needed when output file is stdout.")
@@ -125,7 +98,7 @@ def check_options_sidecar(options: Namespace) -> None:
)
def check_options_preprocessing(options: Namespace) -> None:
def check_options_preprocessing(options: OcrOptions) -> None:
if options.clean_final:
options.clean = True
if options.unpaper_args and not options.clean:
@@ -152,98 +125,45 @@ def check_options_preprocessing(options: Namespace) -> None:
raise BadArgsError("--unpaper-args: " + str(e)) from e
def _pages_from_ranges(ranges: str) -> set[int]:
pages: list[int] = []
page_groups = ranges.replace(' ', '').split(',')
for group in page_groups:
if not group:
continue
try:
start, end = group.split('-')
except ValueError:
pages.append(int(group) - 1)
else:
try:
new_pages = list(range(int(start) - 1, int(end)))
if not new_pages:
raise BadArgsError(
f"invalid page subrange '{start}-{end}'"
) from None
pages.extend(new_pages)
except ValueError:
raise BadArgsError(f"invalid page subrange '{group}'") from None
if not pages:
raise BadArgsError(
f"The string of page ranges '{ranges}' did not contain any recognizable "
f"page ranges."
)
if not monotonic(pages):
log.warning(
"List of pages to process contains duplicate pages, or pages that are "
"out of order"
)
if any(page < 0 for page in pages):
raise BadArgsError("pages refers to a page number less than 1")
log.debug("OCRing only these pages: %s", pages)
return set(pages)
def check_options_ocr_behavior(options: Namespace) -> None:
exclusive_options = sum(
(1 if opt else 0)
for opt in (options.force_ocr, options.skip_text, options.redo_ocr)
)
if exclusive_options >= 2:
raise BadArgsError("Choose only one of --force-ocr, --skip-text, --redo-ocr.")
if options.pages:
options.pages = _pages_from_ranges(options.pages)
def check_options_metadata(options: Namespace) -> None:
docinfo = [options.title, options.author, options.keywords, options.subject]
for s in (m for m in docinfo if m):
for char in s:
if unicodedata.category(char) == 'Co' or ord(char) >= 0x10000:
hexchar = hex(ord(char))[2:].upper()
raise ValueError(
"One of the metadata strings contains "
"an unsupported Unicode character: "
f"{char} (U+{hexchar})"
)
def check_options_pillow(options: Namespace) -> None:
PIL.Image.MAX_IMAGE_PIXELS = int(options.max_image_mpixels * 1_000_000)
if PIL.Image.MAX_IMAGE_PIXELS == 0:
PIL.Image.MAX_IMAGE_PIXELS = None # type: ignore
def _check_plugin_invariant_options(options: Namespace) -> None:
def _check_plugin_invariant_options(options: OcrOptions) -> None:
check_platform()
check_options_metadata(options)
check_options_output(options)
set_lossless_reconstruction(options)
check_options_sidecar(options)
check_options_preprocessing(options)
check_options_ocr_behavior(options)
check_options_pillow(options)
def _check_plugin_options(options: Namespace, plugin_manager: PluginManager) -> None:
plugin_manager.hook.check_options(options=options)
ocr_engine_languages = plugin_manager.hook.get_ocr_engine().languages(options)
def _check_plugin_options(
options: OcrOptions, plugin_manager: OcrmypdfPluginManager
) -> None:
# First, let plugins check their external dependencies
plugin_manager.check_options(options=options)
# Then check OCR engine language support
ocr_engine_languages = plugin_manager.get_ocr_engine(options=options).languages(
options
)
check_options_languages(options, ocr_engine_languages)
# Finally, run comprehensive validation using the coordinator
from ocrmypdf._validation_coordinator import ValidationCoordinator
def check_options(options: Namespace, plugin_manager: PluginManager) -> None:
coordinator = ValidationCoordinator(plugin_manager)
coordinator.validate_all_options(options)
def check_options(options: OcrOptions, plugin_manager: OcrmypdfPluginManager) -> None:
"""Check options for validity and consistency.
This function coordinates validation across the entire system:
1. Core validation (platform, files, preprocessing)
2. Plugin external dependency validation
3. Plugin-specific validation (handled by plugin models)
4. Cross-cutting validation (handled by validation coordinator)
"""
_check_plugin_invariant_options(options)
_check_plugin_options(options, plugin_manager)
def create_input_file(options: Namespace, work_folder: Path) -> tuple[Path, str]:
def create_input_file(options: OcrOptions, work_folder: Path) -> tuple[Path, str]:
if options.input_file == '-':
# stdin
log.info('reading file from standard input')
@@ -288,7 +208,7 @@ def create_input_file(options: Namespace, work_folder: Path) -> tuple[Path, str]
raise InputFileError(msg) from e
def check_requested_output_file(options: Namespace) -> None:
def check_requested_output_file(options: OcrOptions) -> None:
if options.output_file == '-':
if sys.stdout.isatty():
raise BadArgsError(
@@ -306,7 +226,7 @@ def check_requested_output_file(options: Namespace) -> None:
def report_output_file_size(
options: Namespace,
options: OcrOptions,
input_file: Path,
output_file: Path,
optimize_messages: Sequence[str] | None = None,
@@ -335,13 +255,15 @@ def report_output_file_size(
'clean_final',
'remove_background',
'oversample',
'force_ocr',
}
for arg in image_preproc:
if getattr(options, arg, False):
reasons.append(
f"--{arg.replace('_', '-')} was issued, causing transcoding."
)
# Check force_ocr via the backward-compatible property
if options.force_ocr:
reasons.append("--force-ocr (or --mode force) was issued, causing transcoding.")
reasons.extend(optimize_messages)
src/ocrmypdf/_validation_coordinator.py
+144
View File
@@ -0,0 +1,144 @@
# SPDX-FileCopyrightText: 2024 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Validation coordinator for plugin options and cross-cutting concerns."""
from __future__ import annotations
import logging
import os
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import pluggy
from ocrmypdf._options import OcrOptions
log = logging.getLogger(__name__)
class ValidationCoordinator:
"""Coordinates validation across plugin models and core options."""
def __init__(self, plugin_manager: pluggy.PluginManager):
self.plugin_manager = plugin_manager
self.registry = getattr(plugin_manager, '_option_registry', None)
def validate_all_options(self, options: OcrOptions) -> None:
"""Run comprehensive validation on all options.
This runs validation in the correct order:
1. Plugin self-validation (already done by Pydantic)
2. Plugin context validation (requires external context)
3. Cross-cutting validation (between plugins and core)
Args:
options: The options to validate
"""
# Step 1: Plugin context validation
self._validate_plugin_contexts(options)
# Step 2: Cross-cutting validation
self._validate_cross_cutting_concerns(options)
def _validate_plugin_contexts(self, options: OcrOptions) -> None:
"""Validate plugin options that require external context."""
# For now, we'll run the plugin validation directly since the models
# are still being integrated. This ensures the validation warnings
# and checks still work as expected.
# Run Tesseract validation
self._validate_tesseract_options(options)
# Run Optimize validation
self._validate_optimize_options(options)
def _validate_tesseract_options(self, options: OcrOptions) -> None:
"""Validate Tesseract options."""
# Check pagesegmode warning
if options.tesseract.pagesegmode in (0, 2):
log.warning(
"The tesseract-pagesegmode you selected will disable OCR. "
"This may cause processing to fail."
)
# Check downsample consistency
if (
options.tesseract.downsample_above != 32767
and not options.tesseract.downsample_large_images
):
log.warning(
"The --tesseract-downsample-above argument will have no effect unless "
"--tesseract-downsample-large-images is also given."
)
# Note: blocked languages (equ, osd) are checked earlier in
# check_options_languages() to ensure the check runs before
# the missing language check.
def _validate_optimize_options(self, options: OcrOptions) -> None:
"""Validate optimization options."""
# Check optimization consistency
if options.optimize == 0 and any(
[
options.png_quality and options.png_quality > 0,
options.jpeg_quality and options.jpeg_quality > 0,
]
):
log.warning(
"The arguments --png-quality and --jpeg-quality "
"will be ignored because --optimize=0."
)
def _validate_cross_cutting_concerns(self, options: OcrOptions) -> None:
"""Validate cross-cutting concerns that span multiple plugins."""
from ocrmypdf._options import ProcessingMode
# Handle deprecated pdf_renderer values
self._handle_deprecated_pdf_renderer(options)
# Note: Mutual exclusivity of force_ocr/skip_text/redo_ocr is now enforced
# by the ProcessingMode enum - only one mode can be active at a time.
# Validate redo mode compatibility
if options.mode == ProcessingMode.redo and (
options.deskew or options.clean_final or options.remove_background
):
raise ValueError(
"--redo-ocr (or --mode redo) is not currently compatible with "
"--deskew, --clean-final, and --remove-background"
)
# Validate output type compatibility
if options.output_type == 'none' and str(options.output_file) not in (
os.devnull,
'-',
):
raise ValueError(
"Since you specified `--output-type none`, the output file "
f"{options.output_file} cannot be produced. Set the output file to "
"`-` to suppress this message."
)
# Validate PDF/A image compression compatibility
if (
options.ghostscript.pdfa_image_compression
and options.ghostscript.pdfa_image_compression != 'auto'
and not options.output_type.startswith('pdfa')
):
log.warning(
"--pdfa-image-compression argument only applies when "
"--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
)
def _handle_deprecated_pdf_renderer(self, options: OcrOptions) -> None:
"""Handle deprecated pdf_renderer values by redirecting to fpdf2."""
if options.pdf_renderer in ('hocr', 'hocrdebug'):
log.info(
"The '%s' PDF renderer has been removed. Using 'fpdf2' instead, "
"which provides full international language support, proper RTL "
"rendering, and improved text positioning.",
options.pdf_renderer,
)
# Modify the options object to use fpdf2
object.__setattr__(options, 'pdf_renderer', 'fpdf2')
src/ocrmypdf/api.py
+517 -171
View File
@@ -43,24 +43,22 @@ import logging
import os
import sys
import threading
from argparse import Namespace
from collections.abc import Iterable, Sequence
from enum import IntEnum
from io import IOBase
from pathlib import Path
from typing import BinaryIO
from typing import BinaryIO, overload
from warnings import warn
import pluggy
from ocrmypdf._logging import PageNumberFilter
from ocrmypdf._options import OcrOptions
from ocrmypdf._pipelines.hocr_to_ocr_pdf import run_hocr_to_ocr_pdf_pipeline
from ocrmypdf._pipelines.ocr import run_pipeline, run_pipeline_cli
from ocrmypdf._pipelines.pdf_to_hocr import run_hocr_pipeline
from ocrmypdf._plugin_manager import get_plugin_manager
from ocrmypdf._plugin_manager import OcrmypdfPluginManager, get_plugin_manager
from ocrmypdf._validation import check_options
from ocrmypdf.cli import ArgumentParser, get_parser
from ocrmypdf.helpers import is_iterable_notstr
from ocrmypdf.exceptions import ExitCode
StrPath = Path | str | bytes
PathOrIO = BinaryIO | StrPath
@@ -71,6 +69,67 @@ PathOrIO = BinaryIO | StrPath
_api_lock = threading.Lock()
def setup_plugin_infrastructure(
plugins: Sequence[Path | str] | None = None,
plugin_manager: OcrmypdfPluginManager | None = None,
) -> OcrmypdfPluginManager:
"""Set up plugin infrastructure with proper initialization.
This function handles:
1. Creating or validating the plugin manager
2. Calling plugin initialization hooks
3. Setting up plugin option registry
Args:
plugins: List of plugin paths/names to load
plugin_manager: Existing plugin manager (if any)
Returns:
Properly initialized plugin manager
Raises:
ValueError: If both plugins and plugin_manager are provided
"""
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
if not plugins:
plugins = []
elif isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins)
# Create plugin manager if not provided
if not plugin_manager:
plugin_manager = get_plugin_manager(plugins)
# Initialize plugins (pass the underlying pluggy manager)
plugin_manager.initialize(plugin_manager=plugin_manager.pluggy)
# Initialize plugin option registry
from ocrmypdf._plugin_registry import PluginOptionRegistry
registry = PluginOptionRegistry()
# Let plugins register their option models
option_models = plugin_manager.register_options()
all_plugin_models: dict[str, type] = {}
for plugin_options in option_models:
if plugin_options: # Skip None returns
for namespace, model_class in plugin_options.items():
registry.register_option_model(namespace, model_class)
all_plugin_models[namespace] = model_class
# Register plugin models with OcrOptions for dynamic nested access
OcrOptions.register_plugin_models(all_plugin_models)
# Store registry in plugin manager for later access
plugin_manager._option_registry = registry
return plugin_manager
class Verbosity(IntEnum):
"""Verbosity level for configure_logging."""
@@ -86,7 +145,7 @@ def configure_logging(
*,
progress_bar_friendly: bool = True,
manage_root_logger: bool = False,
plugin_manager: pluggy.PluginManager | None = None,
plugin_manager: OcrmypdfPluginManager | None = None,
):
"""Set up logging.
@@ -133,7 +192,7 @@ def configure_logging(
console = None
if plugin_manager and progress_bar_friendly:
console = plugin_manager.hook.get_logging_console()
console = plugin_manager.get_logging_console()
if not console:
console = logging.StreamHandler(stream=sys.stderr)
@@ -165,6 +224,8 @@ def configure_logging(
pdfminer_log.setLevel(logging.ERROR)
pil_log = logging.getLogger('PIL')
pil_log.setLevel(logging.INFO)
fonttools_log = logging.getLogger('fontTools')
fonttools_log.setLevel(logging.ERROR)
if manage_root_logger:
logging.captureWarnings(True)
@@ -172,98 +233,130 @@ def configure_logging(
return log
def _kwargs_to_cmdline(
*, defer_kwargs: set[str], **kwargs
) -> tuple[list[str | bytes], dict[str, str | bytes]]:
"""Convert kwargs to command line arguments."""
cmdline: list[str | bytes] = []
deferred = {}
for arg, val in kwargs.items():
if val is None:
continue
def _check_no_conflicting_ocr_params(
locals_dict: dict,
kwargs: dict,
excluded: set[str] | None = None,
) -> None:
"""Check that no individual OCR parameters conflict with OcrOptions.
# Skip arguments that are handled elsewhere
if arg in defer_kwargs:
deferred[arg] = val
continue
When a user passes an OcrOptions object, they should not also pass
individual OCR parameters (except plugins/plugin_manager which are
handled separately).
cmd_style_arg = arg.replace('_', '-')
Args:
locals_dict: The locals() dict from the calling function.
kwargs: The **kwargs dict from the calling function.
excluded: Parameter names to exclude from conflict checking.
# Booleans are special: add only if True, omit for False
if isinstance(val, bool):
if val:
cmdline.append(f"--{cmd_style_arg}")
continue
Raises:
ValueError: If conflicting parameters are found.
"""
if excluded is None:
excluded = set()
if is_iterable_notstr(val):
for elem in val:
cmdline.append(f"--{cmd_style_arg}")
cmdline.append(elem)
continue
# Parameters that are allowed alongside OcrOptions
allowed_with_options = {
'input_file_or_options',
'options', # The OcrOptions object itself after assignment
'plugins',
'plugin_manager',
'kwargs',
} | excluded
# We have a parameter
cmdline.append(f"--{cmd_style_arg}")
if isinstance(val, int | float):
cmdline.append(str(val))
elif isinstance(val, str):
cmdline.append(val)
elif isinstance(val, Path):
cmdline.append(str(val))
else:
raise TypeError(f"{arg}: {val} ({type(val)})")
return cmdline, deferred
# Check all locals that are OCR parameters (not None and not allowed)
conflicts = [
name
for name, value in locals_dict.items()
if value is not None and name not in allowed_with_options
]
# Check kwargs
conflicts.extend(kwargs.keys())
if conflicts:
raise ValueError(
f"When passing OcrOptions as the first argument, do not pass "
f"additional OCR parameters. Conflicting parameters: "
f"{', '.join(sorted(conflicts))}. "
f"Set these values in OcrOptions instead."
)
def create_options(
*, input_file: PathOrIO, output_file: PathOrIO, parser: ArgumentParser, **kwargs
) -> Namespace:
) -> OcrOptions:
"""Construct an options object from the input/output files and keyword arguments.
Args:
input_file: Input file path or file object.
output_file: Output file path or file object.
parser: ArgumentParser object.
parser: ArgumentParser object (kept for compatibility,
may be used for plugin validation).
**kwargs: Keyword arguments.
Returns:
argparse.Namespace: A Namespace object containing the parsed arguments.
OcrOptions: An options object containing the parsed arguments.
Raises:
TypeError: If the type of a keyword argument is not supported.
"""
cmdline, deferred = _kwargs_to_cmdline(
defer_kwargs={'progress_bar', 'plugins', 'parser', 'input_file', 'output_file'},
**kwargs,
)
if isinstance(input_file, BinaryIO | IOBase):
cmdline.append('stream://input_file')
else:
cmdline.append(os.fspath(input_file))
if isinstance(output_file, BinaryIO | IOBase):
cmdline.append('stream://output_file')
else:
cmdline.append(os.fspath(output_file))
if 'sidecar' in kwargs and isinstance(kwargs['sidecar'], BinaryIO | IOBase):
cmdline.append('--sidecar')
cmdline.append('stream://sidecar')
# Prepare kwargs for direct OcrOptions construction
options_kwargs = kwargs.copy()
parser.enable_api_mode()
options = parser.parse_args(cmdline)
for keyword, val in deferred.items():
setattr(options, keyword, val)
# Set input and output files
options_kwargs['input_file'] = input_file
options_kwargs['output_file'] = output_file
if options.input_file == 'stream://input_file':
options.input_file = input_file
if options.output_file == 'stream://output_file':
options.output_file = output_file
if options.sidecar == 'stream://sidecar':
options.sidecar = kwargs['sidecar']
# Handle special stream cases for sidecar
if 'sidecar' in options_kwargs and isinstance(
options_kwargs['sidecar'], BinaryIO | IOBase
):
# Keep the stream object as-is - OcrOptions can handle it
pass
return options
# Remove None values to let OcrOptions use its defaults
options_kwargs = {k: v for k, v in options_kwargs.items() if v is not None}
# Remove any kwargs that aren't OcrOptions fields and store in extra_attrs
extra_attrs = {}
ocr_fields = set(OcrOptions.model_fields.keys())
# Legacy mode flags are handled by OcrOptions model validator
legacy_mode_flags = {'force_ocr', 'skip_text', 'redo_ocr'}
# Known extra attributes that should be preserved
known_extra = {'progress_bar', 'plugins'}
for key in list(options_kwargs.keys()):
if key in ocr_fields or key in legacy_mode_flags or key in known_extra:
continue
extra_attrs[key] = options_kwargs.pop(key)
# Create OcrOptions directly
try:
options = OcrOptions(**options_kwargs)
# Add any extra attributes
if extra_attrs:
options.extra_attrs.update(extra_attrs)
return options
except Exception as e:
# If direct construction fails, provide a helpful error message
raise TypeError(f"Failed to create OcrOptions: {e}") from e
def ocr( # noqa: D417
input_file: PathOrIO,
@overload
def ocr(
options: OcrOptions,
/,
*,
plugins: Iterable[Path | str] | None = None,
plugin_manager: OcrmypdfPluginManager | None = None,
) -> ExitCode: ...
@overload
def ocr(
input_file_or_options: PathOrIO,
output_file: PathOrIO,
*,
language: Iterable[str] | None = None,
@@ -284,6 +377,7 @@ def ocr( # noqa: D417
unpaper_args: str | None = None,
oversample: int | None = None,
remove_vectors: bool | None = None,
mode: str | None = None,
force_ocr: bool | None = None,
skip_text: bool | None = None,
redo_ocr: bool | None = None,
@@ -301,6 +395,7 @@ def ocr( # noqa: D417
tesseract_oem: int | None = None,
tesseract_thresholding: int | None = None,
pdf_renderer: str | None = None,
rasterizer: str | None = None,
tesseract_timeout: float | None = None,
tesseract_non_ocr_timeout: float | None = None,
tesseract_downsample_above: int | None = None,
@@ -314,13 +409,89 @@ def ocr( # noqa: D417
continue_on_soft_render_error: bool | None = None,
invalidate_digital_signatures: bool | None = None,
plugins: Iterable[Path | str] | None = None,
plugin_manager=None,
plugin_manager: OcrmypdfPluginManager | None = None,
keep_temporary_files: bool | None = None,
progress_bar: bool | None = None,
**kwargs,
):
) -> ExitCode: ...
def ocr( # noqa: D417
input_file_or_options: PathOrIO | OcrOptions,
output_file: PathOrIO | None = None,
*,
language: Iterable[str] | None = None,
image_dpi: int | None = None,
output_type: str | None = None,
sidecar: PathOrIO | None = None,
jobs: int | None = None,
use_threads: bool | None = None,
title: str | None = None,
author: str | None = None,
subject: str | None = None,
keywords: str | None = None,
rotate_pages: bool | None = None,
remove_background: bool | None = None,
deskew: bool | None = None,
clean: bool | None = None,
clean_final: bool | None = None,
unpaper_args: str | None = None,
oversample: int | None = None,
remove_vectors: bool | None = None,
mode: str | None = None,
force_ocr: bool | None = None, # Legacy, use mode='force' instead
skip_text: bool | None = None, # Legacy, use mode='skip' instead
redo_ocr: bool | None = None, # Legacy, use mode='redo' instead
skip_big: float | None = None,
optimize: int | None = None,
jpg_quality: int | None = None,
png_quality: int | None = None,
jbig2_lossy: bool | None = None, # Deprecated, ignored
jbig2_page_group_size: int | None = None, # Deprecated, ignored
jbig2_threshold: float | None = None,
pages: str | None = None,
max_image_mpixels: float | None = None,
tesseract_config: Iterable[str] | None = None,
tesseract_pagesegmode: int | None = None,
tesseract_oem: int | None = None,
tesseract_thresholding: int | None = None,
pdf_renderer: str | None = None,
rasterizer: str | None = None,
tesseract_timeout: float | None = None,
tesseract_non_ocr_timeout: float | None = None,
tesseract_downsample_above: int | None = None,
tesseract_downsample_large_images: bool | None = None,
rotate_pages_threshold: float | None = None,
pdfa_image_compression: str | None = None,
color_conversion_strategy: str | None = None,
user_words: os.PathLike | None = None,
user_patterns: os.PathLike | None = None,
fast_web_view: float | None = None,
continue_on_soft_render_error: bool | None = None,
invalidate_digital_signatures: bool | None = None,
plugins: Iterable[Path | str] | None = None,
plugin_manager: OcrmypdfPluginManager | None = None,
keep_temporary_files: bool | None = None,
progress_bar: bool | None = None,
**kwargs,
) -> ExitCode:
"""Run OCRmyPDF on one PDF or image.
This function supports two calling conventions:
**New style (recommended):**
>>> from ocrmypdf import ocr
>>> from ocrmypdf._options import OcrOptions
>>> options = OcrOptions(
... input_file="input.pdf",
... output_file="output.pdf",
... languages=["eng"],
... )
>>> ocr(options)
**Old style:**
>>> ocr("input.pdf", "output.pdf", language=["eng"])
For most arguments, see documentation for the equivalent command line parameter.
This API takes a threading lock, because OCRmyPDF uses global state in particular
@@ -337,24 +508,33 @@ def ocr( # noqa: D417
A few specific arguments are discussed here:
Args:
input_file_or_options: Either an OcrOptions object containing all settings,
or a path/stream for the input file (old-style API).
output_file: Output file path or stream. Required when using old-style API
with input_file as first argument. Must be None when passing OcrOptions.
use_threads: Use worker threads instead of processes. This reduces
performance but may make debugging easier since it is easier to set
breakpoints.
input_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
interpreted as file system path to the input file. If the object
appears to be a readable stream (with methods such as ``.read()``
and ``.seek()``), the object will be read in its entirety and saved to
a temporary file. If ``input_file`` is ``"-"``, standard input will be
read.
output_file: If a :class:`pathlib.Path`, ``str`` or ``bytes``, this is
interpreted as file system path to the output file. If the object
appears to be a writable stream (with methods such as ``.write()`` and
``.seek()``), the output will be written to this stream. If
``output_file`` is ``"-"``, the output will be written to ``sys.stdout``
(provided that standard output does not seem to be a terminal device).
When a stream is used as output, whether via a writable object or
``"-"``, some final validation steps are not performed (we do not read
back the stream after it is written).
plugins: List of plugin paths to load. Can be passed alongside OcrOptions.
plugin_manager: Pre-configured plugin manager. Can be passed alongside
OcrOptions.
For input_file (old-style API): If a :class:`pathlib.Path`, ``str`` or
``bytes``, this is interpreted as file system path to the input file.
If the object appears to be a readable stream (with methods such as
``.read()`` and ``.seek()``), the object will be read in its entirety
and saved to a temporary file. If ``input_file`` is ``"-"``, standard
input will be read.
For output_file (old-style API): If a :class:`pathlib.Path`, ``str`` or
``bytes``, this is interpreted as file system path to the output file.
If the object appears to be a writable stream (with methods such as
``.write()`` and ``.seek()``), the output will be written to this
stream. If ``output_file`` is ``"-"``, the output will be written to
``sys.stdout`` (provided that standard output does not seem to be a
terminal device). When a stream is used as output, whether via a
writable object or ``"-"``, some final validation steps are not
performed (we do not read back the stream after it is written).
Raises:
ocrmypdf.MissingDependencyError: If a required dependency program is missing or
@@ -373,45 +553,119 @@ def ocr( # noqa: D417
OCRmyPDF does not remove passwords.
ocrmypdf.TesseractConfigError: If Tesseract reported its configuration was not
valid.
ValueError: If OcrOptions is passed along with other OCR parameters, or if
both plugins and plugin_manager are provided.
TypeError: If output_file is missing when using the old-style API.
Returns:
:class:`ocrmypdf.ExitCode`
"""
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
# Detect calling convention: OcrOptions object vs individual parameters
if isinstance(input_file_or_options, OcrOptions):
# New-style API: OcrOptions passed directly
options = input_file_or_options
# Check for conflicting parameters
# (all should be None except plugins/plugin_manager)
_check_no_conflicting_ocr_params(locals(), kwargs)
# plugins and plugin_manager can still be passed alongside OcrOptions
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
# Use plugins from OcrOptions if not explicitly passed
if plugins is None:
plugins = options.plugins or []
if isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins) if plugins else []
# Run the pipeline with the OcrOptions
with _api_lock:
plugin_manager = setup_plugin_infrastructure(
plugins=plugins, plugin_manager=plugin_manager
)
parser = get_parser()
plugin_manager.add_options(parser=parser)
check_options(options, plugin_manager)
return run_pipeline(options=options, plugin_manager=plugin_manager)
if not plugins:
plugins = []
elif isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins)
# Old-style API: positional arguments
input_file = input_file_or_options
# No new variable names should be assigned until these two steps are run
create_options_kwargs = {
k: v
for k, v in locals().items()
if k not in {'input_file', 'output_file', 'kwargs', 'plugin_manager'}
}
create_options_kwargs.update(kwargs)
if output_file is None:
raise TypeError(
"ocr() missing required argument: 'output_file'. "
"Either pass output_file as the second argument, or pass "
"an OcrOptions object as the first argument."
)
parser = get_parser()
with _api_lock:
if not plugin_manager:
plugin_manager = get_plugin_manager(plugins)
plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
if 'verbose' in kwargs:
warn("ocrmypdf.ocr(verbose=) is ignored. Use ocrmypdf.configure_logging().")
if not plugins:
plugins = []
elif isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins)
options = create_options(
input_file=input_file,
output_file=output_file,
parser=parser,
**create_options_kwargs,
)
check_options(options, plugin_manager)
return run_pipeline(options=options, plugin_manager=plugin_manager)
# No new variable names should be assigned until these two steps are run
create_options_kwargs = {
k: v
for k, v in locals().items()
if k
not in {
'input_file_or_options',
'input_file',
'output_file',
'kwargs',
'plugin_manager',
}
}
create_options_kwargs.update(kwargs)
parser = get_parser()
with _api_lock:
# Set up plugin infrastructure with proper initialization
plugin_manager = setup_plugin_infrastructure(
plugins=plugins, plugin_manager=plugin_manager
)
# Get parser and let plugins add their options
parser = get_parser()
plugin_manager.add_options(parser=parser)
if 'verbose' in kwargs:
warn(
"ocrmypdf.ocr(verbose=) is ignored. "
"Use ocrmypdf.configure_logging()."
)
# Warn about deprecated jbig2 options and remove from kwargs
if jbig2_lossy:
warn(
"jbig2_lossy is deprecated and will be ignored. "
"Lossy JBIG2 has been removed due to character substitution risks."
)
create_options_kwargs.pop('jbig2_lossy', None)
if jbig2_page_group_size:
warn("jbig2_page_group_size is deprecated and will be ignored.")
create_options_kwargs.pop('jbig2_page_group_size', None)
options = create_options(
input_file=input_file,
output_file=output_file,
parser=parser,
**create_options_kwargs,
)
check_options(options, plugin_manager)
return run_pipeline(options=options, plugin_manager=plugin_manager)
def _pdf_to_hocr( # noqa: D417
@@ -434,9 +688,10 @@ def _pdf_to_hocr( # noqa: D417
unpaper_args: str | None = None,
oversample: int | None = None,
remove_vectors: bool | None = None,
force_ocr: bool | None = None,
skip_text: bool | None = None,
redo_ocr: bool | None = None,
mode: str | None = None,
force_ocr: bool | None = None, # Legacy, use mode='force' instead
skip_text: bool | None = None, # Legacy, use mode='skip' instead
redo_ocr: bool | None = None, # Legacy, use mode='redo' instead
skip_big: float | None = None,
pages: str | None = None,
max_image_mpixels: float | None = None,
@@ -449,6 +704,7 @@ def _pdf_to_hocr( # noqa: D417
tesseract_downsample_above: int | None = None,
tesseract_downsample_large_images: bool | None = None,
rotate_pages_threshold: float | None = None,
rasterizer: str | None = None,
user_words: os.PathLike | None = None,
user_patterns: os.PathLike | None = None,
continue_on_soft_render_error: bool | None = None,
@@ -478,33 +734,72 @@ def _pdf_to_hocr( # noqa: D417
output_folder: Output folder path.
**kwargs: Keyword arguments.
"""
# No new variable names should be assigned until these two steps are run
create_options_kwargs = {
k: v
for k, v in locals().items()
if k not in {'input_pdf', 'output_folder', 'kwargs'}
}
create_options_kwargs.update(kwargs)
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
parser = get_parser()
if not plugins:
plugins = []
elif isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins)
# Prepare kwargs for direct OcrOptions construction
options_kwargs = kwargs.copy()
# Set input file and handle special output_folder case
options_kwargs['input_file'] = input_pdf
options_kwargs['output_file'] = '/dev/null' # Placeholder for hOCR pipeline
# Add all the function parameters
for param_name, param_value in locals().items():
if (
param_name
not in {'input_pdf', 'output_folder', 'kwargs', 'plugin_manager', 'plugins'}
and param_value is not None
):
options_kwargs[param_name] = param_value
# Handle plugins
if plugins:
options_kwargs['plugins'] = plugins
# Remove None values to let OcrOptions use its defaults
options_kwargs = {k: v for k, v in options_kwargs.items() if v is not None}
# Add output_folder to options_kwargs since it's now a proper field
options_kwargs['output_folder'] = output_folder
# Remove any kwargs that aren't OcrOptions fields and store in extra_attrs
extra_attrs = {}
ocr_fields = set(OcrOptions.model_fields.keys())
# Legacy mode flags are handled by OcrOptions model validator
legacy_mode_flags = {'force_ocr', 'skip_text', 'redo_ocr'}
known_extra = {'progress_bar', 'plugins'}
for key in list(options_kwargs.keys()):
if key in ocr_fields or key in legacy_mode_flags or key in known_extra:
continue
extra_attrs[key] = options_kwargs.pop(key)
with _api_lock:
if not plugin_manager:
plugin_manager = get_plugin_manager(plugins)
plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
cmdline, deferred = _kwargs_to_cmdline(
defer_kwargs={'input_pdf', 'output_folder', 'plugins'},
**create_options_kwargs,
# Set up plugin infrastructure with proper initialization
plugin_manager = setup_plugin_infrastructure(
plugins=plugins, plugin_manager=plugin_manager
)
cmdline.append(str(input_pdf))
cmdline.append(str(output_folder))
parser.enable_api_mode()
options = parser.parse_args(cmdline)
for keyword, val in deferred.items():
setattr(options, keyword, val)
delattr(options, 'output_file')
setattr(options, 'output_folder', output_folder)
plugin_manager.add_options(parser=get_parser())
# Create OcrOptions directly
try:
options = OcrOptions(**options_kwargs)
# Add any extra attributes
if extra_attrs:
options.extra_attrs.update(extra_attrs)
except Exception as e:
raise TypeError(
f"Failed to create OcrOptions for hOCR pipeline: {e}"
) from e
return run_hocr_pipeline(options=options, plugin_manager=plugin_manager)
@@ -518,8 +813,8 @@ def _hocr_to_ocr_pdf( # noqa: D417
optimize: int | None = None,
jpg_quality: int | None = None,
png_quality: int | None = None,
jbig2_lossy: bool | None = None,
jbig2_page_group_size: int | None = None,
jbig2_lossy: bool | None = None, # Deprecated, ignored
jbig2_page_group_size: int | None = None, # Deprecated, ignored
jbig2_threshold: float | None = None,
pdfa_image_compression: str | None = None,
color_conversion_strategy: str | None = None,
@@ -544,33 +839,83 @@ def _hocr_to_ocr_pdf( # noqa: D417
output_file: Output PDF file path.
**kwargs: Keyword arguments.
"""
# No new variable names should be assigned until these two steps are run
create_options_kwargs = {
k: v
for k, v in locals().items()
if k not in {'work_folder', 'output_pdf', 'kwargs'}
}
create_options_kwargs.update(kwargs)
if plugins and plugin_manager:
raise ValueError("plugins= and plugin_manager are mutually exclusive")
parser = get_parser()
if not plugins:
plugins = []
elif isinstance(plugins, str | Path):
plugins = [plugins]
else:
plugins = list(plugins)
# Prepare kwargs for direct OcrOptions construction
options_kwargs = kwargs.copy()
# Set output file and handle special work_folder case
options_kwargs['input_file'] = '/dev/null' # Placeholder for hOCR to PDF pipeline
options_kwargs['output_file'] = output_file
# Add all the function parameters
for param_name, param_value in locals().items():
if (
param_name
not in {'work_folder', 'output_file', 'kwargs', 'plugin_manager', 'plugins'}
and param_value is not None
):
options_kwargs[param_name] = param_value
# Handle plugins
if plugins:
options_kwargs['plugins'] = plugins
# Remove None values to let OcrOptions use its defaults
options_kwargs = {k: v for k, v in options_kwargs.items() if v is not None}
# Warn about deprecated jbig2 options and remove from kwargs
if jbig2_lossy:
warn(
"jbig2_lossy is deprecated and will be ignored. "
"Lossy JBIG2 has been removed due to character substitution risks."
)
options_kwargs.pop('jbig2_lossy', None)
if jbig2_page_group_size:
warn("jbig2_page_group_size is deprecated and will be ignored.")
options_kwargs.pop('jbig2_page_group_size', None)
# Add work_folder to options_kwargs since it's now a proper field
options_kwargs['work_folder'] = work_folder
# Remove any kwargs that aren't OcrOptions fields and store in extra_attrs
extra_attrs = {}
ocr_fields = set(OcrOptions.model_fields.keys())
# Legacy mode flags are handled by OcrOptions model validator
legacy_mode_flags = {'force_ocr', 'skip_text', 'redo_ocr'}
known_extra = {'progress_bar', 'plugins'}
for key in list(options_kwargs.keys()):
if key in ocr_fields or key in legacy_mode_flags or key in known_extra:
continue
extra_attrs[key] = options_kwargs.pop(key)
with _api_lock:
if not plugin_manager:
plugin_manager = get_plugin_manager(plugins)
plugin_manager.hook.add_options(parser=parser) # pylint: disable=no-member
cmdline, deferred = _kwargs_to_cmdline(
defer_kwargs={'work_folder', 'output_file', 'plugins'},
**create_options_kwargs,
# Set up plugin infrastructure with proper initialization
plugin_manager = setup_plugin_infrastructure(
plugins=plugins, plugin_manager=plugin_manager
)
cmdline.append(str(work_folder))
cmdline.append(str(output_file))
parser.enable_api_mode()
options = parser.parse_args(cmdline)
for keyword, val in deferred.items():
setattr(options, keyword, val)
delattr(options, 'input_file')
setattr(options, 'work_folder', work_folder)
plugin_manager.add_options(parser=get_parser())
# Create OcrOptions directly
try:
options = OcrOptions(**options_kwargs)
# Add any extra attributes
if extra_attrs:
options.extra_attrs.update(extra_attrs)
except Exception as e:
raise TypeError(
f"Failed to create OcrOptions for hOCR to PDF pipeline: {e}"
) from e
return run_hocr_to_ocr_pdf_pipeline(
options=options, plugin_manager=plugin_manager
@@ -588,4 +933,5 @@ __all__ = [
'ocr',
'run_pipeline',
'run_pipeline_cli',
'setup_plugin_infrastructure',
]
src/ocrmypdf/builtin_plugins/concurrency.py
+10 -7
View File
@@ -7,6 +7,7 @@ from __future__ import annotations
import logging
import logging.handlers
import multiprocessing
import multiprocessing.queues
import os
import queue
import signal
@@ -15,7 +16,7 @@ import threading
from collections.abc import Callable, Iterable
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
from contextlib import suppress
from typing import Union
from typing import TYPE_CHECKING
from rich.console import Console as RichConsole
@@ -25,12 +26,14 @@ from ocrmypdf._progressbar import RichProgressBar
from ocrmypdf.exceptions import InputFileError
from ocrmypdf.helpers import remove_all_log_handlers
FuturesExecutorClass = Union[ # noqa: UP007
type[ThreadPoolExecutor], type[ProcessPoolExecutor]
]
Queue = Union[multiprocessing.Queue, queue.Queue] # noqa: UP007
UserInit = Callable[[], None]
WorkerInit = Callable[[Queue, UserInit, int], None]
if TYPE_CHECKING:
from typing import TypeAlias
Queue: TypeAlias = multiprocessing.queues.Queue | queue.Queue
UserInit: TypeAlias = Callable[[], None]
WorkerInit: TypeAlias = Callable[[Queue, UserInit, int], None]
FuturesExecutorClass = type[ThreadPoolExecutor] | type[ProcessPoolExecutor]
def log_listener(q: Queue):
src/ocrmypdf/builtin_plugins/ghostscript.py
+133 -61
View File
@@ -5,13 +5,17 @@
from __future__ import annotations
import logging
from enum import StrEnum
from pathlib import Path
from typing import Annotated
from packaging.version import Version
from pikepdf import Name, Pdf, Stream
from pydantic import BaseModel, Field
from ocrmypdf import hookimpl
from ocrmypdf._exec import ghostscript
from ocrmypdf._options import ProcessingMode
from ocrmypdf.exceptions import MissingDependencyError
from ocrmypdf.subprocess import check_external_program
@@ -22,79 +26,134 @@ log = logging.getLogger(__name__)
BLACKLISTED_GS_VERSIONS: frozenset[Version] = frozenset()
class ColorConversionStrategy(StrEnum):
"""Ghostscript color conversion strategies."""
CMYK = 'CMYK'
GRAY = 'Gray'
LEAVE_COLOR_UNCHANGED = 'LeaveColorUnchanged'
RGB = 'RGB'
USE_DEVICE_INDEPENDENT_COLOR = 'UseDeviceIndependentColor'
class PdfaImageCompression(StrEnum):
"""PDF/A image compression methods."""
AUTO = 'auto'
JPEG = 'jpeg'
LOSSLESS = 'lossless'
class GhostscriptOptions(BaseModel):
"""Options specific to Ghostscript operations."""
color_conversion_strategy: Annotated[
ColorConversionStrategy,
Field(description="Ghostscript color conversion strategy"),
] = ColorConversionStrategy.LEAVE_COLOR_UNCHANGED
pdfa_image_compression: Annotated[
PdfaImageCompression, Field(description="PDF/A image compression method")
] = PdfaImageCompression.AUTO
@classmethod
def add_arguments_to_parser(cls, parser, namespace: str = 'ghostscript'):
"""Add Ghostscript-specific arguments to the argument parser.
Args:
parser: The argument parser to add arguments to
namespace: The namespace prefix for argument names (not used for ghostscript
for backward compatibility)
"""
gs = parser.add_argument_group("Ghostscript", "Advanced control of Ghostscript")
gs.add_argument(
'--color-conversion-strategy',
action='store',
type=str,
choices=[ccs.value for ccs in ColorConversionStrategy],
default=ColorConversionStrategy.LEAVE_COLOR_UNCHANGED.value,
help="Set Ghostscript color conversion strategy",
)
gs.add_argument(
'--pdfa-image-compression',
choices=[pc.value for pc in PdfaImageCompression],
default=PdfaImageCompression.AUTO.value,
help="Specify how to compress images in the output PDF/A. 'auto' lets "
"OCRmyPDF decide. 'jpeg' changes all grayscale and color images to "
"JPEG compression. 'lossless' uses PNG-style lossless compression "
"for all images. Monochrome images are always compressed using a "
"lossless codec. Compression settings "
"are applied to all pages, including those for which OCR was "
"skipped. Not supported for --output-type=pdf ; that setting "
"preserves the original compression of all images.",
)
@hookimpl
def register_options():
"""Register Ghostscript option model."""
return {'ghostscript': GhostscriptOptions}
@hookimpl
def add_options(parser):
gs = parser.add_argument_group("Ghostscript", "Advanced control of Ghostscript")
gs.add_argument(
'--color-conversion-strategy',
action='store',
type=str,
metavar='STRATEGY',
choices=ghostscript.COLOR_CONVERSION_STRATEGIES,
default='LeaveColorUnchanged',
help="Set Ghostscript color conversion strategy",
)
gs.add_argument(
'--pdfa-image-compression',
choices=['auto', 'jpeg', 'lossless'],
default='auto',
help="Specify how to compress images in the output PDF/A. 'auto' lets "
"OCRmyPDF decide. 'jpeg' changes all grayscale and color images to "
"JPEG compression. 'lossless' uses PNG-style lossless compression "
"for all images. Monochrome images are always compressed using a "
"lossless codec. Compression settings "
"are applied to all pages, including those for which OCR was "
"skipped. Not supported for --output-type=pdf ; that setting "
"preserves the original compression of all images.",
)
# Use the model's CLI generation method
GhostscriptOptions.add_arguments_to_parser(parser)
@hookimpl
def check_options(options):
"""Check that the options are valid for this plugin."""
check_external_program(
program='gs',
package='ghostscript',
version_checker=ghostscript.version,
need_version='9.54', # RHEL 9's version; Ubuntu 22.04 has 9.55
)
gs_version = ghostscript.version()
if gs_version in BLACKLISTED_GS_VERSIONS:
raise MissingDependencyError(
f"Ghostscript {gs_version} contains serious regressions and is not "
"supported. Please upgrade to a newer version."
)
if Version('10.0.0') <= gs_version < Version('10.02.1') and (
options.skip_text or options.redo_ocr
):
raise MissingDependencyError(
f"Ghostscript 10.0.0 through 10.02.0 (your version: {gs_version}) "
"contain serious regressions that corrupt PDFs with existing text, "
"such as those processed using --skip-text or --redo-ocr. "
"Please upgrade to a "
"newer version, or use --output-type pdf to avoid Ghostscript, or "
"use --force-ocr to discard existing text."
# Only require Ghostscript for pdfa* output types (not 'auto' or 'pdf')
# 'auto' mode uses best-effort PDF/A without Ghostscript fallback
if options.output_type.startswith('pdfa'):
check_external_program(
program='gs',
package='ghostscript',
version_checker=ghostscript.version,
need_version='9.54', # RHEL 9's version; Ubuntu 22.04 has 9.55
)
gs_version = ghostscript.version()
if gs_version in BLACKLISTED_GS_VERSIONS:
raise MissingDependencyError(
f"Ghostscript {gs_version} contains serious regressions and is not "
"supported. Please upgrade to a newer version."
)
if Version('10.0.0') <= gs_version < Version('10.02.1') and (
options.mode in (ProcessingMode.skip, ProcessingMode.redo)
):
raise MissingDependencyError(
f"Ghostscript 10.0.0 through 10.02.0 (your version: {gs_version}) "
"contain serious regressions that corrupt PDFs with existing text, "
"such as those processed using --skip-text or --redo-ocr "
"(or --mode skip/redo). Please upgrade to a newer version, or use "
"--output-type pdf to avoid Ghostscript, or use --force-ocr "
"(or --mode force) to discard existing text."
)
if gs_version >= Version('10.6.0'):
log.warning(
"Ghostscript 10.6.x contains JPEG encoding errors that may corrupt "
"images. OCRmyPDF will attempt to mitigate, but this version is "
"strongly not recommended. Please upgrade to a newer version. "
"As of 2025-12, 10.6.0 is the latest version of Ghostscript."
)
if options.output_type == 'pdfa':
options.output_type = 'pdfa-2'
if gs_version >= Version('10.6.0') and options.output_type.startswith('pdfa'):
log.warning(
"Ghostscript 10.6.x contains JPEG encoding errors that may corrupt "
"images. OCRmyPDF will attempt to mitigate, but this version is "
"strongly not recommended. Please upgrade to a newer version. "
"As of 2025-12, 10.6.0 is the latest version of Ghostscript."
)
if options.output_type == 'pdfa':
options.output_type = 'pdfa-2'
if options.color_conversion_strategy not in ghostscript.COLOR_CONVERSION_STRATEGIES:
if (
options.ghostscript.color_conversion_strategy
not in ghostscript.COLOR_CONVERSION_STRATEGIES
):
raise ValueError(
f"Invalid color conversion strategy: {options.color_conversion_strategy}"
f"Invalid color conversion strategy: "
f"{options.ghostscript.color_conversion_strategy}"
)
if options.pdfa_image_compression != 'auto' and not options.output_type.startswith(
'pdfa'
if (
options.ghostscript.pdfa_image_compression != 'auto'
and options.output_type not in ('auto', 'pdfa', 'pdfa-1', 'pdfa-2', 'pdfa-3')
):
log.warning(
"--pdfa-image-compression argument only applies when "
"--output-type is one of 'pdfa', 'pdfa-1', or 'pdfa-2'"
"--output-type is 'auto' or one of 'pdfa', 'pdfa-1', 'pdfa-2', 'pdfa-3'"
)
@@ -109,8 +168,15 @@ def rasterize_pdf_page(
rotation,
filter_vector,
stop_on_soft_error,
options,
use_cropbox,
):
"""Rasterize a single page of a PDF file using Ghostscript."""
# Check if user explicitly requested a different rasterizer
if options is not None and options.rasterizer == 'pypdfium':
# Let pypdfium handle it (it will error in check_options if unavailable)
return None
ghostscript.rasterize_pdf(
input_file,
output_file,
@@ -121,6 +187,7 @@ def rasterize_pdf_page(
rotation=rotation,
filter_vector=filter_vector,
stop_on_error=stop_on_soft_error,
use_cropbox=use_cropbox,
)
return output_file
@@ -275,11 +342,16 @@ def generate_pdfa(
stop_on_soft_error,
):
"""Generate a PDF/A from the list of PDF pages and PDF/A metadata."""
# Normalize output_type at point of use
output_type = context.options.output_type
if output_type == 'pdfa':
output_type = 'pdfa-2'
ghostscript.generate_pdfa(
pdf_pages=[pdfmark, *pdf_pages],
output_file=output_file,
compression=context.options.pdfa_image_compression,
color_conversion_strategy=context.options.color_conversion_strategy,
compression=context.options.ghostscript.pdfa_image_compression,
color_conversion_strategy=context.options.ghostscript.color_conversion_strategy,
pdf_version=pdf_version,
pdfa_part=pdfa_part,
progressbar_class=progressbar_class,
src/ocrmypdf/builtin_plugins/null_ocr.py
+159
View File
@@ -0,0 +1,159 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Built-in plugin implementing a null OCR engine (no OCR).
This plugin provides an OCR engine that produces no text output. It is useful
when users want OCRmyPDF's image processing, PDF/A conversion, or optimization
features without performing actual OCR.
Usage:
ocrmypdf --ocr-engine none input.pdf output.pdf
"""
from __future__ import annotations
from pathlib import Path
from typing import TYPE_CHECKING
from PIL import Image
from ocrmypdf import hookimpl
from ocrmypdf.hocrtransform import BoundingBox, OcrClass, OcrElement
from ocrmypdf.pluginspec import OcrEngine, OrientationConfidence
if TYPE_CHECKING:
from ocrmypdf._options import OcrOptions
class NullOcrEngine(OcrEngine):
"""A no-op OCR engine that produces no text output.
Use this when you want OCRmyPDF's image processing, PDF/A conversion,
or optimization features without performing actual OCR.
"""
@staticmethod
def version() -> str:
"""Return version string."""
return "none"
@staticmethod
def creator_tag(options: OcrOptions) -> str:
"""Return creator tag for PDF metadata."""
return "OCRmyPDF (no OCR)"
def __str__(self) -> str:
"""Return human-readable engine name."""
return "No OCR engine"
@staticmethod
def languages(options: OcrOptions) -> set[str]:
"""Return supported languages (empty set for null engine)."""
return set()
@staticmethod
def get_orientation(input_file: Path, options: OcrOptions) -> OrientationConfidence:
"""Return neutral orientation (no rotation detected)."""
return OrientationConfidence(angle=0, confidence=0.0)
@staticmethod
def get_deskew(input_file: Path, options: OcrOptions) -> float:
"""Return zero deskew angle."""
return 0.0
@staticmethod
def supports_generate_ocr() -> bool:
"""Return True - this engine supports the generate_ocr() API."""
return True
@staticmethod
def generate_ocr(
input_file: Path,
options: OcrOptions,
page_number: int = 0,
) -> tuple[OcrElement, str]:
"""Generate empty OCR results.
Args:
input_file: The image file (used to get dimensions).
options: OCR options (ignored).
page_number: Page number (stored in result).
Returns:
A tuple of (empty OcrElement page, empty string).
"""
# Get image dimensions
with Image.open(input_file) as img:
width, height = img.size
dpi_info = img.info.get('dpi', (72, 72))
dpi = dpi_info[0] if isinstance(dpi_info, tuple) else dpi_info
# Create empty page element with correct dimensions
page = OcrElement(
ocr_class=OcrClass.PAGE,
bbox=BoundingBox(left=0, top=0, right=width, bottom=height),
dpi=float(dpi),
page_number=page_number,
)
return page, ""
@staticmethod
def generate_hocr(
input_file: Path,
output_hocr: Path,
output_text: Path,
options: OcrOptions,
) -> None:
"""Generate empty hOCR file.
Creates minimal valid hOCR output with no text content.
"""
# Get image dimensions for hOCR bbox
with Image.open(input_file) as img:
width, height = img.size
hocr_content = f'''<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>OCRmyPDF - No OCR</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<meta name='ocr-system' content='OCRmyPDF null engine'/>
</head>
<body>
<div class='ocr_page' title='bbox 0 0 {width} {height}'>
</div>
</body>
</html>
'''
output_hocr.write_text(hocr_content, encoding='utf-8')
output_text.write_text('', encoding='utf-8')
@staticmethod
def generate_pdf(
input_file: Path,
output_pdf: Path,
output_text: Path,
options: OcrOptions,
) -> None:
"""NullOcrEngine cannot generate PDFs directly.
Use pdf_renderer='fpdf2' instead of 'sandwich'.
"""
raise NotImplementedError(
"NullOcrEngine cannot generate PDFs directly. "
"Use --pdf-renderer fpdf2 instead of sandwich mode."
)
@hookimpl
def get_ocr_engine(options):
"""Return NullOcrEngine when --ocr-engine none is selected."""
if options is not None:
ocr_engine = getattr(options, 'ocr_engine', 'auto')
if ocr_engine != 'none':
return None
return NullOcrEngine()
src/ocrmypdf/builtin_plugins/optimize.py
+158 -85
View File
@@ -8,6 +8,9 @@ import argparse
import logging
from collections.abc import Sequence
from pathlib import Path
from typing import Annotated
from pydantic import BaseModel, Field, model_validator
from ocrmypdf import Executor, PdfContext, hookimpl
from ocrmypdf._exec import jbig2enc, pngquant
@@ -19,87 +22,165 @@ from ocrmypdf.subprocess import check_external_program
log = logging.getLogger(__name__)
class OptimizeOptions(BaseModel):
"""Options specific to PDF optimization."""
level: Annotated[
int,
Field(
ge=0,
le=3,
description="Optimization level (0=none, 1=safe, 2=lossy, 3=aggressive)",
),
] = 1
jpeg_quality: Annotated[
int, Field(ge=0, le=100, description="JPEG quality level for optimization")
] = 0
png_quality: Annotated[
int, Field(ge=0, le=100, description="PNG quality level for optimization")
] = 0
jbig2_threshold: Annotated[
float,
Field(ge=0.4, le=0.9, description="JBIG2 symbol classification threshold"),
] = 0.85
@classmethod
def add_arguments_to_parser(cls, parser, namespace: str = 'optimize'):
"""Add optimization-specific arguments to the argument parser.
Args:
parser: The argument parser to add arguments to
namespace: The namespace prefix for argument names
(not used for optimize for backward compatibility)
"""
optimizing = parser.add_argument_group(
"Optimization options", "Control how the PDF is optimized after OCR"
)
optimizing.add_argument(
'-O',
'--optimize',
type=int,
choices=range(0, 4),
default=1,
help=(
"Control how PDF is optimized after processing:"
"0 - do not optimize; "
"1 - do safe, lossless optimizations (default); "
"2 - do lossy JPEG and JPEG2000 optimizations; "
"3 - do more aggressive lossy JPEG and JPEG2000 optimizations. "
"To enable lossy JBIG2, see --jbig2-lossy."
),
)
optimizing.add_argument(
'--jpeg-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
help=(
"Adjust JPEG quality level for JPEG optimization. "
"100 is best quality and largest output size; "
"1 is lowest quality and smallest output; "
"0 uses the default."
),
)
optimizing.add_argument(
'--jpg-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
dest='jpeg_quality',
help=argparse.SUPPRESS, # Alias for --jpeg-quality
)
optimizing.add_argument(
'--png-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
help=(
"Adjust PNG quality level to use when quantizing PNGs. "
"Values have same meaning as with --jpeg-quality"
),
)
# Deprecated arguments - kept for backward compatibility, emit warnings
optimizing.add_argument(
'--jbig2-lossy',
action='store_true',
help=argparse.SUPPRESS, # Deprecated, hidden from help
)
optimizing.add_argument(
'--jbig2-page-group-size',
type=numeric(int, 1, 10000),
default=0,
metavar='N',
help=argparse.SUPPRESS, # Deprecated, hidden from help
)
optimizing.add_argument(
'--jbig2-threshold',
type=numeric(float, 0.4, 0.9),
default=0.85,
metavar='T',
help=(
"Adjust JBIG2 symbol code classification threshold "
"(default 0.85), range 0.4 to 0.9."
),
)
@model_validator(mode='after')
def validate_optimization_consistency(self):
"""Validate optimization options are consistent."""
if self.level == 0 and any([self.png_quality > 0, self.jpeg_quality > 0]):
log.warning(
"The arguments --png-quality and --jpeg-quality "
"will be ignored because --optimize=0."
)
return self
def validate_with_context(
self, external_programs_available: dict[str, bool]
) -> None:
"""Validate options that require external context.
Args:
external_programs_available: Dict of program name -> availability
"""
if self.level >= 2:
if not external_programs_available.get('pngquant', False):
log.warning(
"pngquant is not available, so PNG optimization will be limited"
)
if not external_programs_available.get('jbig2enc', False):
log.warning(
"jbig2enc is not available, so JBIG2 optimization will be limited"
)
@hookimpl
def register_options():
"""Register optimization option model."""
return {'optimize': OptimizeOptions}
@hookimpl
def add_options(parser):
optimizing = parser.add_argument_group(
"Optimization options", "Control how the PDF is optimized after OCR"
)
optimizing.add_argument(
'-O',
'--optimize',
type=int,
choices=range(0, 4),
default=1,
help=(
"Control how PDF is optimized after processing:"
"0 - do not optimize; "
"1 - do safe, lossless optimizations (default); "
"2 - do lossy JPEG and JPEG2000 optimizations; "
"3 - do more aggressive lossy JPEG and JPEG2000 optimizations. "
"To enable lossy JBIG2, see --jbig2-lossy."
),
)
optimizing.add_argument(
'--jpeg-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
help=(
"Adjust JPEG quality level for JPEG optimization. "
"100 is best quality and largest output size; "
"1 is lowest quality and smallest output; "
"0 uses the default."
),
)
optimizing.add_argument(
'--jpg-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
dest='jpeg_quality',
help=argparse.SUPPRESS, # Alias for --jpeg-quality
)
optimizing.add_argument(
'--png-quality',
type=numeric(int, 0, 100),
default=0,
metavar='Q',
help=(
"Adjust PNG quality level to use when quantizing PNGs. "
"Values have same meaning as with --jpeg-quality"
),
)
optimizing.add_argument(
'--jbig2-lossy',
action='store_true',
help=(
"Enable JBIG2 lossy mode (better compression, not suitable for some "
"use cases - see documentation). Only takes effect if --optimize 1 or "
"higher is also enabled."
),
)
optimizing.add_argument(
'--jbig2-page-group-size',
type=numeric(int, 1, 10000),
default=0,
metavar='N',
# Adjust number of pages to consider at once for JBIG2 compression
help=argparse.SUPPRESS,
)
optimizing.add_argument(
'--jbig2-threshold',
type=numeric(float, 0.4, 0.9),
default=0.85,
metavar='T',
help=(
"Adjust JBIG2 symbol code classification threshold "
"(default 0.85), range 0.4 to 0.9."
),
)
# Use the model's CLI generation method
OptimizeOptions.add_arguments_to_parser(parser)
@hookimpl
def check_options(options):
"""Check external dependencies for optimization."""
# Warn about deprecated options
if getattr(options, 'jbig2_lossy', False):
log.warning(
"The --jbig2-lossy option is deprecated and will be ignored. "
"Lossy JBIG2 compression has been removed due to risks of "
"character substitution errors."
)
if getattr(options, 'jbig2_page_group_size', 0) not in (0, None):
log.warning(
"The --jbig2-page-group-size option is deprecated and will be ignored."
)
if options.optimize >= 2:
check_external_program(
program='pngquant',
@@ -117,16 +198,8 @@ def check_options(options):
package='jbig2enc',
version_checker=jbig2enc.version,
need_version='0.28',
required_for='--optimize {2,3} | --jbig2-lossy',
recommended=True if not options.jbig2_lossy else False,
)
if options.optimize == 0 and any(
[options.jbig2_lossy, options.png_quality, options.jpeg_quality]
):
log.warning(
"The arguments --jbig2-lossy, --png-quality, and --jpeg-quality "
"will be ignored because --optimize=0."
required_for='--optimize {2,3}',
recommended=True,
)
src/ocrmypdf/builtin_plugins/pypdfium.py
+256
View File
@@ -0,0 +1,256 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Built-in plugin to implement PDF page rasterization using pypdfium2."""
from __future__ import annotations
import logging
import threading
from contextlib import closing
from pathlib import Path
try:
import pypdfium2 as pdfium
except ImportError:
pdfium = None
from PIL import Image
from ocrmypdf import hookimpl
from ocrmypdf.exceptions import MissingDependencyError
from ocrmypdf.helpers import Resolution
log = logging.getLogger(__name__)
# pypdfium2/PDFium is not thread-safe. All calls to the library must be serialized.
# See: https://pypdfium2.readthedocs.io/en/stable/python_api.html#incompatibility-with-threading
# When using process-based parallelism (use_threads=False), each process has its own
# pdfium instance, so locking is not needed across processes.
_pdfium_lock = threading.Lock()
@hookimpl
def check_options(options):
"""Check that pypdfium2 is available if explicitly requested."""
if options.rasterizer == 'pypdfium' and pdfium is None:
raise MissingDependencyError(
"The --rasterizer pypdfium option requires the pypdfium2 package. "
"Install it with: pip install pypdfium2"
)
def _open_pdf_document(input_file: Path):
"""Open a PDF document using pypdfium2."""
assert pdfium is not None, "pypdfium2 must be available to call this function"
return pdfium.PdfDocument(input_file)
def _calculate_mediabox_crop(page) -> tuple[float, float, float, float]:
"""Calculate crop values to expand rendering from CropBox to MediaBox.
By default pypdfium2 renders to the CropBox. To render the full MediaBox,
we need negative crop values to expand the rendering area.
Returns:
Tuple of (left, bottom, right, top) crop values. Negative values
expand the rendering area beyond the CropBox to the MediaBox.
"""
mediabox = page.get_mediabox() # (left, bottom, right, top)
cropbox = page.get_cropbox() # (left, bottom, right, top), defaults to mediabox
# Calculate how much to expand from cropbox to mediabox
# Negative values = expand, positive = shrink
return (
mediabox[0] - cropbox[0], # Expand left
mediabox[1] - cropbox[1], # Expand bottom
cropbox[2] - mediabox[2], # Expand right
cropbox[3] - mediabox[3], # Expand top
)
def _render_page_to_bitmap(
page,
raster_device: str,
raster_dpi: Resolution,
rotation: int | None,
use_cropbox: bool,
):
"""Render a PDF page to a bitmap."""
# Round DPI to match Ghostscript's precision
raster_dpi = raster_dpi.round(6)
# Get page dimensions BEFORE applying rotation
page_width_pts, page_height_pts = page.get_size()
# Calculate expected output dimensions using separate x/y DPI
expected_width = int(round(page_width_pts * raster_dpi.x / 72.0))
expected_height = int(round(page_height_pts * raster_dpi.y / 72.0))
# Calculate the scale factor based on DPI
# pypdfium2 uses points (72 DPI) as base unit
scale = raster_dpi.to_scalar() / 72.0
# Apply rotation if specified
if rotation:
# pypdfium2 rotation is in degrees, same as our input
# we track rotation in CCW, and pypdfium2 expects CW, so negate
page.set_rotation(-rotation % 360)
# When rotation is 90 or 270, dimensions are swapped in output
if rotation % 180 == 90:
expected_width, expected_height = expected_height, expected_width
# Render the page to a bitmap
# The scale parameter controls the resolution
grayscale = raster_device.lower() in ('pnggray', 'jpeggray')
# Calculate crop to render the appropriate box
# Default (use_cropbox=False) renders MediaBox for consistency with Ghostscript
crop = (0, 0, 0, 0) if use_cropbox else _calculate_mediabox_crop(page)
bitmap = page.render(
scale=scale,
rotation=0, # We already set rotation on the page
crop=crop,
may_draw_forms=True,
draw_annots=True,
grayscale=grayscale,
# Note: pypdfium2 doesn't have a direct equivalent to filter_vector
# This would require more complex implementation if needed
)
return bitmap, expected_width, expected_height
def _process_image_for_output(
pil_image,
raster_device: str,
raster_dpi: Resolution,
page_dpi: Resolution | None,
stop_on_soft_error: bool,
expected_width: int | None = None,
expected_height: int | None = None,
):
"""Process PIL image for output format and set DPI metadata."""
# Correct dimensions if slightly off (within 2 pixels tolerance)
if expected_width and expected_height:
actual_width, actual_height = pil_image.width, pil_image.height
width_diff = abs(actual_width - expected_width)
height_diff = abs(actual_height - expected_height)
# Only resize if off by small amount (1-2 pixels)
if (width_diff <= 2 or height_diff <= 2) and (
width_diff > 0 or height_diff > 0
):
log.debug(
f"Adjusting rendered dimensions from "
f"{actual_width}x{actual_height} to expected "
f"{expected_width}x{expected_height}"
)
pil_image = pil_image.resize(
(expected_width, expected_height),
Image.Resampling.LANCZOS
)
# Set the DPI metadata if page_dpi is specified
if page_dpi:
# PIL expects DPI as a tuple
dpi_tuple = (float(page_dpi.x), float(page_dpi.y))
pil_image.info['dpi'] = dpi_tuple
else:
# Use the raster DPI
dpi_tuple = (float(raster_dpi.x), float(raster_dpi.y))
pil_image.info['dpi'] = dpi_tuple
# Determine output format based on raster_device
if raster_device.lower() in ('png', 'pngmono', 'pnggray', 'png16m', 'pngalpha'):
format_name = 'PNG'
elif raster_device.lower() in ('jpeg', 'jpeggray', 'jpg'):
format_name = 'JPEG'
# Convert RGBA to RGB for JPEG
if pil_image.mode == 'RGBA':
# Create white background
background = pil_image.new('RGB', pil_image.size, (255, 255, 255))
background.paste(
pil_image, mask=pil_image.split()[-1]
) # Use alpha channel as mask
pil_image = background
elif raster_device.lower() in ('tiff', 'tif'):
format_name = 'TIFF'
else:
# Default to PNG for unknown formats
format_name = 'PNG'
if stop_on_soft_error:
raise ValueError(f"Unsupported raster device: {raster_device}")
else:
log.warning(f"Unsupported raster device {raster_device}, using PNG")
return pil_image, format_name
def _save_image(pil_image, output_file: Path, format_name: str):
"""Save PIL image to file with appropriate DPI metadata."""
save_kwargs = {}
if (
format_name in ('PNG', 'TIFF')
and 'dpi' in pil_image.info
or format_name == 'JPEG'
and 'dpi' in pil_image.info
):
save_kwargs['dpi'] = pil_image.info['dpi']
pil_image.save(output_file, format=format_name, **save_kwargs)
@hookimpl
def rasterize_pdf_page(
input_file: Path,
output_file: Path,
raster_device: str,
raster_dpi: Resolution,
pageno: int,
page_dpi: Resolution | None,
rotation: int | None,
filter_vector: bool,
stop_on_soft_error: bool,
options,
use_cropbox: bool,
) -> Path | None:
"""Rasterize a single page of a PDF file using pypdfium2.
Returns None if pypdfium2 is not available or if the user has selected
a different rasterizer, allowing Ghostscript to be used.
"""
# Check if user explicitly requested a different rasterizer
if options is not None and options.rasterizer == 'ghostscript':
return None # Let Ghostscript handle it
if pdfium is None:
return None # Fall back to Ghostscript
# Acquire lock to ensure thread-safe access to pypdfium2
with (
_pdfium_lock,
closing(_open_pdf_document(input_file)) as pdf,
closing(pdf[pageno - 1]) as page,
):
# Render the page to a bitmap
bitmap, expected_width, expected_height = _render_page_to_bitmap(
page, raster_device, raster_dpi, rotation, use_cropbox
)
with closing(bitmap):
# Convert to PIL Image
pil_image = bitmap.to_pil()
# Process and save image outside the lock (PIL operations are thread-safe)
pil_image, format_name = _process_image_for_output(
pil_image,
raster_device,
raster_dpi,
page_dpi,
stop_on_soft_error,
expected_width,
expected_height,
)
_save_image(pil_image, output_file, format_name)
return output_file
src/ocrmypdf/builtin_plugins/tesseract_ocr.py
+341 -161
View File
@@ -7,15 +7,18 @@ from __future__ import annotations
import argparse
import logging
import os
from typing import Annotated
from PIL import Image
from pydantic import BaseModel, Field, field_validator, model_validator
from ocrmypdf import hookimpl
from ocrmypdf._exec import tesseract
from ocrmypdf._exec.tesseract import ThresholdingMethod
from ocrmypdf._jobcontext import PageContext
from ocrmypdf.cli import numeric, str_to_int
from ocrmypdf.cli import numeric
from ocrmypdf.exceptions import BadArgsError, MissingDependencyError
from ocrmypdf.helpers import clamp
from ocrmypdf.helpers import available_cpu_count, clamp
from ocrmypdf.imageops import calculate_downsample, downsample_image
from ocrmypdf.pluginspec import OcrEngine
from ocrmypdf.subprocess import check_external_program
@@ -23,121 +26,286 @@ from ocrmypdf.subprocess import check_external_program
log = logging.getLogger(__name__)
def _thresholding_method_converter(value: str) -> ThresholdingMethod:
"""Convert string argument to ThresholdingMethod enum.
Args:
value: String name of thresholding method (auto, otsu, adaptive-otsu, sauvola)
Returns:
ThresholdingMethod enum value
Raises:
argparse.ArgumentTypeError: If value is not a valid thresholding method
"""
method_map = {
'auto': ThresholdingMethod.AUTO,
'otsu': ThresholdingMethod.OTSU,
'adaptive-otsu': ThresholdingMethod.ADAPTIVE_OTSU,
'sauvola': ThresholdingMethod.SAUVOLA,
}
if value.lower() not in method_map:
import argparse
valid = ', '.join(method_map.keys())
raise argparse.ArgumentTypeError(
f"Invalid thresholding method '{value}'. Must be one of: {valid}"
)
return method_map[value.lower()]
class TesseractOptions(BaseModel):
"""Options specific to Tesseract OCR engine."""
config: Annotated[
list[str], Field(description="Additional Tesseract configuration files")
] = []
pagesegmode: Annotated[
int | None,
Field(ge=0, le=13, description="Set Tesseract page segmentation mode"),
] = None
oem: Annotated[
int | None, Field(ge=0, le=3, description="Set Tesseract OCR engine mode")
] = None
thresholding: Annotated[
ThresholdingMethod,
Field(description="Set Tesseract input image thresholding mode"),
] = ThresholdingMethod.AUTO
timeout: Annotated[
float, Field(ge=0, description="Timeout for OCR operations in seconds")
] = 180.0
non_ocr_timeout: Annotated[
float, Field(ge=0, description="Timeout for non-OCR operations in seconds")
] = 180.0
downsample_large_images: Annotated[
bool, Field(description="Downsample large images before OCR")
] = True
downsample_above: Annotated[
int,
Field(
ge=100,
le=32767,
description="Downsample images larger than this pixel size",
),
] = 32767
user_words: Annotated[
str | None, Field(description="Path to Tesseract user words file")
] = None
user_patterns: Annotated[
str | None, Field(description="Path to Tesseract user patterns file")
] = None
omp_thread_limit: Annotated[
int | None,
Field(
description="Calculated OMP_THREAD_LIMIT for Tesseract subprocesses",
exclude=True,
),
] = None
@classmethod
def add_arguments_to_parser(cls, parser, namespace: str = 'tesseract'):
"""Add Tesseract-specific arguments to the argument parser.
Args:
parser: The argument parser to add arguments to
namespace: The namespace prefix for argument names
"""
tess = parser.add_argument_group(
"Tesseract", "Advanced control of Tesseract OCR"
)
tess.add_argument(
f'--{namespace}-config',
action='append',
metavar='CFG',
default=[],
dest=f'{namespace}_config',
help="Additional Tesseract configuration files -- see documentation.",
)
tess.add_argument(
f'--{namespace}-pagesegmode',
action='store',
type=int,
metavar='PSM',
choices=range(0, 14),
dest=f'{namespace}_pagesegmode',
help="Set Tesseract page segmentation mode (see tesseract --help).",
)
tess.add_argument(
f'--{namespace}-oem',
action='store',
type=int,
metavar='MODE',
choices=range(0, 4),
dest=f'{namespace}_oem',
help=(
"Set Tesseract 4+ OCR engine mode: "
"0 - original Tesseract only; "
"1 - neural nets LSTM only; "
"2 - Tesseract + LSTM; "
"3 - default."
),
)
tess.add_argument(
f'--{namespace}-thresholding',
action='store',
type=_thresholding_method_converter,
default='auto',
dest=f'{namespace}_thresholding',
help=(
"Set Tesseract 5.0+ input image thresholding mode. This may improve "
"OCR results on low quality images or those that contain high "
"contrast color. Options: auto, otsu, adaptive-otsu, sauvola. "
"auto/otsu is the Tesseract default (legacy Otsu); adaptive-otsu "
"is an improved Otsu algorithm with improved sort for background "
"color changes; sauvola is based on local standard deviation."
),
)
tess.add_argument(
f'--{namespace}-timeout',
default=180.0,
type=numeric(float, 0),
metavar='SECONDS',
dest=f'{namespace}_timeout',
help=(
"Give up on OCR after the timeout, but copy the preprocessed page "
"into the final output. This timeout is only used when using Tesseract "
"for OCR. When Tesseract is used for other operations such as "
"deskewing and orientation, the timeout is controlled by "
f"--{namespace}-non-ocr-timeout."
),
)
tess.add_argument(
f'--{namespace}-non-ocr-timeout',
default=180.0,
type=numeric(float, 0),
metavar='SECONDS',
dest=f'{namespace}_non_ocr_timeout',
help=(
"Give up on non-OCR operations such as deskewing and orientation "
f"after timeout. This is a separate timeout from --{namespace}-timeout "
"because these operations are not as expensive as OCR."
),
)
tess.add_argument(
f'--{namespace}-downsample-large-images',
action=argparse.BooleanOptionalAction,
default=True,
dest=f'{namespace}_downsample_large_images',
help=(
"Downsample large images before OCR. Tesseract has "
"an upper limit on the size images it will support."
" If this argument is given, OCRmyPDF will "
"downsample large images to fit Tesseract. This "
"may reduce OCR quality, on large images the most"
" desirable text is usually larger. If this "
"parameter is not supplied, Tesseract will error "
"out and produce no OCR on the page in question. "
"This argument should be used with a high value "
f"of --{namespace}-timeout to ensure Tesseract "
"has enough to time."
),
)
tess.add_argument(
f'--{namespace}-downsample-above',
action='store',
type=numeric(int, 100, 32767),
default=32767,
dest=f'{namespace}_downsample_above',
help=(
"Downsample images larger than this size pixel size (either dimension) "
f"before OCR. --{namespace}-downsample-large-images downsamples when "
"an image exceeds Tesseract's internal limits. This argument causes "
"downsampling to occur when an image exceeds the given size. This may "
"reduce OCR quality, but on large images the most desirable text is "
"usually larger."
),
)
tess.add_argument(
'--user-words',
metavar='FILE',
dest='user_words',
help="Specify the location of the Tesseract user words file. This is a "
"list of words Tesseract should consider while performing OCR in "
"addition to its standard language dictionaries. This can improve "
"OCR quality especially for specialized and technical documents.",
)
tess.add_argument(
'--user-patterns',
metavar='FILE',
dest='user_patterns',
help="Specify the location of the Tesseract user patterns file.",
)
@field_validator('timeout', 'non_ocr_timeout')
@classmethod
def validate_timeout_reasonable(cls, v):
"""Validate timeout values are reasonable."""
if v > 3600: # 1 hour
log.warning(f"Timeout of {v} seconds is very long and may cause issues")
return v
@field_validator('pagesegmode')
@classmethod
def validate_pagesegmode_warning(cls, v):
"""Validate page segmentation mode and warn about problematic values."""
if v in (0, 2):
log.warning(
"The tesseract-pagesegmode you selected will disable OCR. "
"This may cause processing to fail."
)
return v
@model_validator(mode='after')
def validate_downsample_consistency(self):
"""Validate downsample options are consistent."""
if self.downsample_above != 32767 and not self.downsample_large_images:
log.warning(
"The --tesseract-downsample-above argument will have no effect unless "
"--tesseract-downsample-large-images is also given."
)
return self
def validate_with_context(self, languages: list[str]) -> None:
"""Validate options that require external context.
Args:
languages: List of languages being used for OCR
"""
# Validate languages are not internal Tesseract languages
DENIED_LANGUAGES = {'equ', 'osd'}
if DENIED_LANGUAGES & set(languages):
raise BadArgsError(
"The following languages are for Tesseract's internal use "
"and should not be issued explicitly: "
f"{', '.join(DENIED_LANGUAGES & set(languages))}\n"
"Remove them from the -l/--language argument."
)
@hookimpl
def register_options():
"""Register Tesseract option model."""
return {'tesseract': TesseractOptions}
@hookimpl
def add_options(parser):
tess = parser.add_argument_group("Tesseract", "Advanced control of Tesseract OCR")
tess.add_argument(
'--tesseract-config',
action='append',
metavar='CFG',
default=[],
help="Additional Tesseract configuration files -- see documentation.",
)
tess.add_argument(
'--tesseract-pagesegmode',
action='store',
type=int,
metavar='PSM',
choices=range(0, 14),
help="Set Tesseract page segmentation mode (see tesseract --help).",
)
tess.add_argument(
'--tesseract-oem',
action='store',
type=int,
metavar='MODE',
choices=range(0, 4),
help=(
"Set Tesseract 4+ OCR engine mode: "
"0 - original Tesseract only; "
"1 - neural nets LSTM only; "
"2 - Tesseract + LSTM; "
"3 - default."
),
)
tess.add_argument(
'--tesseract-thresholding',
action='store',
type=str_to_int(tesseract.TESSERACT_THRESHOLDING_METHODS),
default='auto',
metavar='METHOD',
help=(
"Set Tesseract 5.0+ input image thresholding mode. This may improve OCR "
"results on low quality images or those that contain high contrast color. "
"legacy-otsu is the Tesseract default; adaptive-otsu is an improved Otsu "
"algorithm with improved sort for background color changes; sauvola is "
"based on local standard deviation."
),
)
tess.add_argument(
'--tesseract-timeout',
default=180.0,
type=numeric(float, 0),
metavar='SECONDS',
help=(
"Give up on OCR after the timeout, but copy the preprocessed page "
"into the final output. This timeout is only used when using Tesseract "
"for OCR. When Tesseract is used for other operations such as "
"deskewing and orientation, the timeout is controlled by "
"--tesseract-non-ocr-timeout."
),
)
tess.add_argument(
'--tesseract-non-ocr-timeout',
default=180.0,
type=numeric(float, 0),
metavar='SECONDS',
help=(
"Give up on non-OCR operations such as deskewing and orientation "
"after timeout. This is a separate timeout from --tesseract-timeout "
"because these operations are not as expensive as OCR."
),
)
tess.add_argument(
'--tesseract-downsample-large-images',
action=argparse.BooleanOptionalAction,
default=True,
help=(
"Downsample large images before OCR. Tesseract has an upper limit on the "
"size images it will support. If this argument is given, OCRmyPDF will "
"downsample large images to fit Tesseract. This may reduce OCR quality, "
"on large images the most desirable text is usually larger. If this "
"parameter is not supplied, Tesseract will error out and produce no OCR "
"on the page in question. This argument should be used with a high value "
"of --tesseract-timeout to ensure Tesseract has enough to time."
),
)
tess.add_argument(
'--tesseract-downsample-above',
action='store',
type=numeric(int, 100, 32767),
default=32767,
help=(
"Downsample images larger than this size pixel size in either dimension "
"before OCR. --tesseract-downsample-large-images downsamples only when "
"an image exceeds Tesseract's internal limits. This argument causes "
"downsampling to occur when an image exceeds the given size. This may "
"reduce OCR quality, but on large images the most desirable text is "
"usually larger."
),
)
tess.add_argument(
'--user-words',
metavar='FILE',
help="Specify the location of the Tesseract user words file. This is a "
"list of words Tesseract should consider while performing OCR in "
"addition to its standard language dictionaries. This can improve "
"OCR quality especially for specialized and technical documents.",
)
tess.add_argument(
'--user-patterns',
metavar='FILE',
help="Specify the location of the Tesseract user patterns file.",
)
# Use the model's CLI generation method - it now handles all Tesseract options
TesseractOptions.add_arguments_to_parser(parser)
@hookimpl
def check_options(options):
"""Check external dependencies and version compatibility for Tesseract."""
check_external_program(
program='tesseract',
package={'linux': 'tesseract-ocr'},
@@ -152,33 +320,16 @@ def check_options(options):
"Please upgrade to a newer or supported older version."
)
# Decide on what renderer to use
if options.pdf_renderer == 'auto':
if {'ara', 'heb', 'fas', 'per'} & set(options.languages):
log.info("Using sandwich renderer since there is an RTL language")
options.pdf_renderer = 'sandwich'
else:
options.pdf_renderer = 'hocr'
if not tesseract.has_thresholding() and options.tesseract_thresholding != 0:
# Check version-specific feature compatibility
if (
not tesseract.has_thresholding()
and options.tesseract.thresholding != ThresholdingMethod.AUTO
):
log.warning(
"The installed version of Tesseract does not support changes to its "
"thresholding method. The --tesseract-threshold argument will be "
"ignored."
)
if options.tesseract_pagesegmode in (0, 2):
log.warning(
"The --tesseract-pagesegmode argument you select will disable OCR. "
"This may cause processing to fail."
)
DENIED_LANGUAGES = {'equ', 'osd'}
if DENIED_LANGUAGES & set(options.languages):
raise BadArgsError(
"The following languages for Tesseract's internal use and should not "
"be issued explicitly: "
f"{', '.join(DENIED_LANGUAGES & set(options.languages))}\n"
"Remove them from the -l/--language argument."
)
@hookimpl
@@ -192,15 +343,17 @@ def validate(pdfinfo, options):
# constraint: (ocrmypdf workers) * (tesseract threads) <= max_workers.
# As of Tesseract 4.1, 3 threads is the most effective on a 4 core/8 thread system.
if not os.environ.get('OMP_THREAD_LIMIT', '').isnumeric():
tess_threads = clamp(options.jobs // len(pdfinfo), 1, 3)
os.environ['OMP_THREAD_LIMIT'] = str(tess_threads)
jobs = options.jobs or available_cpu_count()
tess_threads = clamp(jobs // len(pdfinfo), 1, 3)
else:
tess_threads = int(os.environ['OMP_THREAD_LIMIT'])
# Store the thread limit in options - it will be passed to subprocess env
options.tesseract.omp_thread_limit = tess_threads
log.debug("Using Tesseract OpenMP thread limit %d", tess_threads)
if (
options.tesseract_downsample_above != 32767
and not options.tesseract_downsample_large_images
options.tesseract.downsample_above != 32767
and not options.tesseract.downsample_large_images
):
log.warning(
"The --tesseract-downsample-above argument will have no effect unless "
@@ -216,10 +369,12 @@ def filter_ocr_image(page: PageContext, image: Image.Image) -> Image.Image:
or more than 2**31 bytes. This function resizes the image to fit within
those limits.
"""
threshold = min(page.options.tesseract_downsample_above, 32767)
options = page.options
if options.tesseract_downsample_large_images:
if getattr(options, 'tesseract', None) is None:
return image
threshold = min(options.tesseract.downsample_above, 32767)
if options.tesseract.downsample_large_images:
size = calculate_downsample(
image, max_size=(threshold, threshold), max_bytes=(2**31) - 1
)
@@ -234,10 +389,25 @@ class TesseractOcrEngine(OcrEngine):
def version():
return str(tesseract.version())
@staticmethod
def _determine_renderer(options):
"""Determine the PDF renderer to use based on options and languages."""
if options.pdf_renderer == 'auto':
return 'fpdf2'
return options.pdf_renderer
@staticmethod
def creator_tag(options):
tag = '-PDF' if options.pdf_renderer == 'sandwich' else '-hOCR'
return f"Tesseract OCR{tag} {TesseractOcrEngine.version()}"
renderer = TesseractOcrEngine._determine_renderer(options)
match renderer:
case 'hocr':
return f"OCRmyPDF hOCR + Tesseract OCR {TesseractOcrEngine.version()}"
case 'fpdf2':
return f"OCRmyPDF fpdf2 + Tesseract OCR {TesseractOcrEngine.version()}"
case "sandwich":
return f"Tesseract OCR + PDF {TesseractOcrEngine.version()}"
case _:
return f"Tesseract OCR {TesseractOcrEngine.version()}"
def __str__(self):
return f"Tesseract OCR {TesseractOcrEngine.version()}"
@@ -250,8 +420,9 @@ class TesseractOcrEngine(OcrEngine):
def get_orientation(input_file, options):
return tesseract.get_orientation(
input_file,
engine_mode=options.tesseract_oem,
timeout=options.tesseract_non_ocr_timeout,
engine_mode=options.tesseract.oem,
timeout=options.tesseract.non_ocr_timeout,
omp_thread_limit=options.tesseract.omp_thread_limit,
)
@staticmethod
@@ -259,8 +430,9 @@ class TesseractOcrEngine(OcrEngine):
return tesseract.get_deskew(
input_file,
languages=options.languages,
engine_mode=options.tesseract_oem,
timeout=options.tesseract_non_ocr_timeout,
engine_mode=options.tesseract.oem,
timeout=options.tesseract.non_ocr_timeout,
omp_thread_limit=options.tesseract.omp_thread_limit,
)
@staticmethod
@@ -270,13 +442,14 @@ class TesseractOcrEngine(OcrEngine):
output_hocr=output_hocr,
output_text=output_text,
languages=options.languages,
engine_mode=options.tesseract_oem,
tessconfig=options.tesseract_config,
timeout=options.tesseract_timeout,
pagesegmode=options.tesseract_pagesegmode,
thresholding=options.tesseract_thresholding,
user_words=options.user_words,
user_patterns=options.user_patterns,
engine_mode=options.tesseract.oem,
tessconfig=options.tesseract.config,
timeout=options.tesseract.timeout,
pagesegmode=options.tesseract.pagesegmode,
thresholding=options.tesseract.thresholding,
user_words=options.tesseract.user_words,
user_patterns=options.tesseract.user_patterns,
omp_thread_limit=options.tesseract.omp_thread_limit,
)
@staticmethod
@@ -286,16 +459,23 @@ class TesseractOcrEngine(OcrEngine):
output_pdf=output_pdf,
output_text=output_text,
languages=options.languages,
engine_mode=options.tesseract_oem,
tessconfig=options.tesseract_config,
timeout=options.tesseract_timeout,
pagesegmode=options.tesseract_pagesegmode,
thresholding=options.tesseract_thresholding,
user_words=options.user_words,
user_patterns=options.user_patterns,
engine_mode=options.tesseract.oem,
tessconfig=options.tesseract.config,
timeout=options.tesseract.timeout,
pagesegmode=options.tesseract.pagesegmode,
thresholding=options.tesseract.thresholding,
user_words=options.tesseract.user_words,
user_patterns=options.tesseract.user_patterns,
omp_thread_limit=options.tesseract.omp_thread_limit,
)
@hookimpl
def get_ocr_engine():
def get_ocr_engine(options):
"""Return TesseractOcrEngine when selected or as default."""
if options is not None:
ocr_engine = getattr(options, 'ocr_engine', 'auto')
# Tesseract is selected if explicitly requested or if 'auto'
if ocr_engine not in ('auto', 'tesseract'):
return None
return TesseractOcrEngine()
src/ocrmypdf/cli.py
+128 -50
View File
@@ -6,11 +6,14 @@
from __future__ import annotations
import argparse
from argparse import ArgumentParser
from collections.abc import Callable, Mapping
from typing import Any, TypeVar
from ocrmypdf._defaults import DEFAULT_ROTATE_PAGES_THRESHOLD
from ocrmypdf._defaults import PROGRAM_NAME as _PROGRAM_NAME
from ocrmypdf._options import OcrOptions, ProcessingMode
from ocrmypdf._plugin_manager import OcrmypdfPluginManager
from ocrmypdf._version import __version__ as _VERSION
T = TypeVar('T', int, float)
@@ -51,39 +54,6 @@ def str_to_int(mapping: Mapping[str, int]):
return _str_to_int
class ArgumentParser(argparse.ArgumentParser):
"""Override parser's default behavior of calling sys.exit().
https://stackoverflow.com/questions/5943249/python-argparse-and-controlling-overriding-the-exit-status-code
OCRmyPDF began as a CLI but eventually acquired an API. The API works inside out,
by synthesizing a command line argument. So we subclass the standard parser with
one that doesn't call sys.exit(). Obviously this is not the ideal way to do things
but it works for us.
"""
def __init__(self, *args, **kwargs):
"""Initialize the parser."""
super().__init__(*args, **kwargs)
self._api_mode = False
def enable_api_mode(self):
"""Enable API mode.
When set, the parser will not call sys.exit() on error. OCRmyPDF was originally
a command line program, but now it has an API. The API works by synthesizing
command line arguments.
"""
self._api_mode = True
def error(self, message):
"""Override the default argparse error behavior."""
if not self._api_mode:
super().error(message)
return
raise ValueError(message)
class LanguageSetAction(argparse.Action):
"""Manages a list of languages."""
@@ -96,7 +66,7 @@ class LanguageSetAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
"""Add a language to the set."""
dest = getattr(namespace, self.dest)
if '+' in values:
if isinstance(values, str) and '+' in values:
[dest.append(lang) for lang in values.split('+')]
else:
dest.append(values)
@@ -189,16 +159,17 @@ Online documentation is located at:
)
parser.add_argument(
'--output-type',
choices=['pdfa', 'pdf', 'pdfa-1', 'pdfa-2', 'pdfa-3', 'none'],
default='pdfa',
help="Choose output type. 'pdfa' creates a PDF/A-2b compliant file for "
"long term archiving (default, recommended) but may not suitable "
"for users who want their file altered as little as possible. 'pdfa' "
"also has problems with full Unicode text. 'pdf' minimizes changes "
"to the input file. 'pdf-a1' creates a "
"PDF/A-1b file. 'pdf-a2' is equivalent to 'pdfa'. 'pdf-a3' creates a "
"PDF/A-3b file. 'none' will produce no output, which may be helpful if "
"only the --sidecar is desired.",
choices=['auto', 'pdfa', 'pdf', 'pdfa-1', 'pdfa-2', 'pdfa-3', 'none'],
default='auto',
help="Choose output type. 'auto' (default) produces best-effort PDF/A "
"without requiring Ghostscript - uses verapdf validation when available, "
"otherwise passes through as PDF/A if safe (input already PDF/A or "
"force-ocr was used), or falls back to regular PDF. 'pdfa' creates a "
"PDF/A-2b compliant file for long term archiving (requires Ghostscript "
"as fallback). 'pdf' minimizes changes to the input file. 'pdfa-1' "
"creates a PDF/A-1b file. 'pdfa-2' is equivalent to 'pdfa'. 'pdfa-3' "
"creates a PDF/A-3b file. 'none' will produce no output, which may be "
"helpful if only the --sidecar is desired.",
)
# Use null string '\0' as sentinel to indicate the user supplied no argument,
@@ -337,12 +308,25 @@ Online documentation is located at:
)
ocrsettings = parser.add_argument_group("OCR options", "Control how OCR is applied")
ocrsettings.add_argument(
'-m',
'--mode',
choices=[mode.value for mode in ProcessingMode],
default=ProcessingMode.default.value,
help="Processing mode for pages with existing text. "
"'default' errors if text is found. "
"'force' rasterizes all content and runs OCR (same as --force-ocr). "
"'skip' skips pages with existing text (same as --skip-text). "
"'redo' re-OCRs pages, replacing old invisible text (same as --redo-ocr).",
)
# Legacy flags for backward compatibility - these set the mode internally
ocrsettings.add_argument(
'-f',
'--force-ocr',
action='store_true',
help="Rasterize any text or vector objects on each page, apply OCR, and "
"save the rastered output (this rewrites the PDF)",
"save the rastered output (this rewrites the PDF). "
"Equivalent to --mode force.",
)
ocrsettings.add_argument(
'-s',
@@ -350,7 +334,8 @@ Online documentation is located at:
action='store_true',
help="Skip OCR on any pages that already contain text, but include the "
"page in final output; useful for PDFs that contain a mix of "
"images, text pages, and/or previously OCRed pages",
"images, text pages, and/or previously OCRed pages. "
"Equivalent to --mode skip.",
)
ocrsettings.add_argument(
'--redo-ocr',
@@ -358,7 +343,8 @@ Online documentation is located at:
help="Attempt to detect and remove the hidden OCR layer from files that "
"were previously OCRed with OCRmyPDF or another program. Apply OCR "
"to text found in raster images. Existing visible text objects will "
"not be changed. If there is no existing OCR, OCR will be added.",
"not be changed. If there is no existing OCR, OCR will be added. "
"Equivalent to --mode redo.",
)
ocrsettings.add_argument(
'--skip-big',
@@ -397,10 +383,30 @@ Online documentation is located at:
)
advanced.add_argument(
'--pdf-renderer',
choices=['auto', 'hocr', 'sandwich', 'hocrdebug'],
choices=['auto', 'hocr', 'sandwich', 'hocrdebug', 'fpdf2'],
default='auto',
help="Choose OCR PDF renderer - the default option is to let OCRmyPDF "
"choose. See documentation for discussion.",
help="Choose OCR PDF renderer. 'auto' (recommended) uses fpdf2, which "
"provides full international language support including RTL scripts, "
"proper text positioning, and invisible text that becomes visible when "
"selected. 'sandwich' renders text as a background layer. Legacy 'hocr' "
"and 'hocrdebug' options are deprecated and will use fpdf2.",
)
advanced.add_argument(
'--ocr-engine',
choices=['auto', 'tesseract', 'none'],
default='auto',
help="OCR engine to use. 'auto' (default) selects the best available engine. "
"'tesseract' uses Tesseract OCR. "
"'none' skips OCR entirely, useful for PDF/A conversion or image processing "
"without text recognition.",
)
advanced.add_argument(
'--rasterizer',
choices=['auto', 'ghostscript', 'pypdfium'],
default='auto',
help="Choose PDF page rasterizer. 'auto' prefers pypdfium when available, "
"falling back to Ghostscript. 'pypdfium' is faster but requires the "
"pypdfium2 package. 'ghostscript' uses the traditional Ghostscript rasterizer.",
)
advanced.add_argument(
'--rotate-pages-threshold',
@@ -465,3 +471,75 @@ plugins_only_parser.add_argument(
default=[],
help="Name of plugin to import.",
)
def namespace_to_options(ns) -> OcrOptions:
"""Convert argparse.Namespace to OcrOptions.
This function encapsulates CLI-specific knowledge of how command line
arguments map to our internal options model.
"""
# Extract known fields
known_fields = {}
extra_attrs = {}
# Legacy boolean flags that map to mode - handled by OcrOptions model validator
legacy_mode_flags = {'force_ocr', 'skip_text', 'redo_ocr'}
for key, value in vars(ns).items():
if key in OcrOptions.model_fields:
known_fields[key] = value
elif key in legacy_mode_flags:
# Pass legacy flags to OcrOptions for conversion to mode
known_fields[key] = value
else:
extra_attrs[key] = value
# Handle special cases for hOCR API
if 'output_folder' in extra_attrs and 'output_file' not in known_fields:
known_fields['output_file'] = '/dev/null' # Placeholder
# Handle case where input_file is missing (e.g., in _hocr_to_ocr_pdf)
if 'work_folder' in extra_attrs and 'input_file' not in known_fields:
known_fields['input_file'] = '/dev/null' # Placeholder
instance = OcrOptions(**known_fields)
instance.extra_attrs = extra_attrs
return instance
def get_options_and_plugins(
args=None,
) -> tuple[OcrOptions, OcrmypdfPluginManager]:
"""Parse command line arguments and return OcrOptions and plugin manager.
This is the main entry point for CLI argument processing. It handles
plugin discovery, argument parsing, and conversion to our internal
options model.
Args:
args: Command line arguments. If None, uses sys.argv.
Returns:
Tuple of (OcrOptions, PluginManager)
"""
# Import here to avoid circular imports
from ocrmypdf.api import setup_plugin_infrastructure
# First pass: get plugins so we can register their options
pre_options, _unused = plugins_only_parser.parse_known_args(args=args)
# Set up plugin infrastructure with proper initialization
plugin_manager = setup_plugin_infrastructure(plugins=pre_options.plugins)
# Get parser and let plugins add their options
parser = get_parser()
plugin_manager.add_options(parser=parser)
# Parse all arguments
namespace = parser.parse_args(args=args)
# Convert to OcrOptions
options = namespace_to_options(namespace)
return options, plugin_manager
src/ocrmypdf/data/NotoSans-Regular.ttf
BIN
View File
Binary file not shown.
src/ocrmypdf/data/Occulta.ttf
BIN
View File
Binary file not shown.
src/ocrmypdf/data/pdf.ttf
BIN
View File
Binary file not shown.
src/ocrmypdf/font/__init__.py
+31
View File
@@ -0,0 +1,31 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Font management for OCRmyPDF PDF rendering.
This module provides font infrastructure for the fpdf2 PDF renderer. It includes:
- FontManager: Base class for font loading and glyph checking
- FontProvider: Protocol and implementations for font discovery
- MultiFontManager: Automatic font selection for multilingual documents
- SystemFontProvider: System font discovery
"""
from __future__ import annotations
from ocrmypdf.font.font_manager import FontManager
from ocrmypdf.font.font_provider import (
BuiltinFontProvider,
ChainedFontProvider,
FontProvider,
)
from ocrmypdf.font.multi_font_manager import MultiFontManager
from ocrmypdf.font.system_font_provider import SystemFontProvider
__all__ = [
"FontManager",
"FontProvider",
"BuiltinFontProvider",
"ChainedFontProvider",
"MultiFontManager",
"SystemFontProvider",
]
src/ocrmypdf/font/font_manager.py
+115
View File
@@ -0,0 +1,115 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Base font management for PDF rendering.
This module provides the base FontManager class that handles font loading
and glyph checking using uharfbuzz.
"""
from __future__ import annotations
from pathlib import Path
import uharfbuzz as hb
class FontManager:
"""Manages font loading and glyph checking for PDF rendering.
This base class handles loading fonts with uharfbuzz for glyph checking
and text shaping. Renderer-specific subclasses should extend this to
add their own font objects.
Attributes:
font_path: Path to the font file
font_data: Raw font file bytes
font_index: Index within TTC collection (0 for single-font files)
hb_face: uharfbuzz Face object
hb_font: uharfbuzz Font object
"""
def __init__(self, font_path: Path, font_index: int = 0):
"""Initialize font manager.
Args:
font_path: Path to TrueType/OpenType font file
font_index: Index of font within a TTC collection (default 0).
For single-font files (.ttf, .otf), use 0.
"""
self.font_path = font_path
self.font_index = font_index
# Load font data
self.font_data = font_path.read_bytes()
# Load font with uharfbuzz for glyph checking and text measurement
# Note: uharfbuzz Face also supports font_index for TTC files
self.hb_face = hb.Face(self.font_data, font_index)
self.hb_font = hb.Font(self.hb_face)
def get_hb_font(self) -> hb.Font:
"""Get uharfbuzz Font object for text measurement.
Returns:
UHarfBuzz Font instance
"""
return self.hb_font
def has_glyph(self, codepoint: int) -> bool:
"""Check if font has a glyph for given codepoint.
Args:
codepoint: Unicode codepoint
Returns:
True if font has a real glyph (not .notdef)
"""
glyph_id = self.hb_font.get_nominal_glyph(codepoint)
return glyph_id is not None and glyph_id != 0
def get_font_metrics(self) -> tuple[float, float, float]:
"""Get normalized font metrics (ascent, descent, units_per_em).
Returns:
Tuple of (ascent, descent, units_per_em) where ascent and descent
are in font units. Ascent is positive (above baseline), descent
is typically negative (below baseline).
"""
extents = self.hb_font.get_font_extents('ltr')
units_per_em = self.hb_face.upem
return (extents.ascender, extents.descender, units_per_em)
def get_left_side_bearing(self, char: str, font_size: float) -> float:
"""Get the left side bearing of a character at a given font size.
The left side bearing (lsb) is the horizontal distance from the glyph
origin (x=0) to the leftmost pixel of the glyph. A positive lsb means
there's whitespace before the glyph starts.
Args:
char: Single character to get lsb for
font_size: Font size in points
Returns:
Left side bearing in points. Returns 0 if character not found.
"""
if not char:
return 0.0
codepoint = ord(char)
glyph_id = self.hb_font.get_nominal_glyph(codepoint)
if glyph_id is None or glyph_id == 0:
return 0.0
# Get glyph extents which include left/right bearing info
extents = self.hb_font.get_glyph_extents(glyph_id)
if extents is None:
return 0.0
# x_bearing is the left side bearing in font units
units_per_em = self.hb_face.upem
lsb_units = extents.x_bearing
lsb_pt = lsb_units * font_size / units_per_em
return lsb_pt
src/ocrmypdf/font/font_provider.py
+189
View File
@@ -0,0 +1,189 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Font provider protocol and implementations for PDF rendering."""
from __future__ import annotations
import logging
from pathlib import Path
from typing import Protocol
from ocrmypdf.font.font_manager import FontManager
log = logging.getLogger(__name__)
class FontProvider(Protocol):
"""Protocol for providing fonts to MultiFontManager.
Implementations are responsible for knowing where fonts are located
and loading them. MultiFontManager asks for fonts by name and uses
them for glyph coverage checking.
"""
def get_font(self, font_name: str) -> FontManager | None:
"""Get a FontManager for the named font.
Args:
font_name: Logical font name (e.g., 'NotoSans-Regular')
Returns:
FontManager if font is available, None otherwise
"""
...
def get_available_fonts(self) -> list[str]:
"""Get list of available font names.
Returns:
List of font names that can be retrieved with get_font()
"""
...
def get_fallback_font(self) -> FontManager:
"""Get the glyphless fallback font.
This font must always be available and handles any codepoint.
Returns:
FontManager for the glyphless fallback font (Occulta.ttf)
"""
...
class BuiltinFontProvider:
"""Font provider using builtin fonts from ocrmypdf/data directory."""
# Mapping of logical font names to filenames
# Only Latin (NotoSans) and the glyphless fallback (Occulta.ttf) are bundled.
# All other scripts (Arabic, Devanagari, CJK, etc.) are discovered from
# system fonts by SystemFontProvider to reduce package size.
FONT_FILES = {
'NotoSans-Regular': 'NotoSans-Regular.ttf',
'Occulta': 'Occulta.ttf',
}
def __init__(self, font_dir: Path | None = None):
"""Initialize builtin font provider.
Args:
font_dir: Directory containing font files. If None, uses
the default ocrmypdf/data directory.
"""
if font_dir is None:
font_dir = Path(__file__).parent.parent / "data"
self.font_dir = font_dir
self._fonts: dict[str, FontManager] = {}
self._load_fonts()
def _load_fonts(self) -> None:
"""Load available fonts, logging warnings for missing ones."""
for font_name, font_file in self.FONT_FILES.items():
font_path = self.font_dir / font_file
if not font_path.exists():
if font_name == 'Occulta':
raise FileNotFoundError(
f"Required fallback font not found: {font_path}"
)
log.warning(
"Font %s not found at %s - OCR output quality for some "
"scripts may be affected",
font_name,
font_path,
)
continue
try:
self._fonts[font_name] = FontManager(font_path)
except Exception as e:
if font_name == 'Occulta':
raise ValueError(
f"Failed to load required fallback font {font_file}: {e}"
) from e
log.warning(
"Failed to load font %s: %s - OCR output quality may be affected",
font_name,
e,
)
def get_font(self, font_name: str) -> FontManager | None:
"""Get a FontManager for the named font."""
return self._fonts.get(font_name)
def get_available_fonts(self) -> list[str]:
"""Get list of available font names."""
return list(self._fonts.keys())
def get_fallback_font(self) -> FontManager:
"""Get the glyphless fallback font."""
return self._fonts['Occulta']
class ChainedFontProvider:
"""Font provider that tries multiple providers in order.
This allows combining builtin fonts with system fonts, trying
the builtin provider first and falling back to system fonts
for fonts not bundled with the package.
"""
def __init__(self, providers: list[FontProvider]):
"""Initialize chained font provider.
Args:
providers: List of font providers to try in order.
The first provider that returns a font wins.
"""
if not providers:
raise ValueError("At least one provider is required")
self.providers = providers
def get_font(self, font_name: str) -> FontManager | None:
"""Get a FontManager for the named font.
Tries each provider in order until one returns a font.
Args:
font_name: Logical font name (e.g., 'NotoSans-Regular')
Returns:
FontManager if any provider has the font, None otherwise
"""
for provider in self.providers:
if font := provider.get_font(font_name):
return font
return None
def get_available_fonts(self) -> list[str]:
"""Get list of available font names from all providers.
Returns:
Combined list of font names (deduplicated, order preserved)
"""
seen: set[str] = set()
result: list[str] = []
for provider in self.providers:
for name in provider.get_available_fonts():
if name not in seen:
seen.add(name)
result.append(name)
return result
def get_fallback_font(self) -> FontManager:
"""Get the glyphless fallback font.
Tries each provider until one provides a fallback font.
Returns:
FontManager for the fallback font
Raises:
RuntimeError: If no provider can provide a fallback font
"""
for provider in self.providers:
try:
return provider.get_fallback_font()
except (NotImplementedError, AttributeError, KeyError):
continue
raise RuntimeError("No fallback font available from any provider")
src/ocrmypdf/font/multi_font_manager.py
+334
View File
@@ -0,0 +1,334 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Multi-font management for PDF rendering.
Provides automatic font selection for multilingual documents based on
language hints and glyph coverage analysis.
"""
from __future__ import annotations
import logging
from pathlib import Path
from ocrmypdf.font.font_manager import FontManager
from ocrmypdf.font.font_provider import (
BuiltinFontProvider,
ChainedFontProvider,
FontProvider,
)
from ocrmypdf.font.system_font_provider import SystemFontProvider
log = logging.getLogger(__name__)
class MultiFontManager:
"""Manages multiple fonts with automatic selection and fallback.
This class orchestrates multiple FontManager instances to provide
word-level font selection for multilingual documents. It uses a hybrid
approach combining language hints from hOCR with glyph coverage analysis.
Font selection strategy:
1. Try language-preferred font (if language hint available)
2. Try fallback fonts in order by glyph coverage
3. Fall back to Occulta.ttf (glyphless fallback)
"""
# Language to font mapping
# Keys are ISO 639-2/3 codes or Tesseract language codes
LANGUAGE_FONT_MAP = {
# Arabic script
'ara': 'NotoSansArabic-Regular', # Arabic
'per': 'NotoSansArabic-Regular', # Persian (uses Arabic script)
'fas': 'NotoSansArabic-Regular', # Farsi (alternative code for Persian)
'urd': 'NotoSansArabic-Regular', # Urdu (uses Arabic script)
'pus': 'NotoSansArabic-Regular', # Pashto
'kur': 'NotoSansArabic-Regular', # Kurdish (Arabic script variant)
# Devanagari script
'hin': 'NotoSansDevanagari-Regular', # Hindi
'san': 'NotoSansDevanagari-Regular', # Sanskrit
'mar': 'NotoSansDevanagari-Regular', # Marathi
'nep': 'NotoSansDevanagari-Regular', # Nepali
'kok': 'NotoSansDevanagari-Regular', # Konkani
'bho': 'NotoSansDevanagari-Regular', # Bhojpuri
'mai': 'NotoSansDevanagari-Regular', # Maithili
# CJK
'chi': 'NotoSansCJK-Regular', # Chinese (generic)
'zho': 'NotoSansCJK-Regular', # Chinese (ISO 639-3)
'chi_sim': 'NotoSansCJK-Regular', # Chinese Simplified (Tesseract)
'chi_tra': 'NotoSansCJK-Regular', # Chinese Traditional (Tesseract)
'jpn': 'NotoSansCJK-Regular', # Japanese
'kor': 'NotoSansCJK-Regular', # Korean
# Thai
'tha': 'NotoSansThai-Regular', # Thai
# Hebrew
'heb': 'NotoSansHebrew-Regular', # Hebrew
'yid': 'NotoSansHebrew-Regular', # Yiddish (uses Hebrew script)
# Bengali script
'ben': 'NotoSansBengali-Regular', # Bengali
'asm': 'NotoSansBengali-Regular', # Assamese (uses Bengali script)
# Tamil
'tam': 'NotoSansTamil-Regular', # Tamil
# Gujarati
'guj': 'NotoSansGujarati-Regular', # Gujarati
# Telugu
'tel': 'NotoSansTelugu-Regular', # Telugu
# Kannada
'kan': 'NotoSansKannada-Regular', # Kannada
# Malayalam
'mal': 'NotoSansMalayalam-Regular', # Malayalam
# Myanmar (Burmese)
'mya': 'NotoSansMyanmar-Regular', # Myanmar
# Khmer (Cambodian)
'khm': 'NotoSansKhmer-Regular', # Khmer
# Lao
'lao': 'NotoSansLao-Regular', # Lao
# Georgian
'kat': 'NotoSansGeorgian-Regular', # Georgian
'geo': 'NotoSansGeorgian-Regular', # Georgian (alternative)
# Armenian
'hye': 'NotoSansArmenian-Regular', # Armenian
'arm': 'NotoSansArmenian-Regular', # Armenian (alternative)
# Ethiopic
'amh': 'NotoSansEthiopic-Regular', # Amharic
'tir': 'NotoSansEthiopic-Regular', # Tigrinya
# Sinhala
'sin': 'NotoSansSinhala-Regular', # Sinhala
# Gurmukhi (Punjabi)
'pan': 'NotoSansGurmukhi-Regular', # Punjabi
'pnb': 'NotoSansGurmukhi-Regular', # Western Punjabi
# Oriya
'ori': 'NotoSansOriya-Regular', # Oriya
'ory': 'NotoSansOriya-Regular', # Oriya (alternative)
# Tibetan
'bod': 'NotoSansTibetan-Regular', # Tibetan
'tib': 'NotoSansTibetan-Regular', # Tibetan (alternative)
}
# Ordered fallback chain for fonts (after language-preferred font)
# Order matters: most common scripts first for faster matching
FALLBACK_FONTS = [
'NotoSans-Regular', # Latin, Greek, Cyrillic
'NotoSansArabic-Regular',
'NotoSansDevanagari-Regular',
'NotoSansCJK-Regular',
'NotoSansThai-Regular',
'NotoSansHebrew-Regular',
'NotoSansBengali-Regular',
'NotoSansTamil-Regular',
'NotoSansGujarati-Regular',
'NotoSansTelugu-Regular',
'NotoSansKannada-Regular',
'NotoSansMalayalam-Regular',
'NotoSansMyanmar-Regular',
'NotoSansKhmer-Regular',
'NotoSansLao-Regular',
'NotoSansGeorgian-Regular',
'NotoSansArmenian-Regular',
'NotoSansEthiopic-Regular',
'NotoSansSinhala-Regular',
'NotoSansGurmukhi-Regular',
'NotoSansOriya-Regular',
'NotoSansTibetan-Regular',
]
def __init__(
self,
font_dir: Path | None = None,
*,
font_provider: FontProvider | None = None,
):
"""Initialize multi-font manager.
Args:
font_dir: Directory containing font files. If font_provider is
not specified, this is passed to BuiltinFontProvider.
font_provider: Provider for loading fonts. If None, uses a
ChainedFontProvider that tries builtin fonts first,
then searches system fonts.
"""
if font_provider is not None:
self.font_provider = font_provider
else:
# Use chained provider: try builtin fonts first, then system fonts
self.font_provider = ChainedFontProvider(
[
BuiltinFontProvider(font_dir),
SystemFontProvider(),
]
)
# Font selection cache: (word_text, language) -> font_name
self._selection_cache: dict[tuple[str, str | None], str] = {}
# Track whether we've warned about missing fonts (warn once per script)
self._warned_scripts: set[str] = set()
@property
def fonts(self) -> dict[str, FontManager]:
"""Get all loaded fonts (backward compatibility)."""
return self.get_all_fonts()
def _try_font(
self, font_name: str, word_text: str, cache_key: tuple[str, str | None]
) -> FontManager | None:
"""Try to use a font for the given word.
Args:
font_name: Name of font to try
word_text: Text content to check
cache_key: Cache key for storing successful result
Returns:
FontManager if font exists and has all glyphs, None otherwise
"""
font = self.font_provider.get_font(font_name)
if font is None:
return None
if self._has_all_glyphs(font, word_text):
self._selection_cache[cache_key] = font_name
return font
return None
def select_font_for_word(
self, word_text: str, line_language: str | None
) -> FontManager:
"""Select appropriate font for a word.
Uses a hybrid approach:
1. Language-based selection (if language hint available)
2. Ordered fallback through available fonts by glyph coverage
3. Final fallback to Occulta.ttf (glyphless)
Args:
word_text: The text content of the word
line_language: Language code from hOCR (e.g., 'ara', 'eng')
Returns:
FontManager instance to use for rendering this word
"""
cache_key = (word_text, line_language)
if cache_key in self._selection_cache:
cached_name = self._selection_cache[cache_key]
font = self.font_provider.get_font(cached_name)
if font:
return font
tried_fonts: set[str] = set()
# Phase 1: Try language-preferred font
if line_language and line_language in self.LANGUAGE_FONT_MAP:
preferred = self.LANGUAGE_FONT_MAP[line_language]
tried_fonts.add(preferred)
if result := self._try_font(preferred, word_text, cache_key):
return result
# Phase 2: Try fallback fonts in order
for font_name in self.FALLBACK_FONTS:
if font_name in tried_fonts:
continue
if result := self._try_font(font_name, word_text, cache_key):
return result
# Phase 3: Glyphless fallback (always succeeds)
# Warn if we're falling back for non-ASCII text (likely missing font)
self._warn_missing_font(word_text, line_language)
self._selection_cache[cache_key] = 'Occulta'
return self.font_provider.get_fallback_font()
def _warn_missing_font(self, word_text: str, line_language: str | None) -> None:
"""Warn user about missing font for non-Latin text.
Only warns once per language/script to avoid log spam.
"""
# Determine a key for deduplication (language or 'non-ascii')
warn_key = line_language if line_language else 'unknown'
# Only warn for non-ASCII text and only once per key
if warn_key in self._warned_scripts:
return
# Check if text contains non-ASCII characters
if not any(ord(c) > 127 for c in word_text):
return
self._warned_scripts.add(warn_key)
if line_language and line_language in self.LANGUAGE_FONT_MAP:
font_name = self.LANGUAGE_FONT_MAP[line_language]
log.warning(
"No font found with glyphs for '%s' text. "
"Install %s for better rendering. "
"See https://fonts.google.com/noto",
line_language,
font_name,
)
else:
log.warning(
"No font found with glyphs for some text. "
"Install Noto fonts for better rendering. "
"See https://fonts.google.com/noto"
)
def _has_all_glyphs(self, font: FontManager, text: str) -> bool:
"""Check if a font has glyphs for all characters in text.
Args:
font: FontManager instance to check
text: Text to verify coverage for
Returns:
True if font has real glyphs for all characters (not .notdef)
"""
if not text:
return True
hb_font = font.get_hb_font()
for char in text:
codepoint = ord(char)
glyph_id = hb_font.get_nominal_glyph(codepoint)
if glyph_id is None or glyph_id == 0: # 0 = .notdef glyph
return False
return True
def has_font(self, font_name: str) -> bool:
"""Check if a named font is available.
Args:
font_name: Name of font to check
Returns:
True if font is available
"""
return self.font_provider.get_font(font_name) is not None
def has_all_glyphs(self, font_name: str, text: str) -> bool:
"""Check if a named font has glyphs for all characters in text.
Args:
font_name: Name of font to check
text: Text to verify coverage for
Returns:
True if font has real glyphs for all characters (not .notdef)
"""
font = self.font_provider.get_font(font_name)
if font is None:
return False
return self._has_all_glyphs(font, text)
def get_all_fonts(self) -> dict[str, FontManager]:
"""Get all loaded font managers.
Returns:
Dictionary mapping font names to FontManager instances
"""
result = {}
for name in self.font_provider.get_available_fonts():
font = self.font_provider.get_font(name)
if font is not None:
result[name] = font
return result
src/ocrmypdf/font/system_font_provider.py
+297
View File
@@ -0,0 +1,297 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""System font discovery for PDF rendering.
Provides lazy discovery of Noto fonts installed on the system across
Linux, macOS, and Windows platforms.
"""
from __future__ import annotations
import logging
import os
import sys
from pathlib import Path
from ocrmypdf.font.font_manager import FontManager
log = logging.getLogger(__name__)
class SystemFontProvider:
"""Discovers and provides system-installed Noto fonts with lazy scanning.
This provider searches standard system font directories for Noto fonts.
Scanning is performed lazily - only when a font is actually requested
and not found in the builtin fonts. Results are cached for the lifetime
of the provider instance.
"""
# System font directories by platform
SYSTEM_FONT_DIRS: dict[str, list[Path]] = {
'linux': [
Path('/usr/share/fonts'),
Path('/usr/local/share/fonts'),
Path.home() / '.fonts',
Path.home() / '.local/share/fonts',
],
'freebsd': [
Path('/usr/local/share/fonts'),
Path.home() / '.fonts',
],
'darwin': [
Path('/Library/Fonts'),
Path('/System/Library/Fonts'),
Path.home() / 'Library/Fonts',
],
# Windows is handled dynamically in _get_font_dirs()
}
# Noto font logical names → possible filenames (priority order)
# The first match found will be used
NOTO_FONT_PATTERNS: dict[str, list[str]] = {
'NotoSans-Regular': [
'NotoSans-Regular.ttf',
'NotoSans-Regular.otf',
],
'NotoSansArabic-Regular': [
'NotoSansArabic-Regular.ttf',
'NotoSansArabic-Regular.otf',
],
'NotoSansDevanagari-Regular': [
'NotoSansDevanagari-Regular.ttf',
'NotoSansDevanagari-Regular.otf',
],
'NotoSansCJK-Regular': [
# Language-specific variants (any will work for CJK)
'NotoSansCJKsc-Regular.otf', # Simplified Chinese
'NotoSansCJKtc-Regular.otf', # Traditional Chinese
'NotoSansCJKjp-Regular.otf', # Japanese
'NotoSansCJKkr-Regular.otf', # Korean
# TTC collections (common on Linux distros)
'NotoSansCJK-Regular.ttc',
'NotoSansCJKsc-Regular.ttc',
# Variable fonts
'NotoSansCJKsc-VF.otf',
],
'NotoSansThai-Regular': [
'NotoSansThai-Regular.ttf',
'NotoSansThai-Regular.otf',
],
'NotoSansHebrew-Regular': [
'NotoSansHebrew-Regular.ttf',
'NotoSansHebrew-Regular.otf',
],
'NotoSansBengali-Regular': [
'NotoSansBengali-Regular.ttf',
'NotoSansBengali-Regular.otf',
],
'NotoSansTamil-Regular': [
'NotoSansTamil-Regular.ttf',
'NotoSansTamil-Regular.otf',
],
'NotoSansGujarati-Regular': [
'NotoSansGujarati-Regular.ttf',
'NotoSansGujarati-Regular.otf',
],
'NotoSansTelugu-Regular': [
'NotoSansTelugu-Regular.ttf',
'NotoSansTelugu-Regular.otf',
],
'NotoSansKannada-Regular': [
'NotoSansKannada-Regular.ttf',
'NotoSansKannada-Regular.otf',
],
'NotoSansMalayalam-Regular': [
'NotoSansMalayalam-Regular.ttf',
'NotoSansMalayalam-Regular.otf',
],
'NotoSansMyanmar-Regular': [
'NotoSansMyanmar-Regular.ttf',
'NotoSansMyanmar-Regular.otf',
],
'NotoSansKhmer-Regular': [
'NotoSansKhmer-Regular.ttf',
'NotoSansKhmer-Regular.otf',
],
'NotoSansLao-Regular': [
'NotoSansLao-Regular.ttf',
'NotoSansLao-Regular.otf',
],
'NotoSansGeorgian-Regular': [
'NotoSansGeorgian-Regular.ttf',
'NotoSansGeorgian-Regular.otf',
],
'NotoSansArmenian-Regular': [
'NotoSansArmenian-Regular.ttf',
'NotoSansArmenian-Regular.otf',
],
'NotoSansEthiopic-Regular': [
'NotoSansEthiopic-Regular.ttf',
'NotoSansEthiopic-Regular.otf',
],
'NotoSansSinhala-Regular': [
'NotoSansSinhala-Regular.ttf',
'NotoSansSinhala-Regular.otf',
],
'NotoSansGurmukhi-Regular': [
'NotoSansGurmukhi-Regular.ttf',
'NotoSansGurmukhi-Regular.otf',
],
'NotoSansOriya-Regular': [
'NotoSansOriya-Regular.ttf',
'NotoSansOriya-Regular.otf',
],
'NotoSansTibetan-Regular': [
'NotoSansTibetan-Regular.ttf',
'NotoSansTibetan-Regular.otf',
],
}
def __init__(self) -> None:
"""Initialize system font provider with empty caches."""
# Cache: font_name -> FontManager (successfully loaded fonts)
self._font_cache: dict[str, FontManager] = {}
# Negative cache: font names we've searched for but not found
self._not_found: set[str] = set()
# Cached font directories (computed lazily)
self._font_dirs: list[Path] | None = None
def _get_platform(self) -> str:
"""Get the current platform identifier.
Returns:
Platform string: 'linux', 'darwin', 'windows', or 'freebsd'
"""
if sys.platform == 'win32':
return 'windows'
elif sys.platform == 'darwin':
return 'darwin'
elif 'freebsd' in sys.platform:
return 'freebsd'
else:
return 'linux'
def _get_font_dirs(self) -> list[Path]:
"""Get font directories for the current platform.
Returns:
List of paths to search for fonts (may include non-existent paths)
"""
if self._font_dirs is not None:
return self._font_dirs
platform = self._get_platform()
if platform == 'windows':
# Get Windows font directories from environment
windir = os.environ.get('WINDIR', r'C:\Windows')
self._font_dirs = [Path(windir) / 'Fonts']
# User-installed fonts (Windows 10+)
localappdata = os.environ.get('LOCALAPPDATA')
if localappdata:
self._font_dirs.append(
Path(localappdata) / 'Microsoft' / 'Windows' / 'Fonts'
)
else:
self._font_dirs = list(self.SYSTEM_FONT_DIRS.get(platform, []))
return self._font_dirs
def _find_font_file(self, font_name: str) -> Path | None:
"""Search system directories for a font file.
Args:
font_name: Logical font name (e.g., 'NotoSansCJK-Regular')
Returns:
Path to font file if found, None otherwise
"""
if font_name not in self.NOTO_FONT_PATTERNS:
return None
patterns = self.NOTO_FONT_PATTERNS[font_name]
for font_dir in self._get_font_dirs():
if not font_dir.exists():
continue
for pattern in patterns:
# Search recursively for the font file
try:
matches = list(font_dir.rglob(pattern))
if matches:
log.debug(
"Found system font %s at %s", font_name, matches[0]
)
return matches[0]
except PermissionError:
# Skip directories we can't read
continue
return None
def get_font(self, font_name: str) -> FontManager | None:
"""Get a FontManager for the named font (lazy loading).
This method implements lazy scanning: fonts are only searched for
when first requested. Results (both positive and negative) are
cached for subsequent calls.
Args:
font_name: Logical font name (e.g., 'NotoSansCJK-Regular')
Returns:
FontManager if font is found and loadable, None otherwise
"""
# Check positive cache first
if font_name in self._font_cache:
return self._font_cache[font_name]
# Check negative cache (already searched, not found)
if font_name in self._not_found:
return None
# Lazy scan for this specific font
font_path = self._find_font_file(font_name)
if font_path is not None:
try:
fm = FontManager(font_path)
self._font_cache[font_name] = fm
return fm
except Exception as e:
log.warning(
"Found font %s at %s but failed to load: %s",
font_name,
font_path,
e,
)
# Cache negative result
self._not_found.add(font_name)
return None
def get_available_fonts(self) -> list[str]:
"""Get list of font names this provider can potentially find.
Note: This returns all font names we know patterns for, not
necessarily fonts that are actually installed. Use get_font()
to check if a specific font is available.
Returns:
List of logical font names
"""
return list(self.NOTO_FONT_PATTERNS.keys())
def get_fallback_font(self) -> FontManager:
"""Get the glyphless fallback font.
Raises:
NotImplementedError: System provider doesn't provide fallback.
Use BuiltinFontProvider for the fallback font.
"""
raise NotImplementedError(
"SystemFontProvider does not provide a fallback font. "
"Use BuiltinFontProvider for Occulta.ttf fallback."
)
src/ocrmypdf/fpdf_renderer/__init__.py
+21
View File
@@ -0,0 +1,21 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""fpdf2-based PDF renderer for OCR text layers.
This module provides the PDF renderer using fpdf2 for creating
searchable OCR text layers.
"""
from __future__ import annotations
from ocrmypdf.fpdf_renderer.renderer import (
DebugRenderOptions,
Fpdf2MultiPageRenderer,
Fpdf2PdfRenderer,
)
__all__ = [
"DebugRenderOptions",
"Fpdf2PdfRenderer",
"Fpdf2MultiPageRenderer",
]
src/ocrmypdf/fpdf_renderer/renderer.py
+833
View File
@@ -0,0 +1,833 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""fpdf2-based PDF renderer for OCR text layers.
This module provides PDF rendering using fpdf2 for creating searchable
OCR text layers.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from itertools import pairwise
from math import atan, degrees
from pathlib import Path
from fpdf import FPDF
from fpdf.enums import TextMode
from pikepdf import Matrix, Rectangle
from ocrmypdf.font import FontManager, MultiFontManager
from ocrmypdf.models.ocr_element import OcrClass, OcrElement
log = logging.getLogger(__name__)
def transform_point(matrix: Matrix, x: float, y: float) -> tuple[float, float]:
"""Transform a point (x, y) by a matrix.
Args:
matrix: pikepdf Matrix to apply
x: X coordinate
y: Y coordinate
Returns:
Tuple of (transformed_x, transformed_y)
"""
# Use a degenerate rectangle to transform a single point
rect = Rectangle(x, y, x, y)
transformed = matrix.transform(rect)
return (transformed.llx, transformed.lly)
def transform_box(
matrix: Matrix, left: float, top: float, right: float, bottom: float
) -> tuple[float, float, float, float]:
"""Transform a bounding box by a matrix.
Args:
matrix: pikepdf Matrix to apply
left: Left edge of box
top: Top edge of box
right: Right edge of box
bottom: Bottom edge of box
Returns:
Tuple of (llx, lly, width, height) of the transformed box
"""
rect = Rectangle(left, top, right, bottom)
transformed = matrix.transform(rect)
return (
transformed.llx,
transformed.lly,
transformed.width,
transformed.height,
)
@dataclass
class DebugRenderOptions:
"""Options for debug visualization during rendering.
When enabled, draws colored lines/shapes to visualize OCR structure.
"""
render_baseline: bool = False # Magenta lines along baselines
render_line_bbox: bool = False # Blue rectangles around lines
render_word_bbox: bool = False # Green rectangles around words
class CoordinateTransform:
"""Manages coordinate transformations for fpdf2 rendering.
Handles conversion from OCR pixel coordinates (top-left origin) to
PDF points. fpdf2 uses top-left origin like hOCR, so no Y-flip needed.
"""
def __init__(self, dpi: float, page_width_px: float, page_height_px: float):
"""Initialize coordinate transform."""
self.dpi = dpi
self.page_width_px = page_width_px
self.page_height_px = page_height_px
@property
def page_width_pt(self) -> float:
"""Page width in PDF points."""
return self.page_width_px * 72.0 / self.dpi
@property
def page_height_pt(self) -> float:
"""Page height in PDF points."""
return self.page_height_px * 72.0 / self.dpi
def px_to_pt(self, value: float) -> float:
"""Convert pixels to PDF points."""
return value * 72.0 / self.dpi
def bbox_to_pt(self, bbox) -> tuple[float, float, float, float]:
"""Convert BoundingBox from pixels to points."""
return (
self.px_to_pt(bbox.left),
self.px_to_pt(bbox.top),
self.px_to_pt(bbox.right),
self.px_to_pt(bbox.bottom),
)
class Fpdf2PdfRenderer:
"""Renders OcrElement trees to PDF using fpdf2.
This class provides the core rendering logic for converting OCR output
into PDF text layers using fpdf2's text drawing capabilities.
"""
def __init__(
self,
page: OcrElement,
dpi: float,
multi_font_manager: MultiFontManager,
invisible_text: bool = True,
debug_render_options: DebugRenderOptions | None = None,
):
"""Initialize renderer.
Args:
page: Root OcrElement (must be ocr_page)
dpi: Source image DPI
multi_font_manager: MultiFontManager instance
invisible_text: If True, render text as invisible (text mode 3)
debug_render_options: Options for debug visualization
Raises:
ValueError: If page is not an ocr_page or lacks a bounding box
"""
if page.ocr_class != OcrClass.PAGE:
raise ValueError("Root element must be ocr_page")
if page.bbox is None:
raise ValueError("Page must have bounding box")
self.page = page
self.dpi = dpi
self.multi_font_manager = multi_font_manager
self.invisible_text = invisible_text
self.debug_options = debug_render_options or DebugRenderOptions()
# Setup coordinate transform
self.coord_transform = CoordinateTransform(
dpi=dpi,
page_width_px=page.bbox.width,
page_height_px=page.bbox.height,
)
# Registered fonts: font_path -> fpdf_family_name
self._registered_fonts: dict[str, str] = {}
def render(self, output_path: Path) -> None:
"""Render page to PDF file.
Args:
output_path: Output PDF file path
"""
# Create PDF with custom page size
pdf = FPDF(
unit="pt",
format=(
self.coord_transform.page_width_pt,
self.coord_transform.page_height_pt,
),
)
pdf.set_auto_page_break(auto=False)
# Enable text shaping for complex scripts
pdf.set_text_shaping(True)
# Disable cell margin to ensure precise text positioning
# fpdf2's cell() adds c_margin padding by default, which shifts text
pdf.c_margin = 0
# Set text mode for invisible text
if self.invisible_text:
pdf.text_rendering_mode = TextMode.INVISIBLE
else:
pdf.text_rendering_mode = TextMode.FILL
# Render content to PDF
self.render_to_pdf(pdf)
# Write PDF
pdf.output(str(output_path))
def render_to_pdf(self, pdf: FPDF) -> None:
"""Render page content to an existing FPDF instance.
This method adds a page and renders all content. Used by both
single-page rendering and multi-page rendering.
Args:
pdf: FPDF instance to render into
"""
# Add page with correct dimensions
pdf.add_page(
format=(
self.coord_transform.page_width_pt,
self.coord_transform.page_height_pt,
)
)
# Render all paragraphs
for para in self.page.paragraphs:
self._render_paragraph(pdf, para)
# If no paragraphs, render lines directly
if not self.page.paragraphs:
for line in self.page.lines:
self._render_line(pdf, line)
def _register_font(self, pdf: FPDF, font_manager: FontManager) -> str:
"""Register font with fpdf2 if not already registered.
Args:
pdf: FPDF instance
font_manager: FontManager containing the font
Returns:
Font family name to use with pdf.set_font()
"""
font_path_str = str(font_manager.font_path)
if font_path_str not in self._registered_fonts:
# Use the font filename stem as the family name
family_name = font_manager.font_path.stem
pdf.add_font(family=family_name, fname=font_path_str)
self._registered_fonts[font_path_str] = family_name
return self._registered_fonts[font_path_str]
def _render_paragraph(self, pdf: FPDF, para: OcrElement) -> None:
"""Render a paragraph element.
Args:
pdf: FPDF instance
para: Paragraph OCR element
"""
for line in para.children:
if line.ocr_class in OcrClass.LINE_TYPES:
self._render_line(pdf, line)
def _render_line(self, pdf: FPDF, line: OcrElement) -> None:
"""Render a line element with baseline support.
Strategy (following pikepdf reference implementation):
1. Create a baseline_matrix that transforms from hOCR coordinates to
a coordinate system aligned with the text baseline
2. For each word, transform its hOCR bbox using baseline_matrix.inverse()
to get its position in the baseline coordinate system
3. Render words along the baseline with horizontal scaling
Args:
pdf: FPDF instance
line: Line OCR element
"""
if line.bbox is None:
return
# Validate line bbox
if line.bbox.height <= 0:
log.error(
"line box is invalid so we cannot render it: box=%s text=%s",
line.bbox,
line.text if hasattr(line, 'text') else '',
)
return
# Convert line bbox to PDF points
line_left_pt = self.coord_transform.px_to_pt(line.bbox.left)
line_top_pt = self.coord_transform.px_to_pt(line.bbox.top)
line_right_pt = self.coord_transform.px_to_pt(line.bbox.right)
line_bottom_pt = self.coord_transform.px_to_pt(line.bbox.bottom)
# Note: line_width_pt and line_height_pt not needed since we compute
# dimensions in the un-rotated coordinate system via matrix transform
# Debug rendering: draw line bbox (in page coordinates)
if self.debug_options.render_line_bbox:
self._render_debug_line_bbox(
pdf, line_left_pt, line_top_pt, line_right_pt, line_bottom_pt
)
# Get textangle (rotation of the entire line)
textangle = line.textangle or 0.0
# Build line_size_aabb_matrix: transforms from page coords to un-rotated
# line coords. The hOCR bbox is the minimum axis-aligned bounding box
# enclosing the rotated text.
# Start at top-left corner of line bbox, then rotate by -textangle
line_size_aabb_matrix = (
Matrix()
.translated(line_left_pt, line_top_pt)
.rotated(-textangle) # textangle is counter-clockwise per hOCR spec
)
# Get the line dimensions in the un-rotated coordinate system
# Transform line bbox corners to get the un-rotated dimensions
inv_line_matrix = line_size_aabb_matrix.inverse()
# Transform bottom-right corner to get line dimensions in rotated space
_, _, line_size_width, line_size_height = transform_box(
inv_line_matrix, line_left_pt, line_top_pt, line_right_pt, line_bottom_pt
)
# Get baseline information (slope and intercept)
slope = 0.0
intercept_pt = 0.0
if line.baseline is not None:
slope = line.baseline.slope
intercept_pt = self.coord_transform.px_to_pt(line.baseline.intercept)
if abs(slope) < 0.005:
slope = 0.0
else:
# No baseline provided: calculate from font metrics
default_font_manager = self.multi_font_manager.fonts['NotoSans-Regular']
ascent, descent, units_per_em = default_font_manager.get_font_metrics()
ascent_norm = ascent / units_per_em
descent_norm = descent / units_per_em
# Baseline intercept based on font metrics
intercept_pt = (
-abs(descent_norm)
* line_size_height
/ (ascent_norm + abs(descent_norm))
)
slope_angle_deg = degrees(atan(slope)) if slope != 0.0 else 0.0
# Build baseline_matrix: transforms from page coords to baseline coords
# 1. Start with line_size_aabb_matrix (translates to line corner, rotates)
# 2. Translate down to bottom of un-rotated line (line_size_height)
# 3. Apply baseline intercept offset
# 4. Rotate by baseline slope
baseline_matrix = (
line_size_aabb_matrix.translated(
0, line_size_height
) # Move to bottom of line
.translated(0, intercept_pt) # Apply baseline intercept
.rotated(slope_angle_deg) # Rotate by baseline slope
)
# Calculate font size: height from baseline to top of line
font_size = line_size_height + intercept_pt
if font_size < 1.0:
font_size = line_size_height * 0.8
# Total rotation for rendering (textangle + slope)
total_rotation_deg = -textangle + slope_angle_deg
# Debug rendering: draw baseline
if self.debug_options.render_baseline:
# Baseline starts at origin in baseline coords, extends line width
baseline_start = transform_point(baseline_matrix, 0, 0)
baseline_end = transform_point(baseline_matrix, line_size_width, 0)
pdf.set_draw_color(255, 0, 255) # Magenta
pdf.set_line_width(0.75)
pdf.line(
baseline_start[0], baseline_start[1], baseline_end[0], baseline_end[1]
)
# Extract line language for font selection
line_language = line.language
# Get inverse of baseline_matrix for transforming word bboxes
inv_baseline_matrix = baseline_matrix.inverse()
# Collect words to render
words: list[OcrElement | None] = [
w for w in line.children if w.ocr_class == OcrClass.WORD and w.text
]
# Render each word followed by space (except last)
# Use pairwise to iterate over consecutive word pairs, pairing the last
# word with a None to signal the end of the line.
for current_word, next_word in pairwise(words + [None]):
if current_word: # Don't render EOL sentinel
# Render the current word
self._render_word(
pdf,
current_word,
baseline_matrix,
inv_baseline_matrix,
font_size,
total_rotation_deg,
line_language,
)
if next_word: # Don't render EOL sentinel
self._maybe_render_space(
pdf,
current_word,
next_word,
baseline_matrix,
inv_baseline_matrix,
font_size,
total_rotation_deg,
line_language,
line.direction,
)
def _render_word(
self,
pdf: FPDF,
word: OcrElement,
baseline_matrix: Matrix,
inv_baseline_matrix: Matrix,
font_size: float,
rotation_deg: float,
line_language: str | None,
) -> None:
"""Render a word using word bbox positioning.
Position text so its visual bounding box matches the hOCR word bbox.
This provides more accurate placement than baseline-relative positioning
because we match the actual glyph bounds rather than relying on font
metrics which may not exactly match the OCR'd text appearance.
Args:
pdf: FPDF instance
word: Word OCR element
baseline_matrix: Transform from baseline coords to page coords
inv_baseline_matrix: Transform from page coords to baseline coords
font_size: Font size in points (from line calculation)
rotation_deg: Total rotation angle for text
line_language: Language code from line for font selection
"""
if not word.text or word.bbox is None:
return
# Select appropriate font for this word
font_manager = self.multi_font_manager.select_font_for_word(
word.text, line_language
)
# Register font with fpdf2
font_family = self._register_font(pdf, font_manager)
# Convert word bbox to PDF points
word_left_pt = self.coord_transform.px_to_pt(word.bbox.left)
word_top_pt = self.coord_transform.px_to_pt(word.bbox.top)
word_right_pt = self.coord_transform.px_to_pt(word.bbox.right)
word_bottom_pt = self.coord_transform.px_to_pt(word.bbox.bottom)
word_width_pt = word_right_pt - word_left_pt
# Transform word bbox into baseline coordinate system to get x position
box_llx, _, _, _ = transform_box(
inv_baseline_matrix,
word_left_pt,
word_top_pt,
word_right_pt,
word_bottom_pt,
)
# Debug rendering: draw word bbox (in page coordinates)
if self.debug_options.render_word_bbox:
self._render_debug_word_bbox(
pdf, word_left_pt, word_top_pt, word_right_pt, word_bottom_pt
)
# Use line-based font_size for consistent vertical sizing
word_font_size = font_size
# Set font
pdf.set_font(font_family, size=word_font_size)
# Calculate natural text width at this font size
natural_width = pdf.get_string_width(word.text)
# Calculate horizontal scale to fit word bbox width
if natural_width > 0 and word_width_pt > 0:
scale_x = (word_width_pt / natural_width) * 100
else:
scale_x = 100
# Apply horizontal stretching
pdf.set_stretching(scale_x)
# Get left side bearing of first character to compensate for glyph offset
lsb_pt = font_manager.get_left_side_bearing(word.text[0], word_font_size)
# Transform the baseline-relative x position back to page coordinates
# The word sits at (box_llx, 0) in baseline coords (on the baseline)
page_x, page_y = transform_point(baseline_matrix, box_llx, 0)
# Adjust x position to account for lsb (scaled by horizontal stretch)
adjusted_x = page_x - lsb_pt * (scale_x / 100)
# Calculate y position based on baseline
# In fpdf2, set_xy(x, y) positions text such that the baseline is at:
# baseline_y = set_y + font_size * (ascent / (ascent + |descent|))
# We want baseline at page_y, so:
# page_y = set_y + font_size * (ascent / (ascent + |descent|))
# set_y = page_y - font_size * (ascent / (ascent + |descent|))
ascent, descent, _ = font_manager.get_font_metrics()
total_height = ascent + abs(descent)
baseline_offset_ratio = ascent / total_height
adjusted_y = page_y - word_font_size * baseline_offset_ratio
# Position and draw text with rotation
if abs(rotation_deg) > 0.1:
with pdf.rotation(-rotation_deg, x=page_x, y=page_y):
pdf.set_xy(adjusted_x, adjusted_y)
pdf.cell(text=word.text)
else:
pdf.set_xy(adjusted_x, adjusted_y)
pdf.cell(text=word.text)
# Reset stretching
pdf.set_stretching(100)
def _is_cjk_only(self, text: str) -> bool:
"""Check if text contains only CJK characters.
CJK scripts don't use spaces between words, so we should not insert
spaces between adjacent CJK words.
Args:
text: Text to check
Returns:
True if text contains only CJK characters
"""
for char in text:
cp = ord(char)
# Check if character is in CJK ranges
if not (
0x4E00 <= cp <= 0x9FFF # CJK Unified Ideographs
or 0x3400 <= cp <= 0x4DBF # CJK Extension A
or 0x20000 <= cp <= 0x2A6DF # CJK Extension B
or 0x2A700 <= cp <= 0x2B73F # CJK Extension C
or 0x2B740 <= cp <= 0x2B81F # CJK Extension D
or 0x2B820 <= cp <= 0x2CEAF # CJK Extension E
or 0x2CEB0 <= cp <= 0x2EBEF # CJK Extension F
or 0x30000 <= cp <= 0x3134F # CJK Extension G
or 0x3040 <= cp <= 0x309F # Hiragana
or 0x30A0 <= cp <= 0x30FF # Katakana
or 0x31F0 <= cp <= 0x31FF # Katakana Phonetic Extensions
or 0xAC00 <= cp <= 0xD7AF # Hangul Syllables
or 0x1100 <= cp <= 0x11FF # Hangul Jamo
or 0x3130 <= cp <= 0x318F # Hangul Compatibility Jamo
or 0xA960 <= cp <= 0xA97F # Hangul Jamo Extended-A
or 0xD7B0 <= cp <= 0xD7FF # Hangul Jamo Extended-B
or 0x3000 <= cp <= 0x303F # CJK Symbols and Punctuation
or 0xFF00 <= cp <= 0xFFEF # Halfwidth and Fullwidth Forms
):
return False
return True
def _maybe_render_space(
self,
pdf: FPDF,
current_word: OcrElement,
next_word: OcrElement,
baseline_matrix: Matrix,
inv_baseline_matrix: Matrix,
font_size: float,
rotation_deg: float,
line_language: str | None,
direction: str | None,
) -> None:
"""Render a space character between two words if a gap exists.
This ensures that PDF readers like pdfminer.six can properly segment
words during text extraction. Some PDF readers rely on explicit space
characters rather than inferring word boundaries from positioning.
Args:
pdf: FPDF instance
current_word: The word that was just rendered
next_word: The next word to be rendered
baseline_matrix: Transform from baseline coords to page coords
inv_baseline_matrix: Transform from page coords to baseline coords
font_size: Font size in points
rotation_deg: Total rotation angle for text
line_language: Language code from line for font selection
direction: Text direction ("ltr" or "rtl")
"""
if current_word.bbox is None or next_word.bbox is None:
return
# Skip if both words are CJK-only (no spaces in CJK text)
if self._is_cjk_only(current_word.text) and self._is_cjk_only(next_word.text):
return
# Calculate gap between words
if direction == "rtl":
gap_left = next_word.bbox.right
gap_right = current_word.bbox.left
else:
gap_left = current_word.bbox.right
gap_right = next_word.bbox.left
gap_width_px = gap_right - gap_left
# Use word height as proxy for line height
line_height_px = current_word.bbox.height
# Skip if gap is too small (noise) or words are overlapping
if gap_width_px <= line_height_px * 0.05:
return
# Render space in the gap
self._render_space(
pdf,
gap_left,
gap_right,
current_word.bbox.top,
current_word.bbox.bottom,
baseline_matrix,
inv_baseline_matrix,
font_size,
rotation_deg,
line_language,
)
def _render_space(
self,
pdf: FPDF,
gap_left_px: float,
gap_right_px: float,
gap_top_px: float,
gap_bottom_px: float,
baseline_matrix: Matrix,
inv_baseline_matrix: Matrix,
font_size: float,
rotation_deg: float,
line_language: str | None,
) -> None:
"""Render a space character in a gap between words.
Uses the same baseline transformation logic as word rendering to ensure
proper alignment on rotated or sloped baselines.
Args:
pdf: FPDF instance
gap_left_px: Left edge of gap in pixels
gap_right_px: Right edge of gap in pixels
gap_top_px: Top edge of gap in pixels
gap_bottom_px: Bottom edge of gap in pixels
baseline_matrix: Transform from baseline coords to page coords
inv_baseline_matrix: Transform from page coords to baseline coords
font_size: Font size in points
rotation_deg: Total rotation angle for text
line_language: Language code from line for font selection
"""
# Convert gap to PDF points
gap_left_pt = self.coord_transform.px_to_pt(gap_left_px)
gap_top_pt = self.coord_transform.px_to_pt(gap_top_px)
gap_right_pt = self.coord_transform.px_to_pt(gap_right_px)
gap_bottom_pt = self.coord_transform.px_to_pt(gap_bottom_px)
gap_width_pt = gap_right_pt - gap_left_pt
# Transform gap bbox into baseline coordinate system to get x position
box_llx, _, _, _ = transform_box(
inv_baseline_matrix,
gap_left_pt,
gap_top_pt,
gap_right_pt,
gap_bottom_pt,
)
# Select font (use default font for space)
font_manager = self.multi_font_manager.select_font_for_word(" ", line_language)
font_family = self._register_font(pdf, font_manager)
# Set font
pdf.set_font(font_family, size=font_size)
# Calculate natural space width and scaling
natural_width = pdf.get_string_width(" ")
if natural_width > 0 and gap_width_pt > 0:
scale_x = (gap_width_pt / natural_width) * 100
else:
scale_x = 100
# Apply horizontal stretching
pdf.set_stretching(scale_x)
# Transform the baseline-relative x position back to page coordinates
page_x, page_y = transform_point(baseline_matrix, box_llx, 0)
# Calculate y position based on baseline (same as _render_word)
ascent, descent, _ = font_manager.get_font_metrics()
total_height = ascent + abs(descent)
baseline_offset_ratio = ascent / total_height
adjusted_y = page_y - font_size * baseline_offset_ratio
# Position and draw space with rotation
if abs(rotation_deg) > 0.1:
with pdf.rotation(-rotation_deg, x=page_x, y=page_y):
pdf.set_xy(page_x, adjusted_y)
pdf.cell(text=" ")
else:
pdf.set_xy(page_x, adjusted_y)
pdf.cell(text=" ")
# Reset stretching
pdf.set_stretching(100)
def _render_debug_line_bbox(
self,
pdf: FPDF,
left: float,
top: float,
right: float,
bottom: float,
) -> None:
"""Draw a blue box around the line bbox."""
pdf.set_draw_color(0, 0, 255) # Blue
pdf.set_line_width(0.5)
pdf.rect(left, top, right - left, bottom - top)
def _render_debug_baseline(
self,
pdf: FPDF,
x: float,
y: float,
width: float,
rotation_deg: float,
) -> None:
"""Draw a magenta line along the baseline."""
pdf.set_draw_color(255, 0, 255) # Magenta
pdf.set_line_width(0.75)
if abs(rotation_deg) > 0.1:
with pdf.rotation(rotation_deg, x=x, y=y):
pdf.line(x, y, x + width, y)
else:
pdf.line(x, y, x + width, y)
def _render_debug_word_bbox(
self,
pdf: FPDF,
left: float,
top: float,
right: float,
bottom: float,
) -> None:
"""Draw a green box around the word bbox."""
pdf.set_draw_color(0, 255, 0) # Green
pdf.set_line_width(0.3)
pdf.rect(left, top, right - left, bottom - top)
class Fpdf2MultiPageRenderer:
"""Renders multiple OcrElement pages into a single PDF.
This class handles multi-page documents by delegating to Fpdf2PdfRenderer
for each page while sharing a single FPDF instance and font registration.
"""
def __init__(
self,
pages_data: list[tuple[int, OcrElement, float]],
multi_font_manager: MultiFontManager,
invisible_text: bool = True,
debug_render_options: DebugRenderOptions | None = None,
):
"""Initialize multi-page renderer.
Args:
pages_data: List of (pageno, ocr_tree, dpi) tuples
multi_font_manager: Shared multi-font manager for all pages
invisible_text: Whether to render invisible text
debug_render_options: Options for debug visualization
"""
self.pages_data = pages_data
self.multi_font_manager = multi_font_manager
self.invisible_text = invisible_text
self.debug_options = debug_render_options or DebugRenderOptions()
def render(self, output_path: Path) -> None:
"""Render all pages to a single multi-page PDF.
Args:
output_path: Output PDF file path
"""
if not self.pages_data:
raise ValueError("No pages to render")
# Create PDF (page size will be set per-page)
pdf = FPDF(unit="pt")
pdf.set_auto_page_break(auto=False)
pdf.set_text_shaping(True)
# Disable cell margin to ensure precise text positioning
# fpdf2's cell() adds c_margin padding by default, which shifts text
pdf.c_margin = 0
# Set text mode for invisible text
if self.invisible_text:
pdf.text_rendering_mode = TextMode.INVISIBLE
else:
pdf.text_rendering_mode = TextMode.FILL
# Shared font registration across all pages
shared_registered_fonts: dict[str, str] = {}
# Render each page using Fpdf2PdfRenderer
for _pageno, page, dpi in self.pages_data:
if page.bbox is None:
continue
# Create a renderer for this page
page_renderer = Fpdf2PdfRenderer(
page=page,
dpi=dpi,
multi_font_manager=self.multi_font_manager,
invisible_text=self.invisible_text,
debug_render_options=self.debug_options,
)
# Share font registration to avoid re-registering fonts
page_renderer._registered_fonts = shared_registered_fonts
# Render page content to the shared PDF
page_renderer.render_to_pdf(pdf)
# Write PDF
pdf.output(str(output_path))
src/ocrmypdf/helpers.py
+2 -10
View File
@@ -25,7 +25,6 @@ from typing import (
import img2pdf
import pikepdf
from deprecation import deprecated
log = logging.getLogger(__name__)
@@ -136,11 +135,6 @@ class Resolution(Generic[T]):
return self._isclose(self.x, other.x) and self._isclose(self.y, other.y)
@deprecated(deprecated_in='15.4.0')
class NeverRaise(Exception):
"""An exception that is never raised."""
def safe_symlink(input_file: os.PathLike, soft_link_name: os.PathLike) -> None:
"""Create a symbolic link at ``soft_link_name``, which references ``input_file``.
@@ -200,7 +194,7 @@ def is_iterable_notstr(thing: Any) -> bool:
def monotonic(seq: Sequence) -> bool:
"""Does this sequence increase monotonically?"""
return all(b > a for a, b in zip(seq, seq[1:]))
return all(b > a for a, b in zip(seq, seq[1:], strict=False))
def page_number(input_file: os.PathLike) -> int:
@@ -298,9 +292,7 @@ def check_pdf(input_file: Path) -> bool:
if linearize_msgs:
log.warning(linearize_msgs)
if success and not linearize_msgs:
return True
return False
return bool(success and not linearize_msgs)
def clamp(n: T, smallest: T, largest: T) -> T:
src/ocrmypdf/hocrtransform/__init__.py
+39 -9
View File
@@ -1,18 +1,48 @@
# SPDX-FileCopyrightText: 2023 James R. Barlow
# SPDX-FileCopyrightText: 2023-2025 James R. Barlow
# SPDX-License-Identifier: MIT
"""Transform .hocr and page image to text PDF."""
"""Transform OCR output to text-only PDFs.
This package provides tools for:
1. Parsing OCR output (hOCR format) into generic OcrElement structures
2. Rendering OcrElement structures to searchable PDF text layers
The architecture separates parsing from rendering, allowing:
- Support for multiple OCR input formats (hOCR, ALTO, custom engines)
- Independent improvements to text rendering
- Reuse of the OcrElement data model for other purposes
Main components:
- OcrElement: Generic dataclass representing OCR output structure
- HocrParser: Parses hOCR files into OcrElement trees
- Fpdf2PdfRenderer: Renders OcrElement trees to PDF text layers (via fpdf2)
For PDF rendering, use the fpdf2_renderer module:
from ocrmypdf.fpdf_renderer import Fpdf2PdfRenderer, DebugRenderOptions
"""
from __future__ import annotations
from ocrmypdf.hocrtransform._hocr import (
DebugRenderOptions,
HocrTransform,
HocrTransformError,
from ocrmypdf.hocrtransform.hocr_parser import (
HocrParseError,
HocrParser,
)
from ocrmypdf.models.ocr_element import (
Baseline,
BoundingBox,
FontInfo,
OcrClass,
OcrElement,
)
__all__ = (
'HocrTransform',
'HocrTransformError',
'DebugRenderOptions',
# hOCR parsing
'HocrParser',
'HocrParseError',
# OCR element data model
'OcrElement',
'OcrClass',
'BoundingBox',
'Baseline',
'FontInfo',
)
src/ocrmypdf/hocrtransform/__main__.py
+43 -9
View File
@@ -1,11 +1,15 @@
# SPDX-FileCopyrightText: 2023 James R. Barlow
# SPDX-FileCopyrightText: 2023-2025 James R. Barlow
# SPDX-License-Identifier: MIT
"""Simple CLI for testing HOCR."""
"""Simple CLI for testing HOCR to PDF conversion using fpdf2 renderer."""
from __future__ import annotations
import argparse
from pathlib import Path
from ocrmypdf.hocrtransform import HocrTransform
from ocrmypdf.font import MultiFontManager
from ocrmypdf.fpdf_renderer import DebugRenderOptions, Fpdf2PdfRenderer
from ocrmypdf.hocrtransform.hocr_parser import HocrParser
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Convert hocr file to PDF')
@@ -14,7 +18,7 @@ if __name__ == "__main__":
'--boundingboxes',
action="store_true",
default=False,
help='Show bounding boxes borders',
help='Show bounding boxes borders (debug mode)',
)
parser.add_argument(
'-r',
@@ -27,14 +31,44 @@ if __name__ == "__main__":
'-i',
'--image',
default=None,
help='Path to the image to be placed above the text',
help='Path to the image to be placed above the text (not yet supported)',
)
parser.add_argument('hocrfile', help='Path to the hocr file to be parsed')
parser.add_argument('outputfile', help='Path to the PDF file to be generated')
args = parser.parse_args()
hocr = HocrTransform(hocr_filename=args.hocrfile, dpi=args.resolution)
hocr.to_pdf(
out_filename=args.outputfile,
image_filename=args.image,
# Parse hOCR file
hocr_parser = HocrParser(args.hocrfile)
ocr_page = hocr_parser.parse()
# Use DPI from hOCR if available, otherwise use command-line resolution
dpi = ocr_page.dpi or args.resolution
# Setup debug render options if requested
debug_options = None
if args.boundingboxes:
debug_options = DebugRenderOptions(
render_line_bbox=True,
render_word_bbox=True,
render_baseline=True,
)
# Create multi-font manager with default font directory
font_dir = Path(__file__).parent.parent / "data"
multi_font_manager = MultiFontManager(font_dir)
# Render to PDF using fpdf2
renderer = Fpdf2PdfRenderer(
page=ocr_page,
dpi=dpi,
multi_font_manager=multi_font_manager,
invisible_text=not args.boundingboxes, # Visible text in debug mode
debug_render_options=debug_options,
)
renderer.render(Path(args.outputfile))
if args.image:
print(
f"Warning: Image overlay (--image {args.image}) is not yet supported "
"with the fpdf2 renderer."
)
src/ocrmypdf/hocrtransform/_font.py
-141
View File
@@ -1,141 +0,0 @@
# SPDX-FileCopyrightText: 2023 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
from __future__ import annotations
import logging
import unicodedata
import zlib
from importlib.resources import files as package_files
from pikepdf import (
Dictionary,
Name,
Pdf,
)
from pikepdf.canvas import Font
log = logging.getLogger(__name__)
class EncodableFont(Font):
def text_encode(self, text: str) -> bytes:
raise NotImplementedError()
class GlyphlessFont(EncodableFont):
CID_TO_GID_DATA = zlib.compress(b"\x00\x01" * 65536)
GLYPHLESS_FONT_NAME = 'pdf.ttf'
GLYPHLESS_FONT = (package_files('ocrmypdf.data') / GLYPHLESS_FONT_NAME).read_bytes()
CHAR_ASPECT = 2
def __init__(self):
pass
def text_width(self, text: str, fontsize: float) -> float:
"""Estimate the width of a text string when rendered with the given font."""
# NFKC: split ligatures, combine diacritics
return len(unicodedata.normalize("NFKC", text)) * (fontsize / self.CHAR_ASPECT)
def text_encode(self, text: str) -> bytes:
return text.encode('utf-16be')
def register(self, pdf: Pdf):
"""Register the glyphless font.
Create several data structures in the Pdf to describe the font. While it create
the data, a reference should be set in at least one page's /Resources dictionary
to retain the font in the output PDF and ensure it is usable on that page.
"""
PLACEHOLDER = Name.Placeholder
basefont = pdf.make_indirect(
Dictionary(
BaseFont=Name.GlyphLessFont,
DescendantFonts=[PLACEHOLDER],
Encoding=Name("/Identity-H"),
Subtype=Name.Type0,
ToUnicode=PLACEHOLDER,
Type=Name.Font,
)
)
cid_font_type2 = pdf.make_indirect(
Dictionary(
BaseFont=Name.GlyphLessFont,
CIDToGIDMap=PLACEHOLDER,
CIDSystemInfo=Dictionary(
Ordering="Identity",
Registry="Adobe",
Supplement=0,
),
FontDescriptor=PLACEHOLDER,
Subtype=Name.CIDFontType2,
Type=Name.Font,
DW=1000 // self.CHAR_ASPECT,
)
)
basefont.DescendantFonts = [cid_font_type2]
cid_font_type2.CIDToGIDMap = pdf.make_stream(
self.CID_TO_GID_DATA, Filter=Name.FlateDecode
)
basefont.ToUnicode = pdf.make_stream(
b"/CIDInit /ProcSet findresource begin\n"
b"12 dict begin\n"
b"begincmap\n"
b"/CIDSystemInfo\n"
b"<<\n"
b" /Registry (Adobe)\n"
b" /Ordering (UCS)\n"
b" /Supplement 0\n"
b">> def\n"
b"/CMapName /Adobe-Identify-UCS def\n"
b"/CMapType 2 def\n"
b"1 begincodespacerange\n"
b"<0000> <FFFF>\n"
b"endcodespacerange\n"
b"1 beginbfrange\n"
b"<0000> <FFFF> <0000>\n"
b"endbfrange\n"
b"endcmap\n"
b"CMapName currentdict /CMap defineresource pop\n"
b"end\n"
b"end\n"
)
font_descriptor = pdf.make_indirect(
Dictionary(
Ascent=1000,
CapHeight=1000,
Descent=-1,
Flags=5, # Fixed pitch and symbolic
FontBBox=[0, 0, 1000 // self.CHAR_ASPECT, 1000],
FontFile2=PLACEHOLDER,
FontName=Name.GlyphLessFont,
ItalicAngle=0,
StemV=80,
Type=Name.FontDescriptor,
)
)
font_descriptor.FontFile2 = pdf.make_stream(self.GLYPHLESS_FONT)
cid_font_type2.FontDescriptor = font_descriptor
return basefont
class Courier(EncodableFont):
"""Courier font."""
def text_width(self, text: str, fontsize: float) -> float:
"""Estimate the width of a text string when rendered with the given font."""
return len(text) * fontsize
def text_encode(self, text: str) -> bytes:
return text.encode('pdfdoc', errors='ignore')
def register(self, pdf: Pdf) -> Dictionary:
"""Register the font."""
return pdf.make_indirect(
Dictionary(
BaseFont=Name.Courier,
Type=Name.Font,
Subtype=Name.Type1,
)
)
src/ocrmypdf/hocrtransform/_hocr.py
-540
View File
@@ -1,540 +0,0 @@
# SPDX-FileCopyrightText: 2010 Jonathan Brinley
# SPDX-FileCopyrightText: 2013-2014 Julien Pfefferkorn
# SPDX-FileCopyrightText: 2023 James R. Barlow
# SPDX-FileCopyrightText: 2025 Odin Dahlström
# SPDX-License-Identifier: MIT
"""hOCR transform implementation."""
from __future__ import annotations
import logging
import os
import re
import unicodedata
from dataclasses import dataclass
from itertools import pairwise
from math import atan, pi
from pathlib import Path
from xml.etree import ElementTree
from pikepdf import Matrix, Name, Rectangle
from pikepdf.canvas import (
BLACK,
BLUE,
CYAN,
DARKGREEN,
GREEN,
MAGENTA,
RED,
Canvas,
Text,
TextDirection,
)
from ocrmypdf.hocrtransform._font import EncodableFont as Font
from ocrmypdf.hocrtransform._font import GlyphlessFont
log = logging.getLogger(__name__)
INCH = 72.0
Element = ElementTree.Element
@dataclass
class DebugRenderOptions:
"""A class for managing rendering options."""
render_paragraph_bbox: bool = False
render_baseline: bool = False
render_triangle: bool = False
render_line_bbox: bool = False
render_word_bbox: bool = False
render_space_bbox: bool = False
class HocrTransformError(Exception):
"""Error while applying hOCR transform."""
class HocrTransform:
"""A class for converting documents from the hOCR format.
For details of the hOCR format, see:
http://kba.github.io/hocr-spec/1.2/.
"""
box_pattern = re.compile(
r'''
bbox \s+
(\d+) \s+ # left: uint
(\d+) \s+ # top: uint
(\d+) \s+ # right: uint
(\d+) # bottom: uint
''',
re.VERBOSE,
)
baseline_pattern = re.compile(
r'''
baseline \s+
([\-\+]?\d*\.?\d*) \s+ # +/- decimal float
([\-\+]?\d+) # +/- int
''',
re.VERBOSE,
)
textangle_pattern = re.compile(
r'''
textangle \s+
([\-\+]?\d*\.?\d*) # +/- decimal float
''',
re.VERBOSE,
)
def __init__(
self,
*,
hocr_filename: str | Path,
dpi: float,
debug: bool = False,
fontname: Name = Name("/f-0-0"),
font: Font = GlyphlessFont(),
debug_render_options: DebugRenderOptions | None = None,
):
"""Initialize the HocrTransform object."""
if debug:
log.warning("Use debug_render_options instead", DeprecationWarning)
self.render_options = DebugRenderOptions(
render_baseline=debug,
render_triangle=debug,
render_line_bbox=False,
render_word_bbox=debug,
render_paragraph_bbox=False,
render_space_bbox=False,
)
else:
self.render_options = debug_render_options or DebugRenderOptions()
self.dpi = dpi
self.hocr = ElementTree.parse(os.fspath(hocr_filename))
self._fontname = fontname
self._font = font
# if the hOCR file has a namespace, ElementTree requires its use to
# find elements
matches = re.match(r'({.*})html', self.hocr.getroot().tag)
self.xmlns = ''
if matches:
self.xmlns = matches.group(1)
for div in self.hocr.findall(self._child_xpath('div', 'ocr_page')):
coords = self.element_coordinates(div)
if not coords:
raise HocrTransformError("hocr file is missing page dimensions")
self.width = (coords.urx - coords.llx) / (self.dpi / INCH)
self.height = (coords.ury - coords.lly) / (self.dpi / INCH)
# Stop after first div that has page coordinates
break
def _get_element_text(self, element: Element) -> str:
"""Return the textual content of the element and its children."""
text = element.text if element.text is not None else ''
for child in element:
text += self._get_element_text(child)
text += element.tail if element.tail is not None else ''
return text
@classmethod
def element_coordinates(cls, element: Element) -> Rectangle | None:
"""Get coordinates of the bounding box around an element."""
matches = cls.box_pattern.search(element.attrib.get('title', ''))
if not matches:
return None
return Rectangle(
float(matches.group(1)), # llx = left
float(matches.group(2)), # lly = top
float(matches.group(3)), # urx = right
float(matches.group(4)), # ury = bottom
)
@classmethod
def baseline(cls, element: Element) -> tuple[float, float]:
"""Get baseline's slope and intercept."""
matches = cls.baseline_pattern.search(element.attrib.get('title', ''))
if not matches:
return (0.0, 0.0)
return float(matches.group(1)), int(matches.group(2))
@classmethod
def textangle(cls, element: Element) -> float:
"""Get text angle of an element."""
matches = cls.textangle_pattern.search(element.attrib.get('title', ''))
if not matches:
return 0.0
return float(matches.group(1))
def _child_xpath(self, html_tag: str, html_class: str | None = None) -> str:
xpath = f".//{self.xmlns}{html_tag}"
if html_class:
xpath += f"[@class='{html_class}']"
return xpath
@classmethod
def normalize_text(cls, s: str) -> str:
"""Normalize the given text using the NFKC normalization form."""
return unicodedata.normalize("NFKC", s)
def to_pdf(
self,
*,
out_filename: Path,
image_filename: Path | None = None,
invisible_text: bool = True,
) -> None:
"""Creates a PDF file with an image superimposed on top of the text.
Text is positioned according to the bounding box of the lines in
the hOCR file.
The image need not be identical to the image used to create the hOCR
file.
It can have a lower resolution, different color mode, etc.
Arguments:
out_filename: Path of PDF to write.
image_filename: Image to use for this file. If omitted, the OCR text
is shown.
invisible_text: If True, text is rendered invisible so that is
selectable but never drawn. If False, text is visible and may
be seen if the image is skipped or deleted in Acrobat.
"""
# create the PDF file
# page size in points (1/72 in.)
canvas = Canvas(page_size=(self.width, self.height))
canvas.add_font(self._fontname, self._font)
page_matrix = (
Matrix()
.translated(0, self.height)
.scaled(1, -1)
.scaled(INCH / self.dpi, INCH / self.dpi)
)
log.debug(page_matrix)
with canvas.do.save_state(cm=page_matrix):
self._debug_draw_paragraph_boxes(canvas)
found_lines = False
for par in self.hocr.iterfind(self._child_xpath('p', 'ocr_par')):
for line in (
element
for element in par.iterfind(self._child_xpath('span'))
if 'class' in element.attrib
and element.attrib['class']
in {'ocr_header', 'ocr_line', 'ocr_textfloat', 'ocr_caption'}
):
found_lines = True
direction = self._get_text_direction(par)
inject_word_breaks = self._get_inject_word_breaks(par)
self._do_line(
canvas,
line,
"ocrx_word",
invisible_text,
direction,
inject_word_breaks,
)
if not found_lines:
# Tesseract did not report any lines (just words)
root = self.hocr.find(self._child_xpath('div', 'ocr_page'))
direction = self._get_text_direction(root)
self._do_line(
canvas,
root,
"ocrx_word",
invisible_text,
direction,
True,
)
# put the image on the page, scaled to fill the page
if image_filename is not None:
canvas.do.draw_image(
image_filename, 0, 0, width=self.width, height=self.height
)
# finish up the page and save it
canvas.to_pdf().save(out_filename)
def _get_text_direction(self, par):
"""Get the text direction of the paragraph.
Arabic, Hebrew, Persian, are right-to-left languages.
When the paragraph element is None, defaults to left-to-right.
"""
if par is None:
return TextDirection.LTR
return (
TextDirection.RTL
if par.attrib.get('dir', 'ltr') == 'rtl'
else TextDirection.LTR
)
def _get_inject_word_breaks(self, par):
"""Determine whether word breaks should be injected.
In Chinese, Japanese, and Korean, word breaks are not injected, because
words are usually one or two characters and separators are usually explicit.
In all other languages, we inject word breaks to help word segmentation.
"""
lang = par.attrib.get('lang', '')
log.debug(lang)
if lang in {'chi_sim', 'chi_tra', 'jpn', 'kor'}:
return False
return True
@classmethod
def polyval(cls, poly, x): # pragma: no cover
"""Calculate the value of a polynomial at a point."""
return x * poly[0] + poly[1]
def _do_line(
self,
canvas: Canvas,
line: Element | None,
elemclass: str,
invisible_text: bool,
text_direction: TextDirection,
inject_word_breaks: bool,
):
"""Render the text for a given line.
The canvas's coordinate system must be configured so that hOCR pixel
coordinates are mapped to PDF coordinates.
"""
if line is None:
return
# line_min_aabb (which is created from the "bbox" hOCR property) is so named
# because a Rectangle instance is always an AABB (it has no orientation).
# However, this means that for non-zero values of the "textangle" hOCR
# property, line_min_aabb is not the true bounding box of the hOCR line,
# but rather the minimum AABB that encloses the bounding box of the line.
# The true bounding box of the line must be seen as an OBB, due to the
# existance of the "textangle" hOCR property.
line_min_aabb = self.element_coordinates(line)
if not line_min_aabb:
return
if line_min_aabb.ury <= line_min_aabb.lly:
log.error(
"line box is invalid so we cannot render it: box=%s text=%s",
line_min_aabb,
self._get_element_text(line),
)
return
self._debug_draw_line_bbox(canvas, line_min_aabb)
# Even though line_min_aabb is not the true bounding box of the line,
# it is still possible to derive an AABB (Rectangle) from it that is
# the same size as the true bounding box of the line,
# if we use a coordinate system that is axis-aligned with respect to
# the rotation of the OBB (textangle).
# line_size_aabb_matrix is a transform matrix for such a coordinate
# system, and line_size_aabb is thus an AABB with the same
# size as the true bounding box of the line.
top_left_corner = (line_min_aabb.llx, line_min_aabb.lly)
line_size_aabb_matrix = (
Matrix()
.translated(*top_left_corner)
# Note: negative sign (textangle is counter-clockwise, see hOCR spec)
.rotated(-self.textangle(line))
)
line_size_aabb = line_size_aabb_matrix.inverse().transform(line_min_aabb)
slope, intercept = self.baseline(line)
if abs(slope) < 0.005:
slope = 0.0
slope_angle = atan(slope)
# Final PDF-perspective (bottom-left corner) transform matrix for the
# text baseline, which has an intercept and slope relative to the OBB.
# See "bbox", "textangle" and "baseline" in the hOCR spec for more details.
baseline_matrix = (
line_size_aabb_matrix
# Translate from hOCR perspective (top-left corner) to PDF perspective
# (bottom-left corner).
# Note: it would be incorrect to use line_min_aabb.height here because
# it is not the true height of the OBB of the line, if textangle != 0.
.translated(0, line_size_aabb.height)
.translated(0, intercept)
.rotated(slope_angle / pi * 180)
)
with canvas.do.save_state(cm=baseline_matrix):
text = Text(direction=text_direction)
fontsize = line_size_aabb.height + intercept
text.font(self._fontname, fontsize)
text.render_mode(3 if invisible_text else 0)
self._debug_draw_baseline(
canvas, baseline_matrix.inverse().transform(line_min_aabb), 0
)
canvas.do.fill_color(BLACK) # text in black
elements = line.findall(self._child_xpath('span', elemclass))
for elem, next_elem in pairwise(elements + [None]):
self._do_line_word(
canvas,
baseline_matrix,
text,
fontsize,
elem,
next_elem,
text_direction,
inject_word_breaks,
)
canvas.do.draw_text(text)
def _do_line_word(
self,
canvas: Canvas,
line_matrix: Matrix,
text: Text,
fontsize: float,
elem: Element | None,
next_elem: Element | None,
text_direction: TextDirection,
inject_word_breaks: bool,
):
"""Render the text for a single word."""
if elem is None:
return
elemtxt = self.normalize_text(self._get_element_text(elem).strip())
if elemtxt == '':
return
hocr_box = self.element_coordinates(elem)
if hocr_box is None:
return
box = line_matrix.inverse().transform(hocr_box)
font_width = self._font.text_width(elemtxt, fontsize)
# Debug sketches
self._debug_draw_word_triangle(canvas, box)
self._debug_draw_word_bbox(canvas, box)
# If this word is 0 units wide, our best bet seems to be to suppress this text
if text_direction == TextDirection.RTL:
log.info("RTL: %s", elemtxt)
if font_width > 0:
if text_direction == TextDirection.LTR:
text.text_transform(Matrix(1, 0, 0, -1, box.llx, 0))
elif text_direction == TextDirection.RTL:
text.text_transform(Matrix(-1, 0, 0, -1, box.llx + box.width, 0))
text.horiz_scale(100 * box.width / font_width)
text.show(self._font.text_encode(elemtxt))
# Get coordinates of the next word (if there is one)
hocr_next_box = (
self.element_coordinates(next_elem) if next_elem is not None else None
)
if hocr_next_box is None:
return
# Render a space between this word and the next word. The explicit space helps
# PDF viewers identify the word break, and horizontally scaling it to
# occupy the space the between the words helps the PDF viewer
# avoid combiningthewordstogether.
if not inject_word_breaks:
return
next_box = line_matrix.inverse().transform(hocr_next_box)
if text_direction == TextDirection.LTR:
space_box = Rectangle(box.urx, box.lly, next_box.llx, next_box.ury)
elif text_direction == TextDirection.RTL:
space_box = Rectangle(next_box.urx, box.lly, box.llx, next_box.ury)
self._debug_draw_space_bbox(canvas, space_box)
space_width = self._font.text_width(' ', fontsize)
if space_width > 0 and space_box.width > 0:
if text_direction == TextDirection.LTR:
text.text_transform(Matrix(1, 0, 0, -1, space_box.llx, 0))
elif text_direction == TextDirection.RTL:
text.text_transform(
Matrix(-1, 0, 0, -1, space_box.llx + space_box.width, 0)
)
text.horiz_scale(100 * space_box.width / space_width)
text.show(self._font.text_encode(' '))
def _debug_draw_paragraph_boxes(self, canvas: Canvas, color=CYAN):
"""Draw boxes around paragraphs in the document."""
if not self.render_options.render_paragraph_bbox: # pragma: no cover
return
with canvas.do.save_state():
# draw box around paragraph
canvas.do.stroke_color(color).line_width(0.1)
for elem in self.hocr.iterfind(self._child_xpath('p', 'ocr_par')):
elemtxt = self._get_element_text(elem).strip()
if len(elemtxt) == 0:
continue
ocr_par = self.element_coordinates(elem)
if ocr_par is None:
continue
canvas.do.rect(
ocr_par.llx, ocr_par.lly, ocr_par.width, ocr_par.height, fill=False
)
def _debug_draw_line_bbox(self, canvas: Canvas, line_box: Rectangle, color=BLUE):
"""Render the bounding box of a text line."""
if not self.render_options.render_line_bbox: # pragma: no cover
return
with canvas.do.save_state():
canvas.do.stroke_color(color).line_width(0.15).rect(
line_box.llx, line_box.lly, line_box.width, line_box.height, fill=False
)
def _debug_draw_word_triangle(
self, canvas: Canvas, box: Rectangle, color=RED, line_width=0.1
):
"""Render a triangle that conveys word height and drawing direction."""
if not self.render_options.render_triangle: # pragma: no cover
return
with canvas.do.save_state():
canvas.do.stroke_color(color).line_width(line_width).line(
box.llx, box.lly, box.urx, box.lly
).line(box.urx, box.lly, box.llx, box.ury).line(
box.llx, box.lly, box.llx, box.ury
)
def _debug_draw_word_bbox(
self, canvas: Canvas, box: Rectangle, color=GREEN, line_width=0.1
):
"""Render a box depicting the word."""
if not self.render_options.render_word_bbox: # pragma: no cover
return
with canvas.do.save_state():
canvas.do.stroke_color(color).line_width(line_width).rect(
box.llx, box.lly, box.width, box.height, fill=False
)
def _debug_draw_space_bbox(
self, canvas: Canvas, box: Rectangle, color=DARKGREEN, line_width=0.1
):
"""Render a box depicting the space between two words."""
if not self.render_options.render_space_bbox: # pragma: no cover
return
with canvas.do.save_state():
canvas.do.fill_color(color).line_width(line_width).rect(
box.llx, box.lly, box.width, box.height, fill=True
)
def _debug_draw_baseline(
self,
canvas: Canvas,
line_box: Rectangle,
baseline_lly,
color=MAGENTA,
line_width=0.25,
):
"""Render the text baseline."""
if not self.render_options.render_baseline:
return
with canvas.do.save_state():
canvas.do.stroke_color(color).line_width(line_width).line(
line_box.llx,
baseline_lly,
line_box.urx,
baseline_lly,
)
src/ocrmypdf/hocrtransform/hocr_parser.py
+521
View File
@@ -0,0 +1,521 @@
# SPDX-FileCopyrightText: 2010 Jonathan Brinley
# SPDX-FileCopyrightText: 2013-2014 Julien Pfefferkorn
# SPDX-FileCopyrightText: 2023-2025 James R. Barlow
# SPDX-License-Identifier: MIT
"""Parser for hOCR format files.
This module provides functionality to parse hOCR files (HTML-based OCR format)
and convert them to the engine-agnostic OcrElement tree structure.
For details of the hOCR format, see:
http://kba.github.io/hocr-spec/1.2/
"""
from __future__ import annotations
import logging
import os
import re
import unicodedata
from pathlib import Path
from typing import Literal, cast
from xml.etree import ElementTree as ET
from ocrmypdf.models.ocr_element import (
Baseline,
BoundingBox,
FontInfo,
OcrClass,
OcrElement,
)
TextDirection = Literal["ltr", "rtl"]
log = logging.getLogger(__name__)
Element = ET.Element
class HocrParseError(Exception):
"""Error while parsing hOCR file."""
class HocrParser:
"""Parser for hOCR format files.
Converts hOCR XML/HTML files into OcrElement trees.
The hOCR format uses HTML with special class attributes (ocr_page, ocr_line,
ocrx_word, etc.) and a title attribute containing properties like bbox,
baseline, and confidence scores.
"""
# Regex patterns for parsing hOCR title attributes
_bbox_pattern = re.compile(
r'''
bbox \s+
(\d+) \s+ # left: uint
(\d+) \s+ # top: uint
(\d+) \s+ # right: uint
(\d+) # bottom: uint
''',
re.VERBOSE,
)
_baseline_pattern = re.compile(
r'''
baseline \s+
([\-\+]?\d*\.?\d*) \s+ # slope: +/- decimal float
([\-\+]?\d+) # intercept: +/- int
''',
re.VERBOSE,
)
_textangle_pattern = re.compile(
r'''
textangle \s+
([\-\+]?\d*\.?\d*) # angle: +/- decimal float
''',
re.VERBOSE,
)
_x_wconf_pattern = re.compile(
r'''
x_wconf \s+
(\d+) # confidence: uint (0-100)
''',
re.VERBOSE,
)
_x_fsize_pattern = re.compile(
r'''
x_fsize \s+
(\d*\.?\d+) # font size: float
''',
re.VERBOSE,
)
_x_font_pattern = re.compile(
r'''
x_font \s+
([^\s;]+) # font name: non-whitespace, non-semicolon string
''',
re.VERBOSE,
)
_ppageno_pattern = re.compile(
r'''
ppageno \s+
(\d+) # page number: uint
''',
re.VERBOSE,
)
_scan_res_pattern = re.compile(
r'''
scan_res \s+
(\d+) \s+ # x resolution
(\d+) # y resolution
''',
re.VERBOSE,
)
def __init__(self, hocr_file: str | Path):
"""Initialize the parser with an hOCR file.
Args:
hocr_file: Path to the hOCR file to parse
Raises:
HocrParseError: If the file cannot be parsed
"""
self._hocr_path = Path(hocr_file)
try:
self._tree = ET.parse(os.fspath(hocr_file))
except ET.ParseError as e:
raise HocrParseError(f"Failed to parse hOCR file: {e}") from e
# Detect XML namespace
root_tag = self._tree.getroot().tag
matches = re.match(r'({.*})html', root_tag)
self._xmlns = matches.group(1) if matches else ''
def parse(self) -> OcrElement:
"""Parse the hOCR file and return an OcrElement tree.
Returns:
The root OcrElement (ocr_page) containing the document structure
Raises:
HocrParseError: If no ocr_page element is found
"""
# Find the first ocr_page element
page_div = self._tree.find(self._xpath('div', 'ocr_page'))
if page_div is None:
raise HocrParseError("No ocr_page element found in hOCR file")
return self._parse_page(page_div)
def _xpath(self, html_tag: str, html_class: str | None = None) -> str:
"""Build an XPath expression for finding elements.
Args:
html_tag: HTML tag name (e.g., 'div', 'span', 'p')
html_class: Optional class attribute to match
Returns:
XPath expression string
"""
xpath = f".//{self._xmlns}{html_tag}"
if html_class:
xpath += f"[@class='{html_class}']"
return xpath
def _parse_page(self, page_elem: Element) -> OcrElement:
"""Parse an ocr_page element.
Args:
page_elem: The XML element with class="ocr_page"
Returns:
OcrElement representing the page
"""
title = page_elem.attrib.get('title', '')
bbox = self._parse_bbox(title)
if bbox is None:
raise HocrParseError("ocr_page missing bbox")
# Parse page-level properties
page_number = self._parse_ppageno(title)
dpi = self._parse_scan_res(title)
page = OcrElement(
ocr_class=OcrClass.PAGE,
bbox=bbox,
page_number=page_number,
dpi=dpi,
)
# Parse child paragraphs
for par_elem in page_elem.iterfind(self._xpath('p', 'ocr_par')):
paragraph = self._parse_paragraph(par_elem)
if paragraph is not None:
page.children.append(paragraph)
# If no paragraphs found, check for words directly under page
# (some Tesseract output structures)
if not page.children:
for word_elem in page_elem.iterfind(self._xpath('span', 'ocrx_word')):
word = self._parse_word(word_elem)
if word is not None:
page.children.append(word)
return page
def _parse_paragraph(self, par_elem: Element) -> OcrElement | None:
"""Parse an ocr_par element.
Args:
par_elem: The XML element with class="ocr_par"
Returns:
OcrElement representing the paragraph, or None if empty
"""
title = par_elem.attrib.get('title', '')
bbox = self._parse_bbox(title)
# Get direction and language from attributes
dir_attr = par_elem.attrib.get('dir')
direction: TextDirection | None = (
cast(TextDirection, dir_attr) if dir_attr in ('ltr', 'rtl') else None
)
language = par_elem.attrib.get('lang')
paragraph = OcrElement(
ocr_class=OcrClass.PARAGRAPH,
bbox=bbox,
direction=direction,
language=language,
)
# Parse child lines
line_classes = {
'ocr_line',
'ocr_header',
'ocr_footer',
'ocr_caption',
'ocr_textfloat',
}
for span_elem in par_elem.iterfind(self._xpath('span')):
elem_class = span_elem.attrib.get('class', '')
if elem_class in line_classes:
line = self._parse_line(span_elem, elem_class, direction, language)
if line is not None:
paragraph.children.append(line)
# Return None if paragraph is empty
if not paragraph.children:
return None
return paragraph
def _parse_line(
self,
line_elem: Element,
ocr_class: str,
parent_direction: TextDirection | None,
parent_language: str | None,
) -> OcrElement | None:
"""Parse a line element (ocr_line, ocr_header, etc.).
Args:
line_elem: The XML element representing the line
ocr_class: The hOCR class of the line
parent_direction: Text direction inherited from parent
parent_language: Language inherited from parent
Returns:
OcrElement representing the line, or None if empty
"""
title = line_elem.attrib.get('title', '')
bbox = self._parse_bbox(title)
if bbox is None:
return None
baseline = self._parse_baseline(title)
textangle = self._parse_textangle(title)
# Inherit direction and language from parent if not specified
dir_attr = line_elem.attrib.get('dir')
if dir_attr in ('ltr', 'rtl'):
direction: TextDirection | None = cast(TextDirection, dir_attr)
else:
direction = parent_direction
language = line_elem.attrib.get('lang') or parent_language
line = OcrElement(
ocr_class=ocr_class,
bbox=bbox,
baseline=baseline,
textangle=textangle,
direction=direction,
language=language,
)
# Parse child words
for word_elem in line_elem.iterfind(self._xpath('span', 'ocrx_word')):
word = self._parse_word(word_elem)
if word is not None:
line.children.append(word)
# Return None if line has no words
if not line.children:
return None
return line
def _parse_word(self, word_elem: Element) -> OcrElement | None:
"""Parse an ocrx_word element.
Args:
word_elem: The XML element with class="ocrx_word"
Returns:
OcrElement representing the word, or None if empty
"""
title = word_elem.attrib.get('title', '')
bbox = self._parse_bbox(title)
# Get the text content
text = self._get_element_text(word_elem)
text = self._normalize_text(text)
if not text:
return None
# Parse confidence (x_wconf is 0-100, convert to 0.0-1.0)
confidence = self._parse_x_wconf(title)
if confidence is not None:
confidence = confidence / 100.0
# Parse font info
font = self._parse_font_info(title)
return OcrElement(
ocr_class=OcrClass.WORD,
bbox=bbox,
text=text,
confidence=confidence,
font=font,
)
def _get_element_text(self, element: Element) -> str:
"""Get the full text content of an element including children.
Args:
element: XML element
Returns:
Combined text content
"""
text = element.text if element.text is not None else ''
for child in element:
text += self._get_element_text(child)
text += element.tail if element.tail is not None else ''
return text
@staticmethod
def _normalize_text(text: str) -> str:
"""Normalize text using NFKC normalization.
This splits ligatures and combines diacritics.
Args:
text: Raw text
Returns:
Normalized text, stripped of leading/trailing whitespace
"""
return unicodedata.normalize("NFKC", text).strip()
def _parse_bbox(self, title: str) -> BoundingBox | None:
"""Parse a bbox from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
BoundingBox or None if not found
"""
match = self._bbox_pattern.search(title)
if not match:
return None
try:
return BoundingBox(
left=float(match.group(1)),
top=float(match.group(2)),
right=float(match.group(3)),
bottom=float(match.group(4)),
)
except ValueError:
return None
def _parse_baseline(self, title: str) -> Baseline | None:
"""Parse baseline from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
Baseline or None if not found
"""
match = self._baseline_pattern.search(title)
if not match:
return None
try:
return Baseline(
slope=float(match.group(1)) if match.group(1) else 0.0,
intercept=float(match.group(2)),
)
except ValueError:
return None
def _parse_textangle(self, title: str) -> float | None:
"""Parse textangle from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
Angle in degrees or None if not found
"""
match = self._textangle_pattern.search(title)
if not match:
return None
try:
return float(match.group(1))
except ValueError:
return None
def _parse_x_wconf(self, title: str) -> float | None:
"""Parse word confidence from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
Confidence (0-100) or None if not found
"""
match = self._x_wconf_pattern.search(title)
if not match:
return None
try:
return float(match.group(1))
except ValueError:
return None
def _parse_ppageno(self, title: str) -> int | None:
"""Parse physical page number from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
Page number or None if not found
"""
match = self._ppageno_pattern.search(title)
if not match:
return None
try:
return int(match.group(1))
except ValueError:
return None
def _parse_scan_res(self, title: str) -> float | None:
"""Parse scan resolution (DPI) from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
DPI (using first value if x and y differ) or None if not found
"""
match = self._scan_res_pattern.search(title)
if not match:
return None
try:
# Use the first (x) resolution value
return float(match.group(1))
except ValueError:
return None
def _parse_font_info(self, title: str) -> FontInfo | None:
"""Parse font information from an hOCR title attribute.
Args:
title: The title attribute value
Returns:
FontInfo or None if no font info found
"""
font_match = self._x_font_pattern.search(title)
size_match = self._x_fsize_pattern.search(title)
if not font_match and not size_match:
return None
return FontInfo(
name=font_match.group(1) if font_match else None,
size=float(size_match.group(1)) if size_match else None,
)
src/ocrmypdf/imageops.py
+4 -5
View File
@@ -60,11 +60,10 @@ def _calculate_downsample(
elif size[1] == 0:
size = min(size[0], max_size[0]), 1
if max_pixels is not None:
if size[0] * size[1] > max_pixels:
log.debug("Resizing image to fit image pixel limit")
pixels_factor = sqrt(max_pixels / (size[0] * size[1]))
size = floor(size[0] * pixels_factor), floor(size[1] * pixels_factor)
if max_pixels is not None and size[0] * size[1] > max_pixels:
log.debug("Resizing image to fit image pixel limit")
pixels_factor = sqrt(max_pixels / (size[0] * size[1]))
size = floor(size[0] * pixels_factor), floor(size[1] * pixels_factor)
if max_bytes is not None:
bpp = bytes_per_pixel
src/ocrmypdf/languages.py
+1
View File
@@ -6,6 +6,7 @@
Derived from
https://www.loc.gov/standards/iso639-2/ascii_8bits.html
"""
from __future__ import annotations
from typing import NamedTuple
src/ocrmypdf/models/__init__.py
+6
View File
@@ -0,0 +1,6 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""OCRmyPDF models for plugin options and cross-cutting concerns."""
from __future__ import annotations
src/ocrmypdf/models/ocr_element.py
+267
View File
@@ -0,0 +1,267 @@
# SPDX-FileCopyrightText: 2025 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""OCR element dataclasses for representing OCR output structure.
This module provides a generic, engine-agnostic representation of OCR output.
The OcrElement dataclass can represent structural units from any OCR source
(hOCR, ALTO, custom engines, etc.) in a unified format suitable for rendering.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Literal
@dataclass
class BoundingBox:
"""An axis-aligned bounding box in pixel coordinates.
Coordinates use top-left origin (standard for images and hOCR).
Attributes:
left: Left edge x-coordinate
top: Top edge y-coordinate
right: Right edge x-coordinate
bottom: Bottom edge y-coordinate
"""
left: float
top: float
right: float
bottom: float
@property
def width(self) -> float:
"""Width of the bounding box."""
return self.right - self.left
@property
def height(self) -> float:
"""Height of the bounding box."""
return self.bottom - self.top
def __post_init__(self):
"""Validate bounding box coordinates."""
if self.right < self.left:
raise ValueError(
f"Invalid bounding box: right ({self.right}) < left ({self.left})"
)
if self.bottom < self.top:
raise ValueError(
f"Invalid bounding box: bottom ({self.bottom}) < top ({self.top})"
)
@dataclass
class Baseline:
"""Text baseline information.
The baseline is represented as a linear equation: y = slope * x + intercept.
This describes the line along which text characters sit, relative to the
bottom-left corner of the line's bounding box.
In hOCR, the baseline is specified relative to the bottom of the line's bbox,
with the intercept being the vertical offset from the bottom and the slope
representing rotation (positive = ascending left-to-right).
Attributes:
slope: Slope of the baseline (rise over run)
intercept: Y-intercept of the baseline (vertical offset from bbox bottom)
"""
slope: float = 0.0
intercept: float = 0.0
@dataclass
class FontInfo:
"""Font information for text rendering.
Attributes:
name: Font family name (e.g., "Times New Roman")
size: Font size in points
bold: Whether the font is bold
italic: Whether the font is italic
monospace: Whether the font is monospace
serif: Whether the font is serif (vs sans-serif)
smallcaps: Whether the font uses small caps
underline: Whether the text is underlined
"""
name: str | None = None
size: float | None = None
bold: bool = False
italic: bool = False
monospace: bool = False
serif: bool = False
smallcaps: bool = False
underline: bool = False
@dataclass
class OcrElement:
"""A generic OCR element representing any structural unit of OCR output.
OcrElements form a tree structure where pages contain paragraphs, paragraphs
contain lines, lines contain words, etc. The specific hierarchy depends on
the OCR engine, but this dataclass can represent any of these levels.
The ocr_class field uses hOCR naming conventions (ocr_page, ocr_par, ocr_line,
ocrx_word, etc.) as a common vocabulary, but elements from other sources can
map to these classes.
Common hOCR classes:
- ocr_page: The root element for a page
- ocr_carea: A content/column area
- ocr_par: A paragraph
- ocr_line: A line of text
- ocr_header: A header line
- ocr_footer: A footer line
- ocr_caption: A caption line
- ocr_textfloat: A floating text element
- ocrx_word: A single word
Attributes:
ocr_class: The element type (e.g., "ocr_page", "ocr_line", "ocrx_word")
bbox: Axis-aligned bounding box in source pixel coordinates (top-left origin)
poly: Polygon vertices for oriented/non-rectangular bounds
text: Text content (primarily for leaf nodes like words)
confidence: OCR confidence score (0.0-1.0)
children: Child elements (hierarchical structure)
direction: Text direction ("ltr" or "rtl")
language: Language code (e.g., "eng", "deu", "chi_sim")
baseline: Text baseline information (slope and intercept)
textangle: Text rotation angle in degrees (counter-clockwise from horizontal)
font: Font information (name, size, style)
dpi: Image resolution in dots per inch (typically for page-level)
page_number: Physical page number (0-indexed)
logical_page_number: Logical page number (as printed on the page)
"""
ocr_class: str
# Bounding boxes
bbox: BoundingBox | None = None
poly: list[tuple[float, float]] | None = None
# Text content
text: str = ""
# Confidence (0.0-1.0)
confidence: float | None = None
# Children (hierarchical structure)
children: list[OcrElement] = field(default_factory=list)
# Text direction and language
direction: Literal["ltr", "rtl"] | None = None
language: str | None = None
# Baseline (for lines)
baseline: Baseline | None = None
# Rotation angle in degrees (counter-clockwise)
textangle: float | None = None
# Font information
font: FontInfo | None = None
# Page-level properties
dpi: float | None = None
page_number: int | None = None
logical_page_number: int | None = None
def iter_by_class(self, *ocr_classes: str) -> list[OcrElement]:
"""Iterate over all descendants matching the given class(es).
Args:
*ocr_classes: One or more ocr_class values to match
Returns:
List of all matching descendant elements (depth-first order)
"""
result = []
if self.ocr_class in ocr_classes:
result.append(self)
for child in self.children:
result.extend(child.iter_by_class(*ocr_classes))
return result
def find_by_class(self, *ocr_classes: str) -> OcrElement | None:
"""Find the first descendant matching the given class(es).
Args:
*ocr_classes: One or more ocr_class values to match
Returns:
The first matching element, or None if not found
"""
if self.ocr_class in ocr_classes:
return self
for child in self.children:
result = child.find_by_class(*ocr_classes)
if result is not None:
return result
return None
def get_text_recursive(self) -> str:
"""Get the combined text of this element and all descendants.
Returns:
Combined text content, with words separated by spaces
"""
if self.text:
return self.text
texts = [child.get_text_recursive() for child in self.children]
return " ".join(t for t in texts if t)
@property
def words(self) -> list[OcrElement]:
"""Get all word elements (ocrx_word) in this element's subtree."""
return self.iter_by_class("ocrx_word")
@property
def lines(self) -> list[OcrElement]:
"""Get all line elements in this element's subtree."""
return self.iter_by_class(
"ocr_line", "ocr_header", "ocr_footer", "ocr_caption", "ocr_textfloat"
)
@property
def paragraphs(self) -> list[OcrElement]:
"""Get all paragraph elements (ocr_par) in this element's subtree."""
return self.iter_by_class("ocr_par")
# Type alias for text direction
TextDirection = Literal["ltr", "rtl"]
# hOCR class constants for convenience
class OcrClass:
"""Constants for common OCR element classes."""
# Page-level
PAGE = "ocr_page"
CAREA = "ocr_carea"
# Block-level
PARAGRAPH = "ocr_par"
# Line-level
LINE = "ocr_line"
HEADER = "ocr_header"
FOOTER = "ocr_footer"
CAPTION = "ocr_caption"
TEXTFLOAT = "ocr_textfloat"
# Word-level
WORD = "ocrx_word"
# Character-level
CHAR = "ocrx_cinfo"
# Line types (for convenience)
LINE_TYPES = frozenset({LINE, HEADER, FOOTER, CAPTION, TEXTFLOAT})
src/ocrmypdf/optimize.py
+68 -111
View File
@@ -9,7 +9,6 @@ import logging
import sys
import tempfile
import threading
from collections import defaultdict
from collections.abc import Callable, Iterator, MutableSet, Sequence
from os import fspath
from pathlib import Path
@@ -19,6 +18,7 @@ from zlib import compress
import img2pdf
from packaging.version import Version
from pikepdf import (
Array,
Dictionary,
Name,
Object,
@@ -194,11 +194,9 @@ def extract_image_jbig2(
def _should_optimize_jpeg(options, filtdp):
if options.optimize >= 2:
return True
if options.optimize < 2 and ghostscript.version() >= Version('10.6.0'):
# Ghostscript 10.6.0+ introduced some sort of JPEG encoding issue.
# To resolve this, re-optimize the JPEG anyway.
return True
return False
# Ghostscript 10.6.0+ introduced some sort of JPEG encoding issue.
# To resolve this, re-optimize the JPEG anyway.
return options.optimize < 2 and ghostscript.version() >= Version('10.6.0')
def extract_image_generic(
@@ -245,11 +243,9 @@ def extract_image_generic(
not pim.indexed
and pim.colorspace == Name.ICCBased
and pim.bits_per_component == 1
and not options.jbig2_lossy
):
# We can losslessly optimize 1-bit images to CCITT or JBIG2 without
# paying any attention to the ICC profile, provided we're not doing
# lossy JBIG2
# paying any attention to the ICC profile
pim.as_pil_image().save(png_name(root, xref))
return XrefExt(xref, '.png')
@@ -372,116 +368,75 @@ def extract_images_generic(
return jpegs, pngs
def extract_images_jbig2(pdf: Pdf, root: Path, options) -> dict[int, list[XrefExt]]:
def extract_images_jbig2(pdf: Pdf, root: Path, options) -> list[XrefExt]:
"""Extract any bitonal image that we think we can improve as JBIG2."""
jbig2_groups = defaultdict(list)
for pageno, xref_ext in extract_images(pdf, root, options, extract_image_jbig2):
group = pageno // options.jbig2_page_group_size
jbig2_groups[group].append(xref_ext)
jbig2_images = []
for _pageno, xref_ext in extract_images(pdf, root, options, extract_image_jbig2):
jbig2_images.append(xref_ext)
log.debug(f"Optimizable images: JBIG2 groups: {len(jbig2_groups)}")
return jbig2_groups
log.debug(f"Optimizable images: JBIG2: {len(jbig2_images)}")
return jbig2_images
def _produce_jbig2_images(
jbig2_groups: dict[int, list[XrefExt]], root: Path, options, executor: Executor
jbig2_images: list[XrefExt], root: Path, options, executor: Executor
) -> None:
"""Produce JBIG2 images from their groups."""
"""Produce JBIG2 images using lossless single-image encoding."""
def jbig2_group_args(root: Path, groups: dict[int, list[XrefExt]]):
for group, xref_exts in groups.items():
prefix = f'group{group:08d}'
def jbig2_args():
for xref_ext in jbig2_images:
xref, ext = xref_ext
yield (
fspath(root), # =cwd
(img_name(root, xref, ext) for xref, ext in xref_exts), # =infiles
prefix, # =out_prefix
fspath(root),
img_name(root, xref, ext),
root / f'{xref:08d}.jbig2',
options.jbig2_threshold,
)
def jbig2_single_args(root: Path, groups: dict[int, list[XrefExt]]):
for group, xref_exts in groups.items():
prefix = f'group{group:08d}'
# Second loop is to ensure multiple images per page are unpacked
for n, xref_ext in enumerate(xref_exts):
xref, ext = xref_ext
yield (
fspath(root),
img_name(root, xref, ext),
root / f'{prefix}.{n:04d}',
options.jbig2_threshold,
)
if options.jbig2_page_group_size > 1:
jbig2_args = jbig2_group_args
jbig2_convert = jbig2enc.convert_group
else:
jbig2_args = jbig2_single_args
jbig2_convert = jbig2enc.convert_single
executor(
use_threads=True,
max_workers=options.jobs,
progress_kwargs=dict(
total=len(jbig2_groups),
total=len(jbig2_images),
desc="JBIG2",
unit='item',
unit='image',
disable=not options.progress_bar,
),
task=jbig2_convert,
task_arguments=jbig2_args(root, jbig2_groups),
task=jbig2enc.convert_single,
task_arguments=jbig2_args(),
)
def convert_to_jbig2(
pdf: Pdf,
jbig2_groups: dict[int, list[XrefExt]],
jbig2_images: list[XrefExt],
root: Path,
options,
executor: Executor,
) -> None:
"""Convert images to JBIG2 and insert into PDF.
When the JBIG2 page group size is > 1 we do several JBIG2 images at once
and build a symbol dictionary that will span several pages. Each JBIG2
image must reference to its symbol dictionary. If too many pages shared the
same dictionary JBIG2 encoding becomes more expensive and less efficient.
The default value of 10 was determined through testing. Currently this
must be lossy encoding since jbig2enc does not support refinement coding.
When the JBIG2 symbolic coder is not used, each JBIG2 stands on its own
and needs no dictionary. Currently this must be lossless JBIG2.
Each JBIG2 image is encoded independently using lossless compression.
No symbol dictionary (JBIG2Globals) is used.
"""
jbig2_globals_dict: Dictionary | None
_produce_jbig2_images(jbig2_images, root, options, executor)
_produce_jbig2_images(jbig2_groups, root, options, executor)
for group, xref_exts in jbig2_groups.items():
prefix = f'group{group:08d}'
jbig2_symfile = root / (prefix + '.sym')
if jbig2_symfile.exists():
jbig2_globals_data = jbig2_symfile.read_bytes()
jbig2_globals = Stream(pdf, jbig2_globals_data)
jbig2_globals_dict = Dictionary(JBIG2Globals=jbig2_globals)
elif options.jbig2_page_group_size == 1:
jbig2_globals_dict = None
else:
raise FileNotFoundError(jbig2_symfile)
for n, xref_ext in enumerate(xref_exts):
xref, _ = xref_ext
jbig2_im_file = root / (prefix + f'.{n:04d}')
jbig2_im_data = jbig2_im_file.read_bytes()
im_obj = pdf.get_object(xref, 0)
im_obj.write(
jbig2_im_data, filter=Name.JBIG2Decode, decode_parms=jbig2_globals_dict
)
for xref_ext in jbig2_images:
xref, _ = xref_ext
jbig2_im_file = root / f'{xref:08d}.jbig2'
jbig2_im_data = jbig2_im_file.read_bytes()
im_obj = pdf.get_object(xref, 0)
im_obj.write(jbig2_im_data, filter=Name.JBIG2Decode, decode_parms=None)
def _optimize_jpeg(
xref: Xref, in_jpg: Path, opt_jpg: Path, jpeg_quality: int
xref: Xref, in_jpg: Path, opt_jpg: Path, jpg_quality: int
) -> tuple[Xref, Path | None]:
with Image.open(in_jpg) as im:
im.save(opt_jpg, optimize=True, quality=jpeg_quality)
save_kwargs: dict[str, Any] = {'optimize': True}
if isinstance(jpg_quality, int) and 0 < jpg_quality <= 100:
save_kwargs['quality'] = jpg_quality
im.save(opt_jpg, **save_kwargs)
if opt_jpg.stat().st_size > in_jpg.stat().st_size:
log.debug(f"xref {xref}, jpeg, made larger - skip")
@@ -499,7 +454,7 @@ def transcode_jpegs(
for xref in jpegs:
in_jpg = jpg_name(root, xref)
opt_jpg = in_jpg.with_suffix('.opt.jpg')
yield xref, in_jpg, opt_jpg, options.jpeg_quality
yield xref, in_jpg, opt_jpg, options.jpg_quality
def finish_jpeg(result: tuple[Xref, Path | None], pbar: ProgressBar):
xref, opt_jpg = result
@@ -524,6 +479,16 @@ def transcode_jpegs(
)
def _already_flate_encoded(image: Stream) -> bool:
"""Check if the image already has FlateDecode in its filter chain."""
filt = image.get(Name.Filter)
if filt is None:
return False
if isinstance(filt, Array):
return Name.FlateDecode in list(filt)
return filt == Name.FlateDecode
def _find_deflatable_jpeg(
*, pdf: Pdf, root: Path, image: Stream, xref: Xref, options
) -> XrefExt | None:
@@ -532,6 +497,10 @@ def _find_deflatable_jpeg(
return None
_pim, filtdp = result
# Skip if already FlateDecode compressed - would double-compress
if _already_flate_encoded(image):
return None
if (
filtdp[0] == Name.DCTDecode
and not filtdp[1]
@@ -712,12 +681,10 @@ def optimize(
safe_symlink(input_file, output_file)
return output_file
if options.jpeg_quality == 0:
options.jpeg_quality = DEFAULT_JPEG_QUALITY if options.optimize < 3 else 40
if options.jpg_quality == 0:
options.jpg_quality = DEFAULT_JPEG_QUALITY if options.optimize < 3 else 40
if options.png_quality == 0:
options.png_quality = DEFAULT_PNG_QUALITY if options.optimize < 3 else 30
if options.jbig2_page_group_size == 0:
options.jbig2_page_group_size = 10 if options.jbig2_lossy else 1
with Pdf.open(input_file) as pdf:
root = output_file.parent / 'images'
@@ -731,8 +698,8 @@ def optimize(
# transcode_pngs(pdf, jpegs, jpg_name, root, options)
transcode_pngs(pdf, pngs, png_name, root, options, executor)
jbig2_groups = extract_images_jbig2(pdf, root, options)
convert_to_jbig2(pdf, jbig2_groups, root, options, executor)
jbig2_images = extract_images_jbig2(pdf, root, options)
convert_to_jbig2(pdf, jbig2_images, root, options, executor)
target_file = output_file.with_suffix('.opt.pdf')
pdf.remove_unreferenced_resources()
@@ -767,35 +734,25 @@ def main(infile, outfile, level, jobs=1):
from shutil import copy # pylint: disable=import-outside-toplevel
from tempfile import TemporaryDirectory # pylint: disable=import-outside-toplevel
class OptimizeOptions:
"""Emulate ocrmypdf's options."""
def __init__(
self, input_file, jobs, optimize_, jpeg_quality, png_quality, jb2lossy
):
self.input_file = input_file
self.jobs = jobs
self.optimize = optimize_
self.jpeg_quality = jpeg_quality
self.png_quality = png_quality
self.jbig2_page_group_size = 0
self.jbig2_lossy = jb2lossy
self.jbig2_threshold = 0.85
self.quiet = True
self.progress_bar = False
from ocrmypdf._options import OcrOptions # pylint: disable=import-outside-toplevel
infile = Path(infile)
options = OptimizeOptions(
# Create OcrOptions with optimization-specific settings
options = OcrOptions(
input_file=infile,
output_file=outfile, # Required field
jobs=jobs,
optimize_=int(level),
jpeg_quality=0, # Use default
optimize=int(level),
jpg_quality=0, # Use default
png_quality=0,
jb2lossy=False,
jbig2_threshold=0.85,
quiet=True,
progress_bar=False,
)
with TemporaryDirectory() as tmpdir:
context = PdfContext(options, tmpdir, infile, None, None)
context = PdfContext(options, Path(tmpdir), infile, None, None)
tmpout = Path(tmpdir) / 'out.pdf'
optimize(
infile,
src/ocrmypdf/pdfa.py
+117 -1
View File
@@ -1,16 +1,20 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""Utilities for PDF/A production and confirmation with Ghostspcript."""
"""Utilities for PDF/A production and confirmation with Ghostscript."""
from __future__ import annotations
import base64
import logging
from collections.abc import Iterator
from importlib.resources import files as package_files
from pathlib import Path
import pikepdf
from pikepdf import Array, Dictionary, Name, Pdf, Stream
log = logging.getLogger(__name__)
SRGB_ICC_PROFILE_NAME = 'sRGB.icc'
@@ -131,3 +135,115 @@ def file_claims_pdfa(filename: Path):
pdfa_dict['output'] = 'pdfa'
pdfa_dict['conformance'] = conformance
return pdfa_dict
def _load_srgb_icc_profile() -> bytes:
"""Load the sRGB ICC profile from package data."""
return (package_files('ocrmypdf.data') / SRGB_ICC_PROFILE_NAME).read_bytes()
def _pdfa_part_conformance(output_type: str) -> tuple[str, str]:
"""Extract PDF/A part and conformance from output_type.
Args:
output_type: One of 'pdfa', 'pdfa-1', 'pdfa-2', 'pdfa-3'
Returns:
Tuple of (part, conformance) e.g., ('2', 'B')
"""
mapping = {
'pdfa': ('2', 'B'),
'pdfa-1': ('1', 'B'),
'pdfa-2': ('2', 'B'),
'pdfa-3': ('3', 'B'),
}
return mapping.get(output_type, ('2', 'B'))
def add_pdfa_metadata(pdf: Pdf, part: str, conformance: str) -> None:
"""Add PDF/A XMP metadata declaration to a PDF.
Args:
pdf: An open pikepdf.Pdf object
part: PDF/A part number ('1', '2', or '3')
conformance: Conformance level ('A', 'B', or 'U')
"""
with pdf.open_metadata() as meta:
meta['pdfaid:part'] = part
meta['pdfaid:conformance'] = conformance
def add_srgb_output_intent(pdf: Pdf) -> None:
"""Add sRGB ICC profile as OutputIntent to PDF catalog.
This creates the required PDF/A OutputIntent structure with:
- An ICC profile stream containing sRGB profile
- An OutputIntent dictionary pointing to that profile
- Updates the Catalog's OutputIntents array
Args:
pdf: An open pikepdf.Pdf object
"""
icc_data = _load_srgb_icc_profile()
# Create ICC profile stream
icc_stream = Stream(pdf, icc_data)
icc_stream[Name.N] = 3 # RGB has 3 components
# Create OutputIntent dictionary
output_intent = Dictionary({
'/Type': Name.OutputIntent,
'/S': Name('/GTS_PDFA1'),
'/OutputConditionIdentifier': 'sRGB',
'/DestOutputProfile': icc_stream,
})
# Add to catalog's OutputIntents array
if Name.OutputIntents not in pdf.Root:
pdf.Root[Name.OutputIntents] = Array([])
# Check if sRGB OutputIntent already exists
for intent in pdf.Root.OutputIntents: # type: ignore[attr-defined]
if str(intent.get(Name.OutputConditionIdentifier)) == 'sRGB':
log.debug('sRGB OutputIntent already exists, skipping')
return
pdf.Root.OutputIntents.append(output_intent)
def speculative_pdfa_conversion(
input_file: Path,
output_file: Path,
output_type: str,
) -> Path:
"""Attempt to convert a PDF to PDF/A by adding required structures.
This function creates a copy of the input PDF and adds:
1. sRGB ICC profile as OutputIntent
2. XMP metadata declaring PDF/A conformance
This approach works for PDFs that are already mostly PDF/A compliant
but lack the formal declarations. It does NOT perform color conversion,
font embedding, or other transformations that Ghostscript does.
Args:
input_file: Path to input PDF
output_file: Path where output PDF should be written
output_type: One of 'pdfa', 'pdfa-1', 'pdfa-2', 'pdfa-3'
Returns:
Path to the output file
Raises:
pikepdf.PdfError: If the PDF cannot be opened or modified
"""
part, conformance = _pdfa_part_conformance(output_type)
with Pdf.open(input_file) as pdf:
add_srgb_output_intent(pdf)
add_pdfa_metadata(pdf, part, conformance)
pdf.save(output_file)
log.debug('Speculative PDF/A conversion complete: %s', output_file)
return output_file
src/ocrmypdf/pdfinfo/__init__.py
+3 -2
View File
@@ -6,6 +6,7 @@
from __future__ import annotations
from ocrmypdf.pdfinfo.info import Colorspace, Encoding, PageInfo, PdfInfo
from ocrmypdf.pdfinfo._types import Colorspace, Encoding, FloatRect
from ocrmypdf.pdfinfo.info import PageInfo, PdfInfo
__all__ = ["Colorspace", "Encoding", "PageInfo", "PdfInfo"]
__all__ = ["Colorspace", "Encoding", "FloatRect", "PageInfo", "PdfInfo"]
src/ocrmypdf/pdfinfo/_contentstream.py
+231
View File
@@ -0,0 +1,231 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""PDF content stream interpretation."""
from __future__ import annotations
import re
from collections import defaultdict
from collections.abc import Mapping
from math import hypot, inf, isclose
from typing import NamedTuple
from warnings import warn
from pikepdf import Matrix, Object, PdfInlineImage, parse_content_stream
from ocrmypdf.exceptions import InputFileError
from ocrmypdf.helpers import Resolution
from ocrmypdf.pdfinfo._types import UNIT_SQUARE
class XobjectSettings(NamedTuple):
"""Info about an XObject found in a PDF."""
name: str
shorthand: tuple[float, float, float, float, float, float]
stack_depth: int
class InlineSettings(NamedTuple):
"""Info about an inline image found in a PDF."""
iimage: PdfInlineImage
shorthand: tuple[float, float, float, float, float, float]
stack_depth: int
class ContentsInfo(NamedTuple):
"""Info about various objects found in a PDF."""
xobject_settings: list[XobjectSettings]
inline_images: list[InlineSettings]
found_vector: bool
found_text: bool
name_index: Mapping[str, list[XobjectSettings]]
class TextboxInfo(NamedTuple):
"""Info about a text box found in a PDF."""
bbox: tuple[float, float, float, float]
is_visible: bool
is_corrupt: bool
class VectorMarker:
"""Sentinel indicating vector drawing operations were found on a page."""
class TextMarker:
"""Sentinel indicating text drawing operations were found on a page."""
def _is_unit_square(shorthand):
"""Check if the shorthand represents a unit square transformation."""
values = map(float, shorthand)
pairwise = zip(values, UNIT_SQUARE, strict=False)
return all(isclose(a, b, rel_tol=1e-3) for a, b in pairwise)
def _normalize_stack(graphobjs):
"""Convert runs of qQ's in the stack into single graphobjs."""
for operands, operator in graphobjs:
operator = str(operator)
if re.match(r'Q*q+$', operator): # Zero or more Q, one or more q
for char in operator: # Split into individual
yield ([], char) # Yield individual
else:
yield (operands, operator)
def _interpret_contents(contentstream: Object, initial_shorthand=UNIT_SQUARE):
"""Interpret the PDF content stream.
The stack represents the state of the PDF graphics stack. We are only
interested in the current transformation matrix (CTM) so we only track
this object; a full implementation would need to track many other items.
The CTM is initialized to the mapping from user space to device space.
PDF units are 1/72". In a PDF viewer or printer this matrix is initialized
to the transformation to device space. For example if set to
(1/72, 0, 0, 1/72, 0, 0) then all units would be calculated in inches.
Images are always considered to be (0, 0) -> (1, 1). Before drawing an
image there should be a 'cm' that sets up an image coordinate system
where drawing from (0, 0) -> (1, 1) will draw on the desired area of the
page.
PDF units suit our needs so we initialize ctm to the identity matrix.
According to the PDF specification, the maximum stack depth is 32. Other
viewers tolerate some amount beyond this. We issue a warning if the
stack depth exceeds the spec limit and set a hard limit beyond this to
bound our memory requirements. If the stack underflows behavior is
undefined in the spec, but we just pretend nothing happened and leave the
CTM unchanged.
"""
stack = []
ctm = Matrix(initial_shorthand)
xobject_settings: list[XobjectSettings] = []
inline_images: list[InlineSettings] = []
name_index = defaultdict(lambda: [])
found_vector = False
found_text = False
vector_ops = set('S s f F f* B B* b b*'.split())
text_showing_ops = set("""TJ Tj " '""".split())
image_ops = set('BI ID EI q Q Do cm'.split())
operator_whitelist = ' '.join(vector_ops | text_showing_ops | image_ops)
for n, graphobj in enumerate(
_normalize_stack(parse_content_stream(contentstream, operator_whitelist))
):
operands, operator = graphobj
if operator == 'q':
stack.append(ctm)
if len(stack) > 32: # See docstring
if len(stack) > 128:
raise RuntimeError(
f"PDF graphics stack overflowed hard limit at operator {n}"
)
warn("PDF graphics stack overflowed spec limit")
elif operator == 'Q':
try:
ctm = stack.pop()
except IndexError:
# Keeping the ctm the same seems to be the only sensible thing
# to do. Just pretend nothing happened, keep calm and carry on.
warn("PDF graphics stack underflowed - PDF may be malformed")
elif operator == 'cm':
try:
ctm = Matrix(operands) @ ctm
except ValueError as e:
raise InputFileError(
"PDF content stream is corrupt - this PDF is malformed. "
"Use a PDF editor that is capable of visually inspecting the PDF."
) from e
elif operator == 'Do':
image_name = operands[0]
settings = XobjectSettings(
name=image_name, shorthand=ctm.shorthand, stack_depth=len(stack)
)
xobject_settings.append(settings)
name_index[str(image_name)].append(settings)
elif operator == 'INLINE IMAGE': # BI/ID/EI are grouped into this
iimage = operands[0]
inline = InlineSettings(
iimage=iimage, shorthand=ctm.shorthand, stack_depth=len(stack)
)
inline_images.append(inline)
elif operator in vector_ops:
found_vector = True
elif operator in text_showing_ops:
found_text = True
return ContentsInfo(
xobject_settings=xobject_settings,
inline_images=inline_images,
found_vector=found_vector,
found_text=found_text,
name_index=name_index,
)
def _get_dpi(ctm_shorthand, image_size) -> Resolution:
"""Given the transformation matrix and image size, find the image DPI.
PDFs do not include image resolution information within image data.
Instead, the PDF page content stream describes the location where the
image will be rasterized, and the effective resolution is the ratio of the
pixel size to raster target size.
Normally a scanned PDF has the paper size set appropriately but this is
not guaranteed. The most common case is a cropped image will change the
page size (/CropBox) without altering the page content stream. That means
it is not sufficient to assume that the image fills the page, even though
that is the most common case.
A PDF image may be scaled (always), cropped, translated, rotated in place
to an arbitrary angle (rarely) and skewed. Only equal area mappings can
be expressed, that is, it is not necessary to consider distortions where
the effective DPI varies with position.
To determine the image scale, transform an offset axis vector v0 (0, 0),
width-axis vector v0 (1, 0), height-axis vector vh (0, 1) with the matrix,
which gives the dimensions of the image in PDF units. From there we can
compare to actual image dimensions. PDF uses
row vector * matrix_transposed unlike the traditional
matrix * column vector.
The offset, width and height vectors can be combined in a matrix and
multiplied by the transform matrix. Then we want to calculated
magnitude(width_vector - offset_vector)
and
magnitude(height_vector - offset_vector)
When the above is worked out algebraically, the effect of translation
cancels out, and the vector magnitudes become functions of the nonzero
transformation matrix indices. The results of the derivation are used
in this code.
pdfimages -list does calculate the DPI in some way that is not completely
naive, but it does not get the DPI of rotated images right, so cannot be
used anymore to validate this. Photoshop works, or using Acrobat to
rotate the image back to normal.
It does not matter if the image is partially cropped, or even out of the
/MediaBox.
"""
a, b, c, d, _, _ = ctm_shorthand # pylint: disable=invalid-name
# Calculate the width and height of the image in PDF units
image_drawn = hypot(a, b), hypot(c, d)
def calc(drawn, pixels, inches_per_pt=72.0):
# The scale of the image is pixels per unit of default user space (1/72")
scale = pixels / drawn if drawn != 0 else inf
dpi = scale * inches_per_pt
return dpi
dpi_w, dpi_h = (calc(image_drawn[n], image_size[n]) for n in range(2))
return Resolution(dpi_w, dpi_h)
src/ocrmypdf/pdfinfo/_image.py
+380
View File
@@ -0,0 +1,380 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""PDF image analysis."""
from __future__ import annotations
import logging
from collections.abc import Iterator
from decimal import Decimal
from pikepdf import (
Dictionary,
Matrix,
Name,
Object,
Pdf,
PdfImage,
PdfInlineImage,
Stream,
UnsupportedImageTypeError,
)
from ocrmypdf.helpers import Resolution
from ocrmypdf.pdfinfo._contentstream import (
ContentsInfo,
TextMarker,
VectorMarker,
_get_dpi,
_interpret_contents,
_is_unit_square,
)
from ocrmypdf.pdfinfo._types import (
FRIENDLY_COLORSPACE,
FRIENDLY_COMP,
FRIENDLY_ENCODING,
UNIT_SQUARE,
Colorspace,
Encoding,
)
logger = logging.getLogger()
class ImageInfo:
"""Information about an image found in a PDF.
This gathers information from pikepdf and pdfminer.six, and is pickle-able
so that it can be passed to a worker process, unlike objects from those
libraries.
"""
DPI_PREC = Decimal('1.000')
_comp: int | None
_name: str
def __init__(
self,
*,
name='',
pdfimage: Object | None = None,
inline: PdfInlineImage | None = None,
shorthand=None,
):
"""Initialize an ImageInfo."""
self._name = str(name)
self._shorthand = shorthand
pim: PdfInlineImage | PdfImage
if inline is not None:
self._origin = 'inline'
pim = inline
elif pdfimage is not None and isinstance(pdfimage, Stream):
self._origin = 'xobject'
pim = PdfImage(pdfimage)
else:
raise ValueError("Either pdfimage or inline must be set")
self._width = pim.width
self._height = pim.height
if (smask := pim.obj.get(Name.SMask, None)) is not None and isinstance(
smask, Stream | Dictionary
):
# SMask is pretty much an alpha channel, but in PDF it's possible
# for channel to have different dimensions than the image
# itself. Some PDF writers use this to create a grayscale stencil
# mask. For our purposes, the effective size is the size of the
# larger component (image or smask).
self._width = max(smask.get(Name.Width, 0), self._width)
self._height = max(smask.get(Name.Height, 0), self._height)
if (mask := pim.obj.get(Name.Mask, None)) is not None and isinstance(
mask, Stream | Dictionary
):
# If the image has a /Mask entry, it has an explicit mask.
# /Mask can be a Stream or an Array. If it's a Stream,
# use its /Width and /Height if they are larger than the main
# image's.
self._width = max(mask.get(Name.Width, 0), self._width)
self._height = max(mask.get(Name.Height, 0), self._height)
# If /ImageMask is true, then this image is a stencil mask
# (Images that draw with this stencil mask will have a reference to
# it in their /Mask, but we don't actually need that information)
if pim.image_mask:
self._type = 'stencil'
else:
self._type = 'image'
self._bpc = int(pim.bits_per_component)
try:
self._enc = FRIENDLY_ENCODING.get(pim.filters[0])
except IndexError:
self._enc = None
try:
self._color = FRIENDLY_COLORSPACE.get(pim.colorspace or '')
except NotImplementedError:
self._color = None
if self._enc == Encoding.jpeg2000:
self._color = Colorspace.jpeg2000
self._comp = None
if self._color == Colorspace.icc and isinstance(pim, PdfImage):
self._comp = self._init_icc(pim)
else:
if isinstance(self._color, Colorspace):
self._comp = FRIENDLY_COMP.get(self._color)
# Bit of a hack... infer grayscale if component count is uncertain
# but encoding only supports monochrome.
if self._comp is None and self._enc in (Encoding.ccitt, Encoding.jbig2):
self._comp = FRIENDLY_COMP[Colorspace.gray]
def _init_icc(self, pim: PdfImage):
try:
icc = pim.icc
except UnsupportedImageTypeError as e:
logger.warning(
f"An image with a corrupt or unreadable ICC profile was found. "
f"Output PDF may not match the input PDF visually: {e}. {self}"
)
return None
# Check the ICC profile to determine actual colorspace
if icc is None or not hasattr(icc, 'profile'):
logger.warning(
f"An image with an ICC profile but no ICC profile data was found. "
f"The output PDF may not match the input PDF visually. {self}"
)
return None
try:
if icc.profile.xcolor_space == 'GRAY':
return 1
elif icc.profile.xcolor_space == 'CMYK':
return 4
else:
return 3
except AttributeError:
return None
@property
def name(self):
"""Name of the image as it appears in the PDF."""
return self._name
@property
def type_(self):
"""Type of image, either 'image' or 'stencil'."""
return self._type
@property
def width(self) -> int:
"""Width of the image in pixels."""
return self._width
@property
def height(self) -> int:
"""Height of the image in pixels."""
return self._height
@property
def bpc(self):
"""Bits per component."""
return self._bpc
@property
def color(self):
"""Colorspace of the image."""
return self._color if self._color is not None else '?'
@property
def comp(self):
"""Number of components/channels in the image."""
return self._comp if self._comp is not None else '?'
@property
def enc(self):
"""Encoding of the image."""
return self._enc if self._enc is not None else 'image'
@property
def renderable(self) -> bool:
"""Whether the image is renderable.
Some PDFs in the wild have invalid images that are not renderable,
due to unusual dimensions.
Stencil masks are not also not renderable, since they are not
drawn, but rather they control how rendering happens.
"""
return (
self.dpi.is_finite
and self.width >= 0
and self.height >= 0
and self.type_ != 'stencil'
)
@property
def dpi(self) -> Resolution:
"""Dots per inch of the image.
Calculated based on where and how the image is drawn in the PDF.
"""
return _get_dpi(self._shorthand, (self._width, self._height))
@property
def printed_area(self) -> float:
"""Physical area of the image in square inches."""
if not self.renderable:
return 0.0
return float((self.width / self.dpi.x) * (self.height / self.dpi.y))
def __repr__(self):
"""Return a string representation of the image."""
return (
f"<ImageInfo '{self.name}' {self.type_} {self.width}×{self.height} "
f"{self.color} {self.comp} {self.bpc} {self.enc} {self.dpi}>"
)
def _find_inline_images(contentsinfo: ContentsInfo) -> Iterator[ImageInfo]:
"""Find inline images in the contentstream."""
for n, inline in enumerate(contentsinfo.inline_images):
yield ImageInfo(
name=f'inline-{n:02d}', shorthand=inline.shorthand, inline=inline.iimage
)
def _image_xobjects(container) -> Iterator[tuple[Object, str]]:
"""Search for all XObject-based images in the container.
Usually the container is a page, but it could also be a Form XObject
that contains images. Filter out the Form XObjects which are dealt with
elsewhere.
Generate a sequence of tuples (image, xobj container), where container,
where xobj is the name of the object and image is the object itself,
since the object does not know its own name.
"""
if Name.Resources not in container:
return
resources = container[Name.Resources]
if Name.XObject not in resources:
return
for key, candidate in resources[Name.XObject].items():
if candidate is None or Name.Subtype not in candidate:
continue
if candidate[Name.Subtype] == Name.Image:
pdfimage = candidate
yield (pdfimage, key)
def _find_regular_images(
container: Object, contentsinfo: ContentsInfo
) -> Iterator[ImageInfo]:
"""Find images stored in the container's /Resources /XObject.
Usually the container is a page, but it could also be a Form XObject
that contains images.
Generates images with their DPI at time of drawing.
"""
for pdfimage, xobj in _image_xobjects(container):
if xobj not in contentsinfo.name_index:
continue
for draw in contentsinfo.name_index[xobj]:
if draw.stack_depth == 0 and _is_unit_square(draw.shorthand):
# At least one PDF in the wild (and test suite) draws an image
# when the graphics stack depth is 0, meaning that the image
# gets drawn into a square of 1x1 PDF units (or 1/72",
# or 0.35 mm). The equivalent DPI will be >100,000. Exclude
# these from our DPI calculation for the page.
continue
yield ImageInfo(name=draw.name, pdfimage=pdfimage, shorthand=draw.shorthand)
def _find_form_xobject_images(pdf: Pdf, container: Object, contentsinfo: ContentsInfo):
"""Find any images that are in Form XObjects in the container.
The container may be a page, or a parent Form XObject.
"""
if Name.Resources not in container:
return
resources = container[Name.Resources]
if Name.XObject not in resources:
return
xobjs = resources[Name.XObject].as_dict()
for xobj in xobjs:
candidate = xobjs[xobj]
if candidate is None or candidate.get(Name.Subtype) != Name.Form:
continue
form_xobject = candidate
for settings in contentsinfo.xobject_settings:
if settings.name != xobj:
continue
# Find images once for each time this Form XObject is drawn.
# This could be optimized to cache the multiple drawing events
# but in practice both Form XObjects and multiple drawing of the
# same object are both very rare.
ctm_shorthand = settings.shorthand
yield from _process_content_streams(
pdf=pdf, container=form_xobject, shorthand=ctm_shorthand
)
def _process_content_streams(
*, pdf: Pdf, container: Object, shorthand=None
) -> Iterator[VectorMarker | TextMarker | ImageInfo]:
"""Find all individual instances of images drawn in the container.
Usually the container is a page, but it may also be a Form XObject.
On a typical page images are stored inline or as regular images
in an XObject.
Form XObjects may include inline images, XObject images,
and recursively, other Form XObjects; and also vector graphic objects.
Every instance of an image being drawn somewhere is flattened and
treated as a unique image, since if the same image is drawn multiple times
on one page it may be drawn at differing resolutions, and our objective
is to find the resolution at which the page can be rastered without
downsampling.
"""
if container.get(Name.Type) == Name.Page and Name.Contents in container:
initial_shorthand = shorthand or UNIT_SQUARE
elif (
container.get(Name.Type) == Name.XObject
and container[Name.Subtype] == Name.Form
):
# Set the CTM to the state it was when the "Do" operator was
# encountered that is drawing this instance of the Form XObject
ctm = Matrix(shorthand) if shorthand else Matrix()
# A Form XObject may provide its own matrix to map form space into
# user space. Get this if one exists
form_shorthand = container.get(Name.Matrix, Matrix())
form_matrix = Matrix(form_shorthand)
# Concatenate form matrix with CTM to ensure CTM is correct for
# drawing this instance of the XObject
ctm = form_matrix @ ctm
initial_shorthand = ctm.shorthand
else:
return
contentsinfo = _interpret_contents(container, initial_shorthand)
if contentsinfo.found_vector:
yield VectorMarker()
if contentsinfo.found_text:
yield TextMarker()
yield from _find_inline_images(contentsinfo)
yield from _find_regular_images(container, contentsinfo)
yield from _find_form_xobject_images(pdf, container, contentsinfo)
src/ocrmypdf/pdfinfo/_types.py
+83
View File
@@ -0,0 +1,83 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""PDF type definitions and constants."""
from __future__ import annotations
from enum import Enum, auto
class Colorspace(Enum):
"""Description of common image colorspaces in a PDF."""
# pylint: disable=invalid-name
gray = auto()
rgb = auto()
cmyk = auto()
lab = auto()
icc = auto()
index = auto()
sep = auto()
devn = auto()
pattern = auto()
jpeg2000 = auto()
class Encoding(Enum):
"""Description of common image encodings in a PDF."""
# pylint: disable=invalid-name
ccitt = auto()
jpeg = auto()
jpeg2000 = auto()
jbig2 = auto()
asciihex = auto()
ascii85 = auto()
lzw = auto()
flate = auto()
runlength = auto()
FloatRect = tuple[float, float, float, float]
FRIENDLY_COLORSPACE: dict[str, Colorspace] = {
'/DeviceGray': Colorspace.gray,
'/CalGray': Colorspace.gray,
'/DeviceRGB': Colorspace.rgb,
'/CalRGB': Colorspace.rgb,
'/DeviceCMYK': Colorspace.cmyk,
'/Lab': Colorspace.lab,
'/ICCBased': Colorspace.icc,
'/Indexed': Colorspace.index,
'/Separation': Colorspace.sep,
'/DeviceN': Colorspace.devn,
'/Pattern': Colorspace.pattern,
'/G': Colorspace.gray, # Abbreviations permitted in inline images
'/RGB': Colorspace.rgb,
'/CMYK': Colorspace.cmyk,
'/I': Colorspace.index,
}
FRIENDLY_ENCODING: dict[str, Encoding] = {
'/CCITTFaxDecode': Encoding.ccitt,
'/DCTDecode': Encoding.jpeg,
'/JPXDecode': Encoding.jpeg2000,
'/JBIG2Decode': Encoding.jbig2,
'/CCF': Encoding.ccitt, # Abbreviations permitted in inline images
'/DCT': Encoding.jpeg,
'/AHx': Encoding.asciihex,
'/A85': Encoding.ascii85,
'/LZW': Encoding.lzw,
'/Fl': Encoding.flate,
'/RL': Encoding.runlength,
}
FRIENDLY_COMP: dict[Colorspace, int] = {
Colorspace.gray: 1,
Colorspace.rgb: 3,
Colorspace.cmyk: 4,
Colorspace.lab: 3,
Colorspace.index: 1,
}
UNIT_SQUARE = (1.0, 0.0, 0.0, 1.0, 0.0, 0.0)
src/ocrmypdf/pdfinfo/_worker.py
+143
View File
@@ -0,0 +1,143 @@
# SPDX-FileCopyrightText: 2022 James R. Barlow
# SPDX-License-Identifier: MPL-2.0
"""PDF page info worker process handling."""
from __future__ import annotations
import atexit
import logging
from collections.abc import Container, Sequence
from contextlib import contextmanager
from functools import partial
from pathlib import Path
from typing import TYPE_CHECKING
from pikepdf import Pdf
from ocrmypdf._concurrent import Executor
from ocrmypdf._progressbar import ProgressBar
from ocrmypdf.exceptions import InputFileError
from ocrmypdf.helpers import available_cpu_count, pikepdf_enable_mmap
if TYPE_CHECKING:
from ocrmypdf.pdfinfo.info import PageInfo
from ocrmypdf.pdfinfo.layout import PdfMinerState
logger = logging.getLogger()
worker_pdf = None # pylint: disable=invalid-name
def _pdf_pageinfo_sync_init(pdf: Pdf, infile: Path, pdfminer_loglevel):
global worker_pdf # pylint: disable=global-statement,invalid-name
pikepdf_enable_mmap()
logging.getLogger('pdfminer').setLevel(pdfminer_loglevel)
# If the pdf is not opened, open a copy for our worker process to use
if pdf is None:
worker_pdf = Pdf.open(infile)
def on_process_close():
worker_pdf.close()
# Close when this process exits
atexit.register(on_process_close)
@contextmanager
def _pdf_pageinfo_sync_pdf(thread_pdf: Pdf | None, infile: Path):
if thread_pdf is not None:
yield thread_pdf
elif worker_pdf is not None:
yield worker_pdf
else:
with Pdf.open(infile) as pdf:
yield pdf
def _pdf_pageinfo_sync(
pageno: int,
thread_pdf: Pdf | None,
infile: Path,
check_pages: Container[int],
detailed_analysis: bool,
miner_state: PdfMinerState | None,
) -> PageInfo:
# Import here to avoid circular import - info.py imports this module,
# but PageInfo is defined in info.py
from ocrmypdf.pdfinfo.info import PageInfo
with _pdf_pageinfo_sync_pdf(thread_pdf, infile) as pdf:
return PageInfo(
pdf, pageno, infile, check_pages, detailed_analysis, miner_state
)
def _pdf_pageinfo_concurrent(
pdf,
executor: Executor,
max_workers: int,
use_threads: bool,
infile,
progbar,
check_pages,
detailed_analysis: bool = False,
miner_state: PdfMinerState | None = None,
) -> Sequence[PageInfo | None]:
pages: list[PageInfo | None] = [None] * len(pdf.pages)
def update_pageinfo(page: PageInfo, pbar: ProgressBar):
if not page:
raise InputFileError("Could read a page in the PDF")
pages[page.pageno] = page
pbar.update()
if max_workers is None:
max_workers = available_cpu_count()
total = len(pdf.pages)
n_workers = min(1 + len(pages) // 4, max_workers)
if n_workers == 1:
# If we decided on only one worker, there is no point in using
# a separate process.
use_threads = True
if use_threads and n_workers > 1:
# If we are using threads, there is no point in using more than one
# worker thread - they will just fight over the GIL.
n_workers = 1
# If we use a thread, we can pass the already-open Pdf for them to use
# If we use processes, we pass a None which tells the init function to open its
# own
initial_pdf = pdf if use_threads else None
contexts = (
(n, initial_pdf, infile, check_pages, detailed_analysis, miner_state)
for n in range(total)
)
assert n_workers == 1 if use_threads else n_workers >= 1, "Not multithreadable"
logger.debug(
f"Gathering info with {n_workers} "
+ ('thread' if use_threads else 'process')
+ " workers"
)
executor(
use_threads=use_threads,
max_workers=n_workers,
progress_kwargs=dict(
total=total, desc="Scanning contents", unit='page', disable=not progbar
),
worker_initializer=partial(
_pdf_pageinfo_sync_init,
initial_pdf,
infile,
logging.getLogger('pdfminer').level,
),
task=_pdf_pageinfo_sync,
task_arguments=contexts,
task_finished=update_pageinfo,
)
return pages
src/ocrmypdf/pdfinfo/info.py
+18 -776
View File
@@ -6,677 +6,34 @@
from __future__ import annotations
import atexit
import logging
import re
import statistics
from collections import defaultdict
from collections.abc import Callable, Container, Iterable, Iterator, Mapping, Sequence
from contextlib import contextmanager, nullcontext
from collections.abc import Callable, Container, Iterable, Iterator
from contextlib import nullcontext
from decimal import Decimal
from enum import Enum, auto
from functools import partial
from math import hypot, inf, isclose
from os import PathLike
from pathlib import Path
from typing import NamedTuple
from warnings import warn
from pdfminer.layout import LTPage, LTTextBox
from pikepdf import (
Dictionary,
Matrix,
Name,
Object,
Page,
Pdf,
PdfImage,
PdfInlineImage,
Stream,
UnsupportedImageTypeError,
parse_content_stream,
)
from pikepdf import Name, Page, Pdf
from ocrmypdf._concurrent import Executor, SerialExecutor
from ocrmypdf._progressbar import ProgressBar
from ocrmypdf.exceptions import EncryptedPdfError, InputFileError
from ocrmypdf.helpers import Resolution, available_cpu_count, pikepdf_enable_mmap
from ocrmypdf.exceptions import EncryptedPdfError
from ocrmypdf.helpers import Resolution
from ocrmypdf.pdfinfo._contentstream import TextboxInfo, TextMarker, VectorMarker
from ocrmypdf.pdfinfo._image import ImageInfo, _process_content_streams
from ocrmypdf.pdfinfo._types import FloatRect
from ocrmypdf.pdfinfo._worker import _pdf_pageinfo_concurrent
from ocrmypdf.pdfinfo.layout import (
LTStateAwareChar,
PdfMinerState,
get_page_analysis,
get_text_boxes,
)
logger = logging.getLogger()
class Colorspace(Enum):
"""Description of common image colorspaces in a PDF."""
# pylint: disable=invalid-name
gray = auto()
rgb = auto()
cmyk = auto()
lab = auto()
icc = auto()
index = auto()
sep = auto()
devn = auto()
pattern = auto()
jpeg2000 = auto()
class Encoding(Enum):
"""Description of common image encodings in a PDF."""
# pylint: disable=invalid-name
ccitt = auto()
jpeg = auto()
jpeg2000 = auto()
jbig2 = auto()
asciihex = auto()
ascii85 = auto()
lzw = auto()
flate = auto()
runlength = auto()
FloatRect = tuple[float, float, float, float]
FRIENDLY_COLORSPACE: dict[str, Colorspace] = {
'/DeviceGray': Colorspace.gray,
'/CalGray': Colorspace.gray,
'/DeviceRGB': Colorspace.rgb,
'/CalRGB': Colorspace.rgb,
'/DeviceCMYK': Colorspace.cmyk,
'/Lab': Colorspace.lab,
'/ICCBased': Colorspace.icc,
'/Indexed': Colorspace.index,
'/Separation': Colorspace.sep,
'/DeviceN': Colorspace.devn,
'/Pattern': Colorspace.pattern,
'/G': Colorspace.gray, # Abbreviations permitted in inline images
'/RGB': Colorspace.rgb,
'/CMYK': Colorspace.cmyk,
'/I': Colorspace.index,
}
FRIENDLY_ENCODING: dict[str, Encoding] = {
'/CCITTFaxDecode': Encoding.ccitt,
'/DCTDecode': Encoding.jpeg,
'/JPXDecode': Encoding.jpeg2000,
'/JBIG2Decode': Encoding.jbig2,
'/CCF': Encoding.ccitt, # Abbreviations permitted in inline images
'/DCT': Encoding.jpeg,
'/AHx': Encoding.asciihex,
'/A85': Encoding.ascii85,
'/LZW': Encoding.lzw,
'/Fl': Encoding.flate,
'/RL': Encoding.runlength,
}
FRIENDLY_COMP: dict[Colorspace, int] = {
Colorspace.gray: 1,
Colorspace.rgb: 3,
Colorspace.cmyk: 4,
Colorspace.lab: 3,
Colorspace.index: 1,
}
UNIT_SQUARE = (1.0, 0.0, 0.0, 1.0, 0.0, 0.0)
def _is_unit_square(shorthand):
values = map(float, shorthand)
pairwise = zip(values, UNIT_SQUARE)
return all(isclose(a, b, rel_tol=1e-3) for a, b in pairwise)
class XobjectSettings(NamedTuple):
"""Info about an XObject found in a PDF."""
name: str
shorthand: tuple[float, float, float, float, float, float]
stack_depth: int
class InlineSettings(NamedTuple):
"""Info about an inline image found in a PDF."""
iimage: PdfInlineImage
shorthand: tuple[float, float, float, float, float, float]
stack_depth: int
class ContentsInfo(NamedTuple):
"""Info about various objects found in a PDF."""
xobject_settings: list[XobjectSettings]
inline_images: list[InlineSettings]
found_vector: bool
found_text: bool
name_index: Mapping[str, list[XobjectSettings]]
class TextboxInfo(NamedTuple):
"""Info about a text box found in a PDF."""
bbox: tuple[float, float, float, float]
is_visible: bool
is_corrupt: bool
class VectorMarker:
"""Sentinel indicating vector drawing operations were found on a page."""
class TextMarker:
"""Sentinel indicating text drawing operations were found on a page."""
def _normalize_stack(graphobjs):
"""Convert runs of qQ's in the stack into single graphobjs."""
for operands, operator in graphobjs:
operator = str(operator)
if re.match(r'Q*q+$', operator): # Zero or more Q, one or more q
for char in operator: # Split into individual
yield ([], char) # Yield individual
else:
yield (operands, operator)
def _interpret_contents(contentstream: Object, initial_shorthand=UNIT_SQUARE):
"""Interpret the PDF content stream.
The stack represents the state of the PDF graphics stack. We are only
interested in the current transformation matrix (CTM) so we only track
this object; a full implementation would need to track many other items.
The CTM is initialized to the mapping from user space to device space.
PDF units are 1/72". In a PDF viewer or printer this matrix is initialized
to the transformation to device space. For example if set to
(1/72, 0, 0, 1/72, 0, 0) then all units would be calculated in inches.
Images are always considered to be (0, 0) -> (1, 1). Before drawing an
image there should be a 'cm' that sets up an image coordinate system
where drawing from (0, 0) -> (1, 1) will draw on the desired area of the
page.
PDF units suit our needs so we initialize ctm to the identity matrix.
According to the PDF specification, the maximum stack depth is 32. Other
viewers tolerate some amount beyond this. We issue a warning if the
stack depth exceeds the spec limit and set a hard limit beyond this to
bound our memory requirements. If the stack underflows behavior is
undefined in the spec, but we just pretend nothing happened and leave the
CTM unchanged.
"""
stack = []
ctm = Matrix(initial_shorthand)
xobject_settings: list[XobjectSettings] = []
inline_images: list[InlineSettings] = []
name_index = defaultdict(lambda: [])
found_vector = False
found_text = False
vector_ops = set('S s f F f* B B* b b*'.split())
text_showing_ops = set("""TJ Tj " '""".split())
image_ops = set('BI ID EI q Q Do cm'.split())
operator_whitelist = ' '.join(vector_ops | text_showing_ops | image_ops)
for n, graphobj in enumerate(
_normalize_stack(parse_content_stream(contentstream, operator_whitelist))
):
operands, operator = graphobj
if operator == 'q':
stack.append(ctm)
if len(stack) > 32: # See docstring
if len(stack) > 128:
raise RuntimeError(
f"PDF graphics stack overflowed hard limit at operator {n}"
)
warn("PDF graphics stack overflowed spec limit")
elif operator == 'Q':
try:
ctm = stack.pop()
except IndexError:
# Keeping the ctm the same seems to be the only sensible thing
# to do. Just pretend nothing happened, keep calm and carry on.
warn("PDF graphics stack underflowed - PDF may be malformed")
elif operator == 'cm':
try:
ctm = Matrix(operands) @ ctm
except ValueError:
raise InputFileError(
"PDF content stream is corrupt - this PDF is malformed. "
"Use a PDF editor that is capable of visually inspecting the PDF."
)
elif operator == 'Do':
image_name = operands[0]
settings = XobjectSettings(
name=image_name, shorthand=ctm.shorthand, stack_depth=len(stack)
)
xobject_settings.append(settings)
name_index[str(image_name)].append(settings)
elif operator == 'INLINE IMAGE': # BI/ID/EI are grouped into this
iimage = operands[0]
inline = InlineSettings(
iimage=iimage, shorthand=ctm.shorthand, stack_depth=len(stack)
)
inline_images.append(inline)
elif operator in vector_ops:
found_vector = True
elif operator in text_showing_ops:
found_text = True
return ContentsInfo(
xobject_settings=xobject_settings,
inline_images=inline_images,
found_vector=found_vector,
found_text=found_text,
name_index=name_index,
)
def _get_dpi(ctm_shorthand, image_size) -> Resolution:
"""Given the transformation matrix and image size, find the image DPI.
PDFs do not include image resolution information within image data.
Instead, the PDF page content stream describes the location where the
image will be rasterized, and the effective resolution is the ratio of the
pixel size to raster target size.
Normally a scanned PDF has the paper size set appropriately but this is
not guaranteed. The most common case is a cropped image will change the
page size (/CropBox) without altering the page content stream. That means
it is not sufficient to assume that the image fills the page, even though
that is the most common case.
A PDF image may be scaled (always), cropped, translated, rotated in place
to an arbitrary angle (rarely) and skewed. Only equal area mappings can
be expressed, that is, it is not necessary to consider distortions where
the effective DPI varies with position.
To determine the image scale, transform an offset axis vector v0 (0, 0),
width-axis vector v0 (1, 0), height-axis vector vh (0, 1) with the matrix,
which gives the dimensions of the image in PDF units. From there we can
compare to actual image dimensions. PDF uses
row vector * matrix_transposed unlike the traditional
matrix * column vector.
The offset, width and height vectors can be combined in a matrix and
multiplied by the transform matrix. Then we want to calculated
magnitude(width_vector - offset_vector)
and
magnitude(height_vector - offset_vector)
When the above is worked out algebraically, the effect of translation
cancels out, and the vector magnitudes become functions of the nonzero
transformation matrix indices. The results of the derivation are used
in this code.
pdfimages -list does calculate the DPI in some way that is not completely
naive, but it does not get the DPI of rotated images right, so cannot be
used anymore to validate this. Photoshop works, or using Acrobat to
rotate the image back to normal.
It does not matter if the image is partially cropped, or even out of the
/MediaBox.
"""
a, b, c, d, _, _ = ctm_shorthand # pylint: disable=invalid-name
# Calculate the width and height of the image in PDF units
image_drawn = hypot(a, b), hypot(c, d)
def calc(drawn, pixels, inches_per_pt=72.0):
# The scale of the image is pixels per unit of default user space (1/72")
scale = pixels / drawn if drawn != 0 else inf
dpi = scale * inches_per_pt
return dpi
dpi_w, dpi_h = (calc(image_drawn[n], image_size[n]) for n in range(2))
return Resolution(dpi_w, dpi_h)
class ImageInfo:
"""Information about an image found in a PDF.
This gathers information from pikepdf and pdfminer.six, and is pickle-able
so that it can be passed to a worker process, unlike objects from those
libraries.
"""
DPI_PREC = Decimal('1.000')
_comp: int | None
_name: str
def __init__(
self,
*,
name='',
pdfimage: Object | None = None,
inline: PdfInlineImage | None = None,
shorthand=None,
):
"""Initialize an ImageInfo."""
self._name = str(name)
self._shorthand = shorthand
pim: PdfInlineImage | PdfImage
if inline is not None:
self._origin = 'inline'
pim = inline
elif pdfimage is not None and isinstance(pdfimage, Stream):
self._origin = 'xobject'
pim = PdfImage(pdfimage)
else:
raise ValueError("Either pdfimage or inline must be set")
self._width = pim.width
self._height = pim.height
if (smask := pim.obj.get(Name.SMask, None)) is not None:
# SMask is pretty much an alpha channel, but in PDF it's possible
# for channel to have different dimensions than the image
# itself. Some PDF writers use this to create a grayscale stencil
# mask. For our purposes, the effective size is the size of the
# larger component (image or smask).
if isinstance(smask, Stream | Dictionary):
self._width = max(smask.get(Name.Width, 0), self._width)
self._height = max(smask.get(Name.Height, 0), self._height)
if (mask := pim.obj.get(Name.Mask, None)) is not None:
# If the image has a /Mask entry, it has an explicit mask.
# /Mask can be a Stream or an Array. If it's a Stream,
# use its /Width and /Height if they are larger than the main
# image's.
if isinstance(mask, Stream | Dictionary):
self._width = max(mask.get(Name.Width, 0), self._width)
self._height = max(mask.get(Name.Height, 0), self._height)
# If /ImageMask is true, then this image is a stencil mask
# (Images that draw with this stencil mask will have a reference to
# it in their /Mask, but we don't actually need that information)
if pim.image_mask:
self._type = 'stencil'
else:
self._type = 'image'
self._bpc = int(pim.bits_per_component)
try:
self._enc = FRIENDLY_ENCODING.get(pim.filters[0])
except IndexError:
self._enc = None
try:
self._color = FRIENDLY_COLORSPACE.get(pim.colorspace or '')
except NotImplementedError:
self._color = None
if self._enc == Encoding.jpeg2000:
self._color = Colorspace.jpeg2000
self._comp = None
if self._color == Colorspace.icc and isinstance(pim, PdfImage):
self._comp = self._init_icc(pim)
else:
if isinstance(self._color, Colorspace):
self._comp = FRIENDLY_COMP.get(self._color)
# Bit of a hack... infer grayscale if component count is uncertain
# but encoding only supports monochrome.
if self._comp is None and self._enc in (Encoding.ccitt, Encoding.jbig2):
self._comp = FRIENDLY_COMP[Colorspace.gray]
def _init_icc(self, pim: PdfImage):
try:
icc = pim.icc
except UnsupportedImageTypeError as e:
logger.warning(
f"An image with a corrupt or unreadable ICC profile was found. "
f"Output PDF may not match the input PDF visually: {e}. {self}"
)
return None
# Check the ICC profile to determine actual colorspace
if icc is None or not hasattr(icc, 'profile'):
logger.warning(
f"An image with an ICC profile but no ICC profile data was found. "
f"The output PDF may not match the input PDF visually. {self}"
)
return None
try:
if icc.profile.xcolor_space == 'GRAY':
return 1
elif icc.profile.xcolor_space == 'CMYK':
return 4
else:
return 3
except AttributeError:
return None
@property
def name(self):
"""Name of the image as it appears in the PDF."""
return self._name
@property
def type_(self):
"""Type of image, either 'image' or 'stencil'."""
return self._type
@property
def width(self) -> int:
"""Width of the image in pixels."""
return self._width
@property
def height(self) -> int:
"""Height of the image in pixels."""
return self._height
@property
def bpc(self):
"""Bits per component."""
return self._bpc
@property
def color(self):
"""Colorspace of the image."""
return self._color if self._color is not None else '?'
@property
def comp(self):
"""Number of components/channels in the image."""
return self._comp if self._comp is not None else '?'
@property
def enc(self):
"""Encoding of the image."""
return self._enc if self._enc is not None else 'image'
@property
def renderable(self) -> bool:
"""Whether the image is renderable.
Some PDFs in the wild have invalid images that are not renderable,
due to unusual dimensions.
Stencil masks are not also not renderable, since they are not
drawn, but rather they control how rendering happens.
"""
return (
self.dpi.is_finite
and self.width >= 0
and self.height >= 0
and self.type_ != 'stencil'
)
@property
def dpi(self) -> Resolution:
"""Dots per inch of the image.
Calculated based on where and how the image is drawn in the PDF.
"""
return _get_dpi(self._shorthand, (self._width, self._height))
@property
def printed_area(self) -> float:
"""Physical area of the image in square inches."""
if not self.renderable:
return 0.0
return float((self.width / self.dpi.x) * (self.height / self.dpi.y))
def __repr__(self):
"""Return a string representation of the image."""
return (
f"<ImageInfo '{self.name}' {self.type_} {self.width}×{self.height} "
f"{self.color} {self.comp} {self.bpc} {self.enc} {self.dpi}>"
)
def _find_inline_images(contentsinfo: ContentsInfo) -> Iterator[ImageInfo]:
"""Find inline images in the contentstream."""
for n, inline in enumerate(contentsinfo.inline_images):
yield ImageInfo(
name=f'inline-{n:02d}', shorthand=inline.shorthand, inline=inline.iimage
)
def _image_xobjects(container) -> Iterator[tuple[Object, str]]:
"""Search for all XObject-based images in the container.
Usually the container is a page, but it could also be a Form XObject
that contains images. Filter out the Form XObjects which are dealt with
elsewhere.
Generate a sequence of tuples (image, xobj container), where container,
where xobj is the name of the object and image is the object itself,
since the object does not know its own name.
"""
if Name.Resources not in container:
return
resources = container[Name.Resources]
if Name.XObject not in resources:
return
for key, candidate in resources[Name.XObject].items():
if candidate is None or Name.Subtype not in candidate:
continue
if candidate[Name.Subtype] == Name.Image:
pdfimage = candidate
yield (pdfimage, key)
def _find_regular_images(
container: Object, contentsinfo: ContentsInfo
) -> Iterator[ImageInfo]:
"""Find images stored in the container's /Resources /XObject.
Usually the container is a page, but it could also be a Form XObject
that contains images.
Generates images with their DPI at time of drawing.
"""
for pdfimage, xobj in _image_xobjects(container):
if xobj not in contentsinfo.name_index:
continue
for draw in contentsinfo.name_index[xobj]:
if draw.stack_depth == 0 and _is_unit_square(draw.shorthand):
# At least one PDF in the wild (and test suite) draws an image
# when the graphics stack depth is 0, meaning that the image
# gets drawn into a square of 1x1 PDF units (or 1/72",
# or 0.35 mm). The equivalent DPI will be >100,000. Exclude
# these from our DPI calculation for the page.
continue
yield ImageInfo(name=draw.name, pdfimage=pdfimage, shorthand=draw.shorthand)
def _find_form_xobject_images(pdf: Pdf, container: Object, contentsinfo: ContentsInfo):
"""Find any images that are in Form XObjects in the container.
The container may be a page, or a parent Form XObject.
"""
if Name.Resources not in container:
return
resources = container[Name.Resources]
if Name.XObject not in resources:
return
xobjs = resources[Name.XObject].as_dict()
for xobj in xobjs:
candidate = xobjs[xobj]
if candidate is None or candidate.get(Name.Subtype) != Name.Form:
continue
form_xobject = candidate
for settings in contentsinfo.xobject_settings:
if settings.name != xobj:
continue
# Find images once for each time this Form XObject is drawn.
# This could be optimized to cache the multiple drawing events
# but in practice both Form XObjects and multiple drawing of the
# same object are both very rare.
ctm_shorthand = settings.shorthand
yield from _process_content_streams(
pdf=pdf, container=form_xobject, shorthand=ctm_shorthand
)
def _process_content_streams(
*, pdf: Pdf, container: Object, shorthand=None
) -> Iterator[VectorMarker | TextMarker | ImageInfo]:
"""Find all individual instances of images drawn in the container.
Usually the container is a page, but it may also be a Form XObject.
On a typical page images are stored inline or as regular images
in an XObject.
Form XObjects may include inline images, XObject images,
and recursively, other Form XObjects; and also vector graphic objects.
Every instance of an image being drawn somewhere is flattened and
treated as a unique image, since if the same image is drawn multiple times
on one page it may be drawn at differing resolutions, and our objective
is to find the resolution at which the page can be rastered without
downsampling.
"""
if container.get(Name.Type) == Name.Page and Name.Contents in container:
initial_shorthand = shorthand or UNIT_SQUARE
elif (
container.get(Name.Type) == Name.XObject
and container[Name.Subtype] == Name.Form
):
# Set the CTM to the state it was when the "Do" operator was
# encountered that is drawing this instance of the Form XObject
ctm = Matrix(shorthand) if shorthand else Matrix()
# A Form XObject may provide its own matrix to map form space into
# user space. Get this if one exists
form_shorthand = container.get(Name.Matrix, Matrix())
form_matrix = Matrix(form_shorthand)
# Concatenate form matrix with CTM to ensure CTM is correct for
# drawing this instance of the XObject
ctm = form_matrix @ ctm
initial_shorthand = ctm.shorthand
else:
return
contentsinfo = _interpret_contents(container, initial_shorthand)
if contentsinfo.found_vector:
yield VectorMarker()
if contentsinfo.found_text:
yield TextMarker()
yield from _find_inline_images(contentsinfo)
yield from _find_regular_images(container, contentsinfo)
yield from _find_form_xobject_images(pdf, container, contentsinfo)
def _page_has_text(text_blocks: Iterable[FloatRect], page_width, page_height) -> bool:
"""Smarter text detection that ignores text in margins."""
pw, ph = float(page_width), float(page_height) # pylint: disable=invalid-name
@@ -723,120 +80,6 @@ def simplify_textboxes(
yield TextboxInfo(box.bbox, visible, corrupt)
worker_pdf = None # pylint: disable=invalid-name
def _pdf_pageinfo_sync_init(pdf: Pdf, infile: Path, pdfminer_loglevel):
global worker_pdf # pylint: disable=global-statement,invalid-name
pikepdf_enable_mmap()
logging.getLogger('pdfminer').setLevel(pdfminer_loglevel)
# If the pdf is not opened, open a copy for our worker process to use
if pdf is None:
worker_pdf = Pdf.open(infile)
def on_process_close():
worker_pdf.close()
# Close when this process exits
atexit.register(on_process_close)
@contextmanager
def _pdf_pageinfo_sync_pdf(thread_pdf: Pdf | None, infile: Path):
if thread_pdf is not None:
yield thread_pdf
elif worker_pdf is not None:
yield worker_pdf
else:
with Pdf.open(infile) as pdf:
yield pdf
def _pdf_pageinfo_sync(
pageno: int,
thread_pdf: Pdf | None,
infile: Path,
check_pages: Container[int],
detailed_analysis: bool,
miner_state: PdfMinerState | None,
) -> PageInfo:
with _pdf_pageinfo_sync_pdf(thread_pdf, infile) as pdf:
return PageInfo(
pdf, pageno, infile, check_pages, detailed_analysis, miner_state
)
def _pdf_pageinfo_concurrent(
pdf,
executor: Executor,
max_workers: int,
use_threads: bool,
infile,
progbar,
check_pages,
detailed_analysis: bool = False,
miner_state: PdfMinerState | None = None,
) -> Sequence[PageInfo | None]:
pages: list[PageInfo | None] = [None] * len(pdf.pages)
def update_pageinfo(page: PageInfo, pbar: ProgressBar):
if not page:
raise InputFileError("Could read a page in the PDF")
pages[page.pageno] = page
pbar.update()
if max_workers is None:
max_workers = available_cpu_count()
total = len(pdf.pages)
n_workers = min(1 + len(pages) // 4, max_workers)
if n_workers == 1:
# If we decided on only one worker, there is no point in using
# a separate process.
use_threads = True
if use_threads and n_workers > 1:
# If we are using threads, there is no point in using more than one
# worker thread - they will just fight over the GIL.
n_workers = 1
# If we use a thread, we can pass the already-open Pdf for them to use
# If we use processes, we pass a None which tells the init function to open its
# own
initial_pdf = pdf if use_threads else None
contexts = (
(n, initial_pdf, infile, check_pages, detailed_analysis, miner_state)
for n in range(total)
)
assert n_workers == 1 if use_threads else n_workers >= 1, "Not multithreadable"
logger.debug(
f"Gathering info with {n_workers} "
+ ('thread' if use_threads else 'process')
+ " workers"
)
executor(
use_threads=use_threads,
max_workers=n_workers,
progress_kwargs=dict(
total=total, desc="Scanning contents", unit='page', disable=not progbar
),
worker_initializer=partial(
_pdf_pageinfo_sync_init,
initial_pdf,
infile,
logging.getLogger('pdfminer').level,
),
task=_pdf_pageinfo_sync,
task_arguments=contexts,
task_finished=update_pageinfo,
)
return pages
class PageResolutionProfile(NamedTuple):
"""Information about the resolutions of a page."""
@@ -1061,12 +304,10 @@ class PageInfo:
obj: TextboxInfo, want_visible: bool | None, want_corrupt: bool | None
) -> bool:
result = True
if want_visible is not None:
if obj.is_visible != want_visible:
result = False
if want_corrupt is not None:
if obj.is_corrupt != want_corrupt:
result = False
if want_visible is not None and obj.is_visible != want_visible:
result = False
if want_corrupt is not None and obj.is_corrupt != want_corrupt:
result = False
return result
if not self._textboxes:
@@ -1199,9 +440,10 @@ class PdfInfo:
)
self._needs_rendering = pdf.Root.get(Name.NeedsRendering, False)
if Name.AcroForm in pdf.Root:
if len(pdf.Root.AcroForm.get(Name.Fields, [])) > 0:
self._has_acroform = True
elif Name.XFA in pdf.Root.AcroForm:
if (
len(pdf.Root.AcroForm.get(Name.Fields, [])) > 0
or Name.XFA in pdf.Root.AcroForm
):
self._has_acroform = True
self._has_signature = bool(pdf.Root.AcroForm.get(Name.SigFlags, 0) & 1)
self._is_tagged = bool(
@@ -1209,7 +451,7 @@ class PdfInfo:
)
@property
def pages(self) -> Sequence[PageInfo | None]:
def pages(self) -> list[PageInfo | None]:
"""Return list of PageInfo objects, one per page in the PDF."""
return self._pages
src/ocrmypdf/pdfinfo/layout.py
+2 -2
View File
@@ -58,7 +58,7 @@ def pdfsimplefont__init__(
return
setattr(PDFSimpleFont, '__init__', pdfsimplefont__init__)
PDFSimpleFont.__init__ = pdfsimplefont__init__
# Patch pdfminer.six buffer size
# The parser doesn't properly handle keyword tokens are split across the end of the
@@ -363,7 +363,7 @@ class PdfMinerState:
except StopIteration:
raise InputFileError(
f"pdfminer did not find page {pageno} in the input file."
)
) from None
page = self.page_cache[pageno]
if not page:
raise InputFileError(
src/ocrmypdf/pluginspec.py
+121 -22
View File
@@ -6,15 +6,18 @@
from __future__ import annotations
from abc import ABC, abstractmethod
from argparse import ArgumentParser, Namespace
from argparse import ArgumentParser
from collections.abc import Sequence, Set
from enum import StrEnum
from logging import Handler
from pathlib import Path
from typing import TYPE_CHECKING, NamedTuple
import pluggy
from pydantic import BaseModel
from ocrmypdf import Executor, PdfContext
from ocrmypdf._options import OcrOptions
from ocrmypdf._progressbar import ProgressBar
from ocrmypdf.helpers import Resolution
@@ -23,10 +26,23 @@ if TYPE_CHECKING:
# pylint: disable=ungrouped-imports
from ocrmypdf._jobcontext import PageContext
from ocrmypdf.hocrtransform import OcrElement
from ocrmypdf.pdfinfo import PdfInfo
# pylint: enable=ungrouped-imports
class GhostscriptRasterDevice(StrEnum):
"""Possible raster devices for Ghostscript."""
JPEGGRAY = 'jpeggray'
JPEGCOLOR = 'jpeg'
PNGMONO = 'pngmono'
PNGGRAY = 'pnggray'
PNG256 = 'png256'
PNG16M = 'png16m'
hookspec = pluggy.HookspecMarker('ocrmypdf')
# pylint: disable=unused-argument
@@ -34,7 +50,7 @@ hookspec = pluggy.HookspecMarker('ocrmypdf')
@hookspec(firstresult=True)
def get_logging_console() -> Handler:
def get_logging_console() -> Handler: # type: ignore[return-value]
"""Returns a custom logging handler.
Generally this is necessary when both logging output and a progress bar are both
@@ -87,7 +103,29 @@ def add_options(parser: ArgumentParser) -> None:
@hookspec
def check_options(options: Namespace) -> None:
def register_options() -> dict[str, type[BaseModel]]:
"""Return plugin's option models keyed by namespace.
This hook allows plugins to register their option models with the
plugin option registry. The returned dictionary should map namespace
strings to Pydantic model classes.
Returns:
Dictionary mapping namespace strings to BaseModel classes
Example:
@hookimpl
def register_options():
return {'tesseract': TesseractOptions}
Note:
This hook will be called from the main process during plugin
infrastructure setup, before child worker processes are forked.
"""
@hookspec
def check_options(options: OcrOptions) -> None:
"""Called to ask the plugin to check all of the options.
The plugin may check if options that it added are valid.
@@ -110,7 +148,7 @@ def check_options(options: Namespace) -> None:
@hookspec(firstresult=True)
def get_executor(progressbar_class: type[ProgressBar]) -> Executor:
def get_executor(progressbar_class: type[ProgressBar]) -> Executor: # type: ignore[return-value]
"""Called to obtain an object that manages parallel execution.
This may be used to replace OCRmyPDF's default parallel execution system
@@ -138,7 +176,7 @@ def get_executor(progressbar_class: type[ProgressBar]) -> Executor:
@hookspec(firstresult=True)
def get_progressbar_class() -> type[ProgressBar]:
def get_progressbar_class() -> type[ProgressBar]: # type: ignore[return-value]
"""Called to obtain a class that can be used to monitor progress.
OCRmyPDF will call this function when it wants to display a progress bar.
@@ -158,7 +196,7 @@ def get_progressbar_class() -> type[ProgressBar]:
@hookspec
def validate(pdfinfo: PdfInfo, options: Namespace) -> None:
def validate(pdfinfo: PdfInfo, options: OcrOptions) -> None:
"""Called to give a plugin an opportunity to review *options* and *pdfinfo*.
*options* contains the "work order" to process a particular file. *pdfinfo*
@@ -182,14 +220,16 @@ def validate(pdfinfo: PdfInfo, options: Namespace) -> None:
def rasterize_pdf_page(
input_file: Path,
output_file: Path,
raster_device: str,
raster_device: GhostscriptRasterDevice,
raster_dpi: Resolution,
pageno: int,
page_dpi: Resolution | None,
rotation: int | None,
filter_vector: bool,
stop_on_soft_error: bool,
) -> Path:
options: OcrOptions | None,
use_cropbox: bool,
) -> Path: # type: ignore[return-value]
"""Rasterize one page of a PDF at resolution raster_dpi in canvas units.
The image is sized to match the integer pixels dimensions implied by
@@ -212,6 +252,12 @@ def rasterize_pdf_page(
cannot proceed, it should always raise an exception, regardless of
this setting. One "soft error" would be a missing font that is
required to properly rasterize the PDF.
options: OCRmyPDF options. Plugins may use this to check settings like
``options.rasterizer`` to determine whether they should handle the
request or defer to another plugin. Introduced in version 17.0.
use_cropbox: If True, rasterize the page's CropBox instead of the
MediaBox. Default is False (use MediaBox) for consistency with
Ghostscript's default behavior.
Returns:
Path: output_file if successful
@@ -226,7 +272,7 @@ def rasterize_pdf_page(
@hookspec(firstresult=True)
def filter_ocr_image(page: PageContext, image: Image.Image) -> Image.Image:
def filter_ocr_image(page: PageContext, image: Image.Image) -> Image.Image: # type: ignore[return-value]
"""Called to filter the image before it is sent to OCR.
This is the image that OCR sees, not what the user sees when they view the
@@ -261,7 +307,7 @@ def filter_ocr_image(page: PageContext, image: Image.Image) -> Image.Image:
@hookspec(firstresult=True)
def filter_page_image(page: PageContext, image_filename: Path) -> Path:
def filter_page_image(page: PageContext, image_filename: Path) -> Path: # type: ignore[return-value]
"""Called to filter the whole page before it is inserted into the PDF.
A whole page image is only produced when preprocessing command line arguments
@@ -298,7 +344,7 @@ def filter_page_image(page: PageContext, image_filename: Path) -> Path:
@hookspec(firstresult=True)
def filter_pdf_page(page: PageContext, image_filename: Path, output_pdf: Path) -> Path:
def filter_pdf_page(page: PageContext, image_filename: Path, output_pdf: Path) -> Path: # type: ignore[return-value]
"""Called to convert a filtered whole page image into a PDF.
A whole page image is only produced when preprocessing command line arguments
@@ -368,7 +414,7 @@ class OcrEngine(ABC):
@staticmethod
@abstractmethod
def creator_tag(options: Namespace) -> str:
def creator_tag(options: OcrOptions) -> str:
"""Returns the creator tag to identify this software's role in creating the PDF.
This tag will be inserted in the XMP metadata and DocumentInfo dictionary
@@ -380,7 +426,7 @@ class OcrEngine(ABC):
"""
@abstractmethod
def __str__(self):
def __str__(self) -> str:
"""Returns name of OCR engine and version.
This is used when OCRmyPDF wants to mention the name of the OCR engine
@@ -389,7 +435,7 @@ class OcrEngine(ABC):
@staticmethod
@abstractmethod
def languages(options: Namespace) -> Set[str]:
def languages(options: OcrOptions) -> Set[str]:
"""Returns the set of all languages that are supported by the engine.
Languages are typically given in 3-letter ISO 3166-1 codes, but actually
@@ -398,18 +444,18 @@ class OcrEngine(ABC):
@staticmethod
@abstractmethod
def get_orientation(input_file: Path, options: Namespace) -> OrientationConfidence:
def get_orientation(input_file: Path, options: OcrOptions) -> OrientationConfidence:
"""Returns the orientation of the image."""
@staticmethod
def get_deskew(input_file: Path, options: Namespace) -> float:
def get_deskew(input_file: Path, options: OcrOptions) -> float:
"""Returns the deskew angle of the image, in degrees."""
return 0.0
@staticmethod
@abstractmethod
def generate_hocr(
input_file: Path, output_hocr: Path, output_text: Path, options: Namespace
input_file: Path, output_hocr: Path, output_text: Path, options: OcrOptions
) -> None:
"""Called to produce a hOCR file from a page image and sidecar text file.
@@ -432,7 +478,7 @@ class OcrEngine(ABC):
@staticmethod
@abstractmethod
def generate_pdf(
input_file: Path, output_pdf: Path, output_text: Path, options: Namespace
input_file: Path, output_pdf: Path, output_text: Path, options: OcrOptions
) -> None:
"""Called to produce a text only PDF from a page image.
@@ -452,14 +498,67 @@ class OcrEngine(ABC):
options: The command line options.
"""
@staticmethod
def supports_generate_ocr() -> bool:
"""Return True if this engine supports the generate_ocr() API.
The pipeline uses this to determine whether to call generate_ocr()
or fall back to generate_hocr().
Returns:
False by default. Engines implementing generate_ocr() should
override this to return True.
"""
return False
@staticmethod
def generate_ocr(
input_file: Path,
options: OcrOptions,
page_number: int = 0,
) -> tuple[OcrElement, str]:
"""Generate OCR results as an OcrElement tree.
This is the modern API for OCR engines. Engines implementing this method
can return structured OCR results directly without intermediate file formats.
This function executes in a worker thread or worker process. OCRmyPDF
automatically parallelizes OCR over pages. The OCR engine should not
introduce more parallelism.
Args:
input_file: A page image on which to perform OCR.
options: The command line options.
page_number: Zero-indexed page number (for multi-page context).
Returns:
A tuple of (OcrElement tree for the page, plain text content).
The OcrElement should have ocr_class=OcrClass.PAGE as its root.
Note:
This method is optional. Engines that don't implement it should
leave the default implementation, and the pipeline will fall back to
generate_hocr() or generate_pdf().
"""
raise NotImplementedError("This OcrEngine does not implement generate_ocr()")
@hookspec(firstresult=True)
def get_ocr_engine() -> OcrEngine:
def get_ocr_engine(options: OcrOptions | None) -> OcrEngine: # type: ignore[return-value]
"""Returns an OcrEngine to use for processing this file.
The OcrEngine may be instantiated multiple times, by both the main process
and child process.
When multiple OCR engine plugins are installed, plugins should check
``options.ocr_engine`` and return ``None`` if they are not the selected
engine. The hook caller will then try the next plugin.
Args:
options: The current OcrOptions, used to determine which engine
to select. May be None for backward compatibility with external
plugins.
Note:
This is a :ref:`firstresult hook<firstresult>`.
"""
@@ -475,7 +574,7 @@ def generate_pdfa(
pdfa_part: str,
progressbar_class: type[ProgressBar] | None,
stop_on_soft_error: bool,
) -> Path:
) -> Path: # type: ignore[return-value]
"""Generate a PDF/A.
This API strongly assumes a PDF/A generator with Ghostscript's semantics.
@@ -522,7 +621,7 @@ def optimize_pdf(
context: PdfContext,
executor: Executor,
linearize: bool,
) -> tuple[Path, Sequence[str]]:
) -> tuple[Path, Sequence[str]]: # type: ignore[return-value]
"""Optimize a PDF after image, OCR and metadata processing.
If the input_pdf is a PDF/A, the plugin should modify input_pdf in a way
@@ -559,7 +658,7 @@ def optimize_pdf(
@hookspec(firstresult=True)
def is_optimization_enabled(context: PdfContext) -> bool:
def is_optimization_enabled(context: PdfContext) -> bool: # type: ignore[return-value]
"""For a given PdfContext, OCRmyPDF asks the plugin if optimization is enabled.
An optimization plugin might be installed and active but could be disabled by
src/ocrmypdf/quality.py
+1 -4
View File
@@ -40,8 +40,5 @@ class OcrQualityDictionary:
w != w.lower() and w.lower() in self.dictionary
):
matches += 1
if matches > 0:
hit_ratio = matches / len(text_words)
else:
hit_ratio = 0.0
hit_ratio = matches / len(text_words) if matches > 0 else 0.0
return hit_ratio
src/ocrmypdf/subprocess/__init__.py
+6 -6
View File
@@ -23,13 +23,13 @@ from ocrmypdf.exceptions import MissingDependencyError
log = logging.getLogger(__name__)
Args = Sequence[Path | str]
OsEnviron = os._Environ # pylint: disable=protected-access
Environ = Mapping[str, str] | os._Environ # pylint: disable=protected-access
def run(
args: Args,
*,
env: OsEnviron | None = None,
env: Environ | None = None,
logs_errors_to_stdout: bool = False,
check: bool = False,
**kwargs,
@@ -81,7 +81,7 @@ def run_polling_stderr(
*,
callback: Callable[[str], None],
check: bool = False,
env: OsEnviron | None = None,
env: Environ | None = None,
**kwargs,
) -> CompletedProcess:
"""Run a process like ``ocrmypdf.subprocess.run``, and poll stderr.
@@ -116,8 +116,8 @@ def run_polling_stderr(
def _fix_process_args(
args: Args, env: OsEnviron | None, kwargs
) -> tuple[Args, OsEnviron, logging.Logger, bool]:
args: Args, env: Environ | None, kwargs
) -> tuple[Args, Environ, logging.Logger, bool]:
if not env:
env = os.environ
@@ -142,7 +142,7 @@ def get_version(
*,
version_arg: str = '--version',
regex=r'(\d+(\.\d+)*)',
env: OsEnviron | None = None,
env: Environ | None = None,
) -> str:
"""Get the version of the specified program.
tests/cache/2400dpi/__-l__eng__000001_ocr.png__000001_ocr_hocr__hocr__txt/hocr.bin
Vendored
+133 -132
View File
@@ -5,170 +5,171 @@
<head>
<title></title>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>
<meta name='ocr-system' content='tesseract 5.3.2' />
<meta name='ocr-capabilities' content='ocr_page ocr_carea ocr_par ocr_line ocrx_word ocrp_wconf'/>
<meta name='ocr-system' content='tesseract 5.5.1' />
<meta name='ocr-capabilities' content='ocr_page ocr_carea ocr_par ocr_line ocrx_word ocrp_dir ocrp_lang ocrp_wconf'/>
</head>
<body>
<div class='ocr_page' id='page_1' title='image "/tmp/ocrmypdf.io.jcdzf4up/000001_ocr.png"; bbox 0 0 4000 2864; ppageno 0; scan_res 2400 2400'>
<div class='ocr_carea' id='block_1_1' title="bbox 251 146 2173 237">
<p class='ocr_par' id='par_1_1' lang='eng' title="bbox 251 146 2173 237">
<span class='ocr_line' id='line_1_1' title="bbox 251 146 2173 237; baseline -0.006 5; x_size 99.418808; x_descenders 9.4188042; x_ascenders 32">
<span class='ocrx_word' id='word_1_1' title='bbox 251 154 274 176; x_wconf 89'>i</span>
<span class='ocrx_word' id='word_1_2' title='bbox 1080 146 1152 237; x_wconf 89'>a</span>
<span class='ocrx_word' id='word_1_3' title='bbox 1254 153 1406 235; x_wconf 92'>la</span>
<span class='ocrx_word' id='word_1_4' title='bbox 1500 153 2173 235; x_wconf 95'>Waterman</span>
<div class='ocr_page' id='page_1' title='image "/tmp/ocrmypdf.io.662_0rn0/000001_ocr.png"; bbox 0 0 4001 2864; ppageno 0; scan_res 2400 2400'>
<div class='ocr_carea' id='block_1_1' title="bbox 142 152 2174 236">
<p class='ocr_par' id='par_1_1' lang='eng' title="bbox 142 152 2174 236">
<span class='ocr_line' id='line_1_1' title="bbox 142 152 2174 236; baseline 0.003 -11; x_size 109; x_descenders 30; x_ascenders 24">
<span class='ocrx_word' id='word_1_1' title='bbox 142 157 969 214; x_wconf 89'>Linzensoep</span>
<span class='ocrx_word' id='word_1_2' title='bbox 1080 177 1152 236; x_wconf 95'>a</span>
<span class='ocrx_word' id='word_1_3' title='bbox 1254 152 1406 234; x_wconf 94'>la</span>
<span class='ocrx_word' id='word_1_4' title='bbox 1500 152 2174 234; x_wconf 96'>Waterman</span>
</span>
</p>
</div>
<div class='ocr_separator' id='block_1_2' title="bbox 135 202 2180 295"></div>
<div class='ocr_carea' id='block_1_3' title="bbox 145 414 2929 1221">
<p class='ocr_par' id='par_1_2' lang='eng' title="bbox 145 414 1154 516">
<span class='ocr_line' id='line_1_2' title="bbox 145 414 1154 516; baseline -0.005 -4; x_size 122.38636; x_descenders 24.386362; x_ascenders 40">
<span class='ocrx_word' id='word_1_5' title='bbox 145 414 211 512; x_wconf 60'>4</span>
<span class='ocrx_word' id='word_1_6' title='bbox 318 453 551 516; x_wconf 93'>ons</span>
<span class='ocrx_word' id='word_1_7' title='bbox 660 430 1154 512; x_wconf 91'>linzen</span>
<div class='ocr_separator' id='block_1_2' title="bbox 136 209 2181 286"></div>
<div class='ocr_carea' id='block_1_3' title="bbox 145 413 2929 1220">
<p class='ocr_par' id='par_1_2' lang='eng' title="bbox 145 413 1154 515">
<span class='ocr_line' id='line_1_2' title="bbox 145 413 1154 515; baseline -0.005 -4; x_size 121.97968; x_descenders 23.979671; x_ascenders 40">
<span class='ocrx_word' id='word_1_5' title='bbox 145 413 211 511; x_wconf 49'>4</span>
<span class='ocrx_word' id='word_1_6' title='bbox 318 452 551 515; x_wconf 91'>ons</span>
<span class='ocrx_word' id='word_1_7' title='bbox 660 429 1154 511; x_wconf 84'>linzen</span>
</span>
</p>
<p class='ocr_par' id='par_1_3' lang='eng' title="bbox 146 568 1239 667">
<span class='ocr_line' id='line_1_3' title="bbox 146 568 1239 667; baseline -0.004 -16; x_size 99; x_descenders 17; x_ascenders 24">
<span class='ocrx_word' id='word_1_8' title='bbox 146 569 209 667; x_wconf 94'>3</span>
<span class='ocrx_word' id='word_1_9' title='bbox 323 568 729 652; x_wconf 83'>liter</span>
<span class='ocrx_word' id='word_1_10' title='bbox 821 569 1239 650; x_wconf 96'>water</span>
<p class='ocr_par' id='par_1_3' lang='eng' title="bbox 146 567 1239 666">
<span class='ocr_line' id='line_1_3' title="bbox 146 567 1239 666; baseline -0.004 -16; x_size 99; x_descenders 17; x_ascenders 24">
<span class='ocrx_word' id='word_1_8' title='bbox 146 568 209 666; x_wconf 94'>3</span>
<span class='ocrx_word' id='word_1_9' title='bbox 323 567 729 651; x_wconf 79'>liter</span>
<span class='ocrx_word' id='word_1_10' title='bbox 821 568 1239 649; x_wconf 95'>water</span>
</span>
</p>
<p class='ocr_par' id='par_1_4' lang='eng' title="bbox 148 705 645 804">
<span class='ocr_line' id='line_1_4' title="bbox 148 705 645 804; baseline -0.016 -14; x_size 103; x_descenders 19; x_ascenders 27">
<span class='ocrx_word' id='word_1_11' title='bbox 148 706 211 804; x_wconf 88'>3</span>
<span class='ocrx_word' id='word_1_12' title='bbox 311 705 645 789; x_wconf 52'>uien</span>
<p class='ocr_par' id='par_1_4' lang='eng' title="bbox 148 704 645 803">
<span class='ocr_line' id='line_1_4' title="bbox 148 704 645 803; baseline -0.016 -14; x_size 103; x_descenders 19; x_ascenders 27">
<span class='ocrx_word' id='word_1_11' title='bbox 148 705 211 803; x_wconf 87'>3</span>
<span class='ocrx_word' id='word_1_12' title='bbox 311 704 645 788; x_wconf 78'>uien</span>
</span>
</p>
<p class='ocr_par' id='par_1_5' lang='eng' title="bbox 147 832 1154 950">
<span class='ocr_line' id='line_1_5' title="bbox 147 832 1154 950; baseline -0.004 -27; x_size 118; x_descenders 28; x_ascenders 30">
<span class='ocrx_word' id='word_1_13' title='bbox 147 832 623 950; x_wconf 91'>bloem,</span>
<span class='ocrx_word' id='word_1_14' title='bbox 737 843 1154 924; x_wconf 91'>boter</span>
<p class='ocr_par' id='par_1_5' lang='eng' title="bbox 147 831 1154 949">
<span class='ocr_line' id='line_1_5' title="bbox 147 831 1154 949; baseline -0.004 -27; x_size 118; x_descenders 28; x_ascenders 30">
<span class='ocrx_word' id='word_1_13' title='bbox 147 831 623 949; x_wconf 91'>bloem,</span>
<span class='ocrx_word' id='word_1_14' title='bbox 737 842 1154 923; x_wconf 92'>boter</span>
</span>
</p>
<p class='ocr_par' id='par_1_6' lang='eng' title="bbox 148 979 1241 1087">
<span class='ocr_line' id='line_1_6' title="bbox 148 979 1241 1087; baseline -0.005 -21; x_size 107; x_descenders 24; x_ascenders 24">
<span class='ocrx_word' id='word_1_15' title='bbox 148 983 215 1066; x_wconf 88'>2</span>
<span class='ocrx_word' id='word_1_16' title='bbox 312 983 807 1087; x_wconf 88'>kopjes</span>
<span class='ocrx_word' id='word_1_17' title='bbox 905 979 1241 1062; x_wconf 92'>melk</span>
<p class='ocr_par' id='par_1_6' lang='eng' title="bbox 148 978 1241 1086">
<span class='ocr_line' id='line_1_6' title="bbox 148 978 1241 1086; baseline -0.005 -21; x_size 107; x_descenders 24; x_ascenders 24">
<span class='ocrx_word' id='word_1_15' title='bbox 148 982 215 1065; x_wconf 78'>2</span>
<span class='ocrx_word' id='word_1_16' title='bbox 312 982 807 1086; x_wconf 78'>kopjes</span>
<span class='ocrx_word' id='word_1_17' title='bbox 905 978 1241 1061; x_wconf 90'>melk</span>
</span>
</p>
<p class='ocr_par' id='par_1_7' lang='eng' title="bbox 154 1109 2929 1221">
<span class='ocr_line' id='line_1_7' title="bbox 154 1109 2929 1221; baseline -0.003 -21; x_size 112; x_descenders 28; x_ascenders 26">
<span class='ocrx_word' id='word_1_18' title='bbox 154 1117 791 1221; x_wconf 92'>laurier,</span>
<span class='ocrx_word' id='word_1_19' title='bbox 906 1111 1810 1219; x_wconf 90'>kruidnagel,</span>
<span class='ocrx_word' id='word_1_20' title='bbox 1927 1109 2490 1219; x_wconf 90'>kerrie,</span>
<span class='ocrx_word' id='word_1_21' title='bbox 2615 1110 2929 1195; x_wconf 91'>zout</span>
<p class='ocr_par' id='par_1_7' lang='eng' title="bbox 154 1108 2929 1220">
<span class='ocr_line' id='line_1_7' title="bbox 154 1108 2929 1220; baseline -0.003 -21; x_size 108; x_descenders 24; x_ascenders 26">
<span class='ocrx_word' id='word_1_18' title='bbox 154 1116 791 1220; x_wconf 90'>laurier,</span>
<span class='ocrx_word' id='word_1_19' title='bbox 906 1110 1810 1218; x_wconf 91'>kruidnagel,</span>
<span class='ocrx_word' id='word_1_20' title='bbox 1927 1108 2491 1218; x_wconf 87'>kerrie,</span>
<span class='ocrx_word' id='word_1_21' title='bbox 2616 1109 2929 1194; x_wconf 91'>zout</span>
</span>
</p>
</div>
<div class='ocr_carea' id='block_1_4' title="bbox 147 1383 3706 2731">
<p class='ocr_par' id='par_1_8' lang='eng' title="bbox 147 1383 3706 2731">
<span class='ocr_line' id='line_1_8' title="bbox 149 1383 3612 1474; baseline -0.003 -2; x_size 107.01524; x_descenders 25.01524; x_ascenders 21">
<span class='ocrx_word' id='word_1_22' title='bbox 149 1395 303 1474; x_wconf 93'>De</span>
<span class='ocrx_word' id='word_1_23' title='bbox 411 1390 902 1473; x_wconf 80'>linzgen</span>
<span class='ocrx_word' id='word_1_24' title='bbox 996 1409 1497 1470; x_wconf 89'>wassen</span>
<span class='ocrx_word' id='word_1_25' title='bbox 1595 1409 1752 1470; x_wconf 88'>en</span>
<span class='ocrx_word' id='word_1_26' title='bbox 1853 1385 2166 1467; x_wconf 75'>in-l</span>
<span class='ocrx_word' id='word_1_27' title='bbox 2275 1383 2684 1466; x_wconf 93'>liter</span>
<span class='ocrx_word' id='word_1_28' title='bbox 2775 1383 3278 1464; x_wconf 91'>kokend</span>
<span class='ocrx_word' id='word_1_29' title='bbox 3368 1401 3612 1462; x_wconf 90'>wa-</span>
<div class='ocr_carea' id='block_1_4' title="bbox 147 1382 3707 2731">
<p class='ocr_par' id='par_1_8' lang='eng' title="bbox 147 1382 3707 2731">
<span class='ocr_line' id='line_1_8' title="bbox 149 1382 3613 1474; baseline -0.003 -2; x_size 107.47087; x_descenders 24.470867; x_ascenders 23">
<span class='ocrx_word' id='word_1_22' title='bbox 149 1394 303 1474; x_wconf 93'>De</span>
<span class='ocrx_word' id='word_1_23' title='bbox 411 1389 902 1473; x_wconf 69'>linzen</span>
<span class='ocrx_word' id='word_1_24' title='bbox 997 1408 1497 1470; x_wconf 89'>wassen</span>
<span class='ocrx_word' id='word_1_25' title='bbox 1596 1408 1752 1469; x_wconf 92'>en</span>
<span class='ocrx_word' id='word_1_26' title='bbox 1853 1384 2167 1467; x_wconf 87'>in</span>
<span class='ocrx_word' id='word_1_27' title='bbox 2088 1378 2166 1494; x_wconf 53'>1</span>
<span class='ocrx_word' id='word_1_28' title='bbox 2276 1382 2685 1466; x_wconf 92'>liter</span>
<span class='ocrx_word' id='word_1_29' title='bbox 2776 1382 3278 1464; x_wconf 90'>kokend</span>
<span class='ocrx_word' id='word_1_30' title='bbox 3369 1401 3613 1462; x_wconf 88'>wa-</span>
</span>
<span class='ocr_line' id='line_1_9' title="bbox 157 1516 3520 1632; baseline -0.004 -21; x_size 110; x_descenders 25; x_ascenders 25">
<span class='ocrx_word' id='word_1_30' title='bbox 157 1531 394 1611; x_wconf 94'>ter</span>
<span class='ocrx_word' id='word_1_31' title='bbox 495 1527 558 1609; x_wconf 93'>1</span>
<span class='ocrx_word' id='word_1_32' title='bbox 658 1530 903 1632; x_wconf 84'>dag</span>
<span class='ocrx_word' id='word_1_33' title='bbox 1002 1527 1415 1609; x_wconf 90'>laten</span>
<span class='ocrx_word' id='word_1_34' title='bbox 1505 1525 1979 1611; x_wconf 58'>weken,</span>
<span class='ocrx_word' id='word_1_35' title='bbox 2103 1521 2167 1603; x_wconf 96'>2</span>
<span class='ocrx_word' id='word_1_36' title='bbox 2275 1518 2683 1603; x_wconf 83'>liter</span>
<span class='ocrx_word' id='word_1_37' title='bbox 2777 1519 3194 1601; x_wconf 96'>water</span>
<span class='ocrx_word' id='word_1_38' title='bbox 3286 1516 3520 1599; x_wconf 89'>bij</span>
<span class='ocr_line' id='line_1_9' title="bbox 157 1516 3521 1632; baseline -0.004 -21; x_size 109; x_descenders 25; x_ascenders 24">
<span class='ocrx_word' id='word_1_31' title='bbox 157 1531 394 1611; x_wconf 93'>ter</span>
<span class='ocrx_word' id='word_1_32' title='bbox 495 1527 558 1609; x_wconf 92'>1</span>
<span class='ocrx_word' id='word_1_33' title='bbox 658 1530 903 1632; x_wconf 84'>dag</span>
<span class='ocrx_word' id='word_1_34' title='bbox 1002 1527 1415 1609; x_wconf 90'>laten</span>
<span class='ocrx_word' id='word_1_35' title='bbox 1505 1525 1979 1611; x_wconf 87'>weken,</span>
<span class='ocrx_word' id='word_1_36' title='bbox 2104 1521 2168 1603; x_wconf 96'>2</span>
<span class='ocrx_word' id='word_1_37' title='bbox 2276 1518 2684 1603; x_wconf 85'>liter</span>
<span class='ocrx_word' id='word_1_38' title='bbox 2778 1519 3195 1601; x_wconf 96'>water</span>
<span class='ocrx_word' id='word_1_39' title='bbox 3287 1516 3521 1599; x_wconf 92'>bij</span>
</span>
<span class='ocr_line' id='line_1_10' title="bbox 152 1651 3616 1767; baseline -0.004 -21; x_size 110; x_descenders 27; x_ascenders 23">
<span class='ocrx_word' id='word_1_39' title='bbox 152 1668 302 1747; x_wconf 92'>de</span>
<span class='ocrx_word' id='word_1_40' title='bbox 407 1662 905 1747; x_wconf 91'>linzen</span>
<span class='ocrx_word' id='word_1_41' title='bbox 996 1682 1559 1767; x_wconf 92'>voegen,</span>
<span class='ocrx_word' id='word_1_42' title='bbox 1680 1659 2176 1741; x_wconf 96'>zonder</span>
<span class='ocrx_word' id='word_1_43' title='bbox 2267 1655 2506 1737; x_wconf 96'>het</span>
<span class='ocrx_word' id='word_1_44' title='bbox 2606 1655 3023 1737; x_wconf 92'>water</span>
<span class='ocrx_word' id='word_1_45' title='bbox 3116 1651 3616 1735; x_wconf 91'>waarin</span>
<span class='ocr_line' id='line_1_10' title="bbox 152 1651 3617 1766; baseline -0.004 -20; x_size 108; x_descenders 25; x_ascenders 23">
<span class='ocrx_word' id='word_1_40' title='bbox 152 1668 302 1747; x_wconf 90'>de</span>
<span class='ocrx_word' id='word_1_41' title='bbox 407 1662 905 1747; x_wconf 90'>linzen</span>
<span class='ocrx_word' id='word_1_42' title='bbox 996 1682 1559 1766; x_wconf 89'>voegen,</span>
<span class='ocrx_word' id='word_1_43' title='bbox 1680 1659 2176 1741; x_wconf 96'>zonder</span>
<span class='ocrx_word' id='word_1_44' title='bbox 2268 1655 2507 1737; x_wconf 95'>het</span>
<span class='ocrx_word' id='word_1_45' title='bbox 2607 1655 3024 1737; x_wconf 93'>water</span>
<span class='ocrx_word' id='word_1_46' title='bbox 3117 1651 3617 1735; x_wconf 91'>waarin</span>
</span>
<span class='ocr_line' id='line_1_11' title="bbox 153 1782 3704 1905; baseline -0.004 -20; x_size 117; x_descenders 23; x_ascenders 34">
<span class='ocrx_word' id='word_1_46' title='bbox 153 1824 305 1885; x_wconf 91'>ze</span>
<span class='ocrx_word' id='word_1_47' title='bbox 407 1800 979 1905; x_wconf 85'>geweekt</span>
<span class='ocrx_word' id='word_1_48' title='bbox 1089 1797 1412 1900; x_wconf 96'>zijn</span>
<span class='ocrx_word' id='word_1_49' title='bbox 1510 1796 1672 1878; x_wconf 96'>af</span>
<span class='ocrx_word' id='word_1_50' title='bbox 1770 1782 1914 1876; x_wconf 93'>te</span>
<span class='ocrx_word' id='word_1_51' title='bbox 2019 1792 2576 1899; x_wconf 54'>gieten,</span>
<span class='ocrx_word' id='word_1_52' title='bbox 2690 1790 2850 1874; x_wconf 93'>De</span>
<span class='ocrx_word' id='word_1_53' title='bbox 2948 1791 3357 1872; x_wconf 89'>helft</span>
<span class='ocrx_word' id='word_1_54' title='bbox 3452 1811 3704 1873; x_wconf 96'>van</span>
<span class='ocr_line' id='line_1_11' title="bbox 153 1782 3705 1905; baseline -0.004 -20; x_size 116; x_descenders 22; x_ascenders 34">
<span class='ocrx_word' id='word_1_47' title='bbox 153 1824 305 1885; x_wconf 91'>ze</span>
<span class='ocrx_word' id='word_1_48' title='bbox 407 1800 979 1905; x_wconf 76'>geweekt</span>
<span class='ocrx_word' id='word_1_49' title='bbox 1090 1797 1412 1900; x_wconf 96'>zijn</span>
<span class='ocrx_word' id='word_1_50' title='bbox 1510 1796 1672 1878; x_wconf 96'>af</span>
<span class='ocrx_word' id='word_1_51' title='bbox 1770 1782 1915 1876; x_wconf 93'>te</span>
<span class='ocrx_word' id='word_1_52' title='bbox 2019 1792 2577 1899; x_wconf 56'>gieten,</span>
<span class='ocrx_word' id='word_1_53' title='bbox 2691 1790 2850 1874; x_wconf 91'>De</span>
<span class='ocrx_word' id='word_1_54' title='bbox 2949 1791 3358 1872; x_wconf 90'>helft</span>
<span class='ocrx_word' id='word_1_55' title='bbox 3453 1811 3705 1873; x_wconf 96'>van</span>
</span>
<span class='ocr_line' id='line_1_12' title="bbox 151 1928 3593 2035; baseline -0.003 -13; x_size 109; x_descenders 25; x_ascenders 25">
<span class='ocrx_word' id='word_1_55' title='bbox 151 1942 305 2024; x_wconf 92'>de</span>
<span class='ocrx_word' id='word_1_56' title='bbox 403 1940 735 2021; x_wconf 89'>uien</span>
<span class='ocrx_word' id='word_1_57' title='bbox 829 1938 1330 2022; x_wconf 91'>bakken</span>
<span class='ocrx_word' id='word_1_58' title='bbox 1419 1939 1659 2018; x_wconf 92'>met</span>
<span class='ocrx_word' id='word_1_59' title='bbox 1770 1933 2346 2016; x_wconf 91'>laurier</span>
<span class='ocrx_word' id='word_1_60' title='bbox 2447 1953 2603 2014; x_wconf 91'>en</span>
<span class='ocrx_word' id='word_1_61' title='bbox 2691 1928 3593 2035; x_wconf 63'>Kruidnagel.</span>
<span class='ocr_line' id='line_1_12' title="bbox 151 1928 3594 2035; baseline -0.003 -13; x_size 109; x_descenders 25; x_ascenders 25">
<span class='ocrx_word' id='word_1_56' title='bbox 151 1942 305 2024; x_wconf 92'>de</span>
<span class='ocrx_word' id='word_1_57' title='bbox 403 1940 735 2021; x_wconf 87'>uien</span>
<span class='ocrx_word' id='word_1_58' title='bbox 829 1938 1330 2022; x_wconf 91'>bakken</span>
<span class='ocrx_word' id='word_1_59' title='bbox 1420 1939 1659 2017; x_wconf 93'>met</span>
<span class='ocrx_word' id='word_1_60' title='bbox 1770 1933 2347 2016; x_wconf 89'>laurier</span>
<span class='ocrx_word' id='word_1_61' title='bbox 2448 1953 2604 2014; x_wconf 93'>en</span>
<span class='ocrx_word' id='word_1_62' title='bbox 2692 1928 3594 2035; x_wconf 89'>kruidnagel.</span>
</span>
<span class='ocr_line' id='line_1_13' title="bbox 151 2067 3451 2180; baseline -0.004 -21; x_size 108; x_descenders 25; x_ascenders 24">
<span class='ocrx_word' id='word_1_62' title='bbox 151 2076 473 2159; x_wconf 92'>Alle</span>
<span class='ocrx_word' id='word_1_63' title='bbox 569 2076 965 2180; x_wconf 88'>uien,</span>
<span class='ocrx_word' id='word_1_64' title='bbox 1079 2072 1578 2156; x_wconf 90'>kerrie</span>
<span class='ocrx_word' id='word_1_65' title='bbox 1685 2092 1837 2153; x_wconf 93'>en</span>
<span class='ocrx_word' id='word_1_66' title='bbox 1938 2072 2254 2153; x_wconf 81'>zgout</span>
<span class='ocrx_word' id='word_1_67' title='bbox 2355 2068 2593 2172; x_wconf 43'>bij</span>
<span class='ocrx_word' id='word_1_68' title='bbox 2696 2071 2850 2150; x_wconf 91'>de</span>
<span class='ocrx_word' id='word_1_69' title='bbox 2957 2067 3451 2150; x_wconf 85'>linzen</span>
<span class='ocr_line' id='line_1_13' title="bbox 151 2067 3452 2180; baseline -0.004 -21; x_size 108; x_descenders 25; x_ascenders 24">
<span class='ocrx_word' id='word_1_63' title='bbox 151 2076 473 2159; x_wconf 92'>Alle</span>
<span class='ocrx_word' id='word_1_64' title='bbox 569 2076 965 2180; x_wconf 90'>uien,</span>
<span class='ocrx_word' id='word_1_65' title='bbox 1079 2072 1578 2156; x_wconf 91'>kerrie</span>
<span class='ocrx_word' id='word_1_66' title='bbox 1686 2092 1837 2153; x_wconf 96'>en</span>
<span class='ocrx_word' id='word_1_67' title='bbox 1938 2072 2255 2153; x_wconf 0'>gout</span>
<span class='ocrx_word' id='word_1_68' title='bbox 2356 2068 2594 2172; x_wconf 52'>bijg</span>
<span class='ocrx_word' id='word_1_69' title='bbox 2697 2071 2851 2150; x_wconf 92'>de</span>
<span class='ocrx_word' id='word_1_70' title='bbox 2958 2067 3452 2150; x_wconf 80'>linzen</span>
</span>
<span class='ocr_line' id='line_1_14' title="bbox 147 2205 3614 2318; baseline -0.003 -23; x_size 106; x_descenders 22; x_ascenders 25">
<span class='ocrx_word' id='word_1_70' title='bbox 147 2234 711 2318; x_wconf 88'>voegen,</span>
<span class='ocrx_word' id='word_1_71' title='bbox 826 2210 1234 2295; x_wconf 95'>Alles</span>
<span class='ocrx_word' id='word_1_72' title='bbox 1342 2232 1588 2295; x_wconf 95'>aan</span>
<span class='ocrx_word' id='word_1_73' title='bbox 1679 2212 1831 2291; x_wconf 96'>de</span>
<span class='ocrx_word' id='word_1_74' title='bbox 1928 2208 2263 2290; x_wconf 93'>kook</span>
<span class='ocrx_word' id='word_1_75' title='bbox 2355 2206 3000 2308; x_wconf 54'>brengen,</span>
<span class='ocrx_word' id='word_1_76' title='bbox 3117 2205 3367 2285; x_wconf 95'>Van</span>
<span class='ocrx_word' id='word_1_77' title='bbox 3462 2206 3614 2287; x_wconf 95'>de</span>
<span class='ocr_line' id='line_1_14' title="bbox 147 2205 3615 2318; baseline -0.003 -23; x_size 106; x_descenders 22; x_ascenders 25">
<span class='ocrx_word' id='word_1_71' title='bbox 147 2234 711 2318; x_wconf 89'>voegen,</span>
<span class='ocrx_word' id='word_1_72' title='bbox 826 2210 1234 2295; x_wconf 96'>Alles</span>
<span class='ocrx_word' id='word_1_73' title='bbox 1342 2232 1588 2295; x_wconf 96'>aan</span>
<span class='ocrx_word' id='word_1_74' title='bbox 1679 2212 1831 2291; x_wconf 95'>de</span>
<span class='ocrx_word' id='word_1_75' title='bbox 1928 2208 2264 2290; x_wconf 93'>kook</span>
<span class='ocrx_word' id='word_1_76' title='bbox 2356 2206 3001 2308; x_wconf 35'>brengen,.</span>
<span class='ocrx_word' id='word_1_77' title='bbox 3118 2205 3368 2285; x_wconf 96'>Van</span>
<span class='ocrx_word' id='word_1_78' title='bbox 3463 2206 3615 2287; x_wconf 96'>de</span>
</span>
<span class='ocr_line' id='line_1_15' title="bbox 152 2341 3706 2447; baseline -0.003 -18; x_size 107; x_descenders 24; x_ascenders 24">
<span class='ocrx_word' id='word_1_78' title='bbox 152 2352 565 2434; x_wconf 91'>bloem</span>
<span class='ocrx_word' id='word_1_79' title='bbox 655 2351 896 2431; x_wconf 92'>met</span>
<span class='ocrx_word' id='word_1_80' title='bbox 997 2349 1669 2431; x_wconf 90'>boter</span>
<span class='ocrx_word' id='word_1_81' title='bbox 1514 2337 1683 2455; x_wconf 91'>en</span>
<span class='ocrx_word' id='word_1_82' title='bbox 1757 2343 2095 2426; x_wconf 88'>melk</span>
<span class='ocrx_word' id='word_1_83' title='bbox 2193 2364 2432 2427; x_wconf 93'>een</span>
<span class='ocrx_word' id='word_1_84' title='bbox 2527 2341 2935 2447; x_wconf 90'>papje</span>
<span class='ocrx_word' id='word_1_85' title='bbox 3029 2341 3453 2422; x_wconf 96'>maken</span>
<span class='ocrx_word' id='word_1_86' title='bbox 3549 2362 3706 2422; x_wconf 95'>en</span>
<span class='ocr_line' id='line_1_15' title="bbox 152 2341 3707 2447; baseline -0.003 -18; x_size 107; x_descenders 24; x_ascenders 24">
<span class='ocrx_word' id='word_1_79' title='bbox 152 2352 565 2434; x_wconf 91'>bloem</span>
<span class='ocrx_word' id='word_1_80' title='bbox 655 2351 896 2431; x_wconf 92'>met</span>
<span class='ocrx_word' id='word_1_81' title='bbox 997 2349 1669 2431; x_wconf 91'>boter</span>
<span class='ocrx_word' id='word_1_82' title='bbox 1514 2337 1683 2455; x_wconf 91'>en</span>
<span class='ocrx_word' id='word_1_83' title='bbox 1757 2343 2096 2426; x_wconf 90'>melk</span>
<span class='ocrx_word' id='word_1_84' title='bbox 2194 2364 2433 2427; x_wconf 93'>een</span>
<span class='ocrx_word' id='word_1_85' title='bbox 2528 2341 2936 2447; x_wconf 90'>papje</span>
<span class='ocrx_word' id='word_1_86' title='bbox 3030 2341 3454 2422; x_wconf 96'>maken</span>
<span class='ocrx_word' id='word_1_87' title='bbox 3550 2362 3707 2422; x_wconf 95'>en</span>
</span>
<span class='ocr_line' id='line_1_16' title="bbox 149 2477 3619 2586; baseline -0.003 -16; x_size 107; x_descenders 24; x_ascenders 23">
<span class='ocrx_word' id='word_1_87' title='bbox 149 2489 650 2571; x_wconf 86'>verder</span>
<span class='ocrx_word' id='word_1_88' title='bbox 750 2486 1330 2570; x_wconf 90'>afmaken</span>
<span class='ocrx_word' id='word_1_89' title='bbox 1420 2485 1660 2567; x_wconf 96'>met</span>
<span class='ocrx_word' id='word_1_90' title='bbox 1765 2485 1917 2566; x_wconf 93'>de</span>
<span class='ocrx_word' id='word_1_91' title='bbox 2025 2505 2409 2586; x_wconf 86'>soep,</span>
<span class='ocrx_word' id='word_1_92' title='bbox 2521 2479 2764 2564; x_wconf 96'>Als</span>
<span class='ocrx_word' id='word_1_93' title='bbox 2868 2480 3021 2561; x_wconf 92'>de</span>
<span class='ocrx_word' id='word_1_94' title='bbox 3127 2477 3619 2562; x_wconf 91'>linzen</span>
<span class='ocr_line' id='line_1_16' title="bbox 149 2477 3620 2586; baseline -0.003 -16; x_size 106; x_descenders 23; x_ascenders 23">
<span class='ocrx_word' id='word_1_88' title='bbox 149 2489 650 2571; x_wconf 87'>verder</span>
<span class='ocrx_word' id='word_1_89' title='bbox 750 2486 1330 2570; x_wconf 91'>afmaken</span>
<span class='ocrx_word' id='word_1_90' title='bbox 1420 2485 1660 2567; x_wconf 95'>met</span>
<span class='ocrx_word' id='word_1_91' title='bbox 1765 2485 1918 2566; x_wconf 93'>de</span>
<span class='ocrx_word' id='word_1_92' title='bbox 2025 2505 2410 2586; x_wconf 66'>soep,</span>
<span class='ocrx_word' id='word_1_93' title='bbox 2522 2479 2765 2564; x_wconf 95'>Als</span>
<span class='ocrx_word' id='word_1_94' title='bbox 2869 2480 3022 2561; x_wconf 93'>de</span>
<span class='ocrx_word' id='word_1_95' title='bbox 3128 2477 3620 2562; x_wconf 85'>linzen</span>
</span>
<span class='ocr_line' id='line_1_17' title="bbox 155 2619 2412 2731; baseline -0.005 -20; x_size 98; x_descenders 15; x_ascenders 23">
<span class='ocrx_word' id='word_1_95' title='bbox 155 2647 481 2731; x_wconf 35'>gfgaar</span>
<span class='ocrx_word' id='word_1_96' title='bbox 581 2624 909 2728; x_wconf 87'>Zijn</span>
<span class='ocrx_word' id='word_1_97' title='bbox 1005 2623 1153 2707; x_wconf 95'>is</span>
<span class='ocrx_word' id='word_1_98' title='bbox 1255 2624 1409 2706; x_wconf 93'>de</span>
<span class='ocrx_word' id='word_1_99' title='bbox 1522 2642 1840 2726; x_wconf 91'>soep</span>
<span class='ocrx_word' id='word_1_100' title='bbox 1929 2619 2412 2709; x_wconf 89'>klaar.</span>
<span class='ocr_line' id='line_1_17' title="bbox 155 2619 2413 2731; baseline -0.005 -20; x_size 108; x_descenders 25; x_ascenders 23">
<span class='ocrx_word' id='word_1_96' title='bbox 155 2647 481 2731; x_wconf 47'>gaar</span>
<span class='ocrx_word' id='word_1_97' title='bbox 581 2624 909 2728; x_wconf 88'>Zijn</span>
<span class='ocrx_word' id='word_1_98' title='bbox 1005 2623 1153 2707; x_wconf 95'>is</span>
<span class='ocrx_word' id='word_1_99' title='bbox 1255 2624 1409 2706; x_wconf 93'>de</span>
<span class='ocrx_word' id='word_1_100' title='bbox 1522 2642 1840 2726; x_wconf 91'>soep</span>
<span class='ocrx_word' id='word_1_101' title='bbox 1929 2619 2413 2709; x_wconf 86'>klaar.</span>
</span>
</p>
</div>
tests/cache/2400dpi/__-l__eng__000001_ocr.png__000001_ocr_hocr__hocr__txt/txt.bin
Vendored
+6 -6
View File
@@ -1,4 +1,4 @@
i a la Waterman
Linzensoep a la Waterman
4 ons linzen
@@ -12,13 +12,13 @@ bloem, boter
laurier, kruidnagel, kerrie, zout
De linzgen wassen en in-l liter kokend wa-
De linzen wassen en in 1 liter kokend wa-
ter 1 dag laten weken, 2 liter water bij
de linzen voegen, zonder het water waarin
ze geweekt zijn af te gieten, De helft van
de uien bakken met laurier en Kruidnagel.
Alle uien, kerrie en zgout bij de linzen
voegen, Alles aan de kook brengen, Van de
de uien bakken met laurier en kruidnagel.
Alle uien, kerrie en gout bijg de linzen
voegen, Alles aan de kook brengen,. Van de
bloem met boter en melk een papje maken en
verder afmaken met de soep, Als de linzen
gfgaar Zijn is de soep klaar.
gaar Zijn is de soep klaar.
tests/cache/2400dpi/__-l__eng__000001_ocr.png__000001_ocr_tess__pdf__txt/pdf.bin
Vendored
BIN
View File
Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More