Continuing rst -> md
This commit is contained in:
James R. Barlow
@@ -0,0 +1,84 @@
|
||||
% SPDX-FileCopyrightText: 2025 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
{#ocr-service}
|
||||
|
||||
# Online deployments
|
||||
|
||||
OCRmyPDF is designed to be used as a command line tool, but it can be
|
||||
used in a web service. This document describes some considerations for
|
||||
doing so.
|
||||
|
||||
A basic web service implementation is provided in the source code
|
||||
repository, as `misc/webservice.py`. It is only demonstration quality
|
||||
and is not intended for production use.
|
||||
|
||||
OCRmyPDF is not designed for use as a public web service where a
|
||||
malicious user could upload a chosen PDF. In particular, it is not
|
||||
necessarily secure against PDF malware or PDFs that cause denial of
|
||||
service. For further discussino of security, see
|
||||
[security](security).
|
||||
|
||||
OCRmyPDF relies on Ghostscript, and therefore, if deployed online one
|
||||
should be prepared to comply with Ghostscript\'s Affero GPL license, and
|
||||
any other licenses.
|
||||
|
||||
Setting aside these concerns, a side effect of OCRmyPDF is that it may
|
||||
incidentally sanitize PDFs containing certain types of malware. It
|
||||
repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF
|
||||
structures that are part of an attack. When PDF/A output is selected
|
||||
(the default), the input PDF is partially reconstructed by Ghostscript.
|
||||
When `--force-ocr` is used, all pages are rasterized and reconverted to
|
||||
PDF, which could remove malware in embedded images.
|
||||
|
||||
## Limiting CPU usage
|
||||
|
||||
OCRmyPDF will attempt to use all available CPUs and storage, so
|
||||
executing `nice ocrmypdf` or limiting the number of jobs with the
|
||||
`--jobs` argument may ensure the server remains responsive. Another
|
||||
option would be to run OCRmyPDF jobs inside a Docker container, a
|
||||
virtual machine, or a cloud instance, which can impose its own limits on
|
||||
CPU usage and be terminated \"from orbit\" if it fails to complete.
|
||||
|
||||
## Temporary storage requirements
|
||||
|
||||
OCRmyPDF will use a large amount of temporary storage for its work,
|
||||
proportional to the total number of pixels needed to rasterize the PDF.
|
||||
The raster image of a 8.5×11\" color page at 300 DPI takes 25 MB
|
||||
uncompressed; OCRmyPDF saves its intermediates as PNG, but that still
|
||||
means it requires about 9 MB per intermediate based on average
|
||||
compression ratios. Multiple intermediates per page are also required,
|
||||
depending on the command line given. A rule of thumb would be to allow
|
||||
100 MB of temporary storage per page in a file -- meaning that a small
|
||||
cloud servers or small VM partitions should be provisioned with plenty
|
||||
of extra space, if say, a 500 page file might be sent.
|
||||
|
||||
To change the temporary directory, see [tmpdir](advanced#tmpdir).
|
||||
|
||||
On Amazon Web Services or other cloud vendors, consider setting your
|
||||
temporary directory to [empheral
|
||||
storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html).
|
||||
|
||||
## Timeouts
|
||||
|
||||
To prevent excessively long OCR jobs consider setting
|
||||
`--tesseract-timeout` and/or `--skip-big` arguments. `--skip-big` is
|
||||
particularly helpful if your PDFs include documents such as reports on
|
||||
standard page sizes with large images attached - often large images are
|
||||
not worth OCR\'ing anyway.
|
||||
|
||||
## Document management systems
|
||||
|
||||
If you are looking for a full document management system, consider
|
||||
[paperless-ngx](https://github.com/paperless-ngx/paperless-ngx), which
|
||||
is a web application that uses OCRmyPDF to automatically OCR and archive
|
||||
documents.
|
||||
|
||||
## Commercial OCR alternatives
|
||||
|
||||
The author also provides professional services that include OCR and
|
||||
building databases around PDFs, and is happy to provide consultation.
|
||||
|
||||
Abbyy Cloud OCR is viable commercial alternative with a web services
|
||||
API. Amazon Textract, Google Cloud Vision, and Microsoft Azure Computer
|
||||
Vision provide advanced OCR but have less PDF rendering capability.
|
||||
@@ -1,92 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2023 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
|
||||
.. _ocr-service:
|
||||
|
||||
==================
|
||||
Online deployments
|
||||
==================
|
||||
|
||||
OCRmyPDF is designed to be used as a command line tool, but it can be
|
||||
used in a web service. This document describes some considerations for
|
||||
doing so.
|
||||
|
||||
A basic web service implementation is provided in the source code
|
||||
repository, as ``misc/webservice.py``. It is only demonstration quality
|
||||
and is not intended for production use.
|
||||
|
||||
OCRmyPDF is not designed for use as a public web service where a
|
||||
malicious user could upload a chosen PDF. In particular, it is not
|
||||
necessarily secure against PDF malware or PDFs that cause denial of
|
||||
service. For further discussino of security, see :ref:`security`.
|
||||
|
||||
OCRmyPDF relies on Ghostscript, and therefore, if deployed
|
||||
online one should be prepared to comply with Ghostscript's Affero GPL
|
||||
license, and any other licenses.
|
||||
|
||||
Setting aside these concerns, a side effect of OCRmyPDF is that it may
|
||||
incidentally sanitize PDFs containing certain types of malware. It
|
||||
repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF
|
||||
structures that are part of an attack. When PDF/A output is selected
|
||||
(the default), the input PDF is partially reconstructed by Ghostscript.
|
||||
When ``--force-ocr`` is used, all pages are rasterized and reconverted
|
||||
to PDF, which could remove malware in embedded images.
|
||||
|
||||
Limiting CPU usage
|
||||
------------------
|
||||
|
||||
OCRmyPDF will attempt to use all available CPUs and storage, so
|
||||
executing ``nice ocrmypdf`` or limiting the number of jobs with the
|
||||
``--jobs`` argument may ensure the server remains responsive. Another option
|
||||
would be to run OCRmyPDF jobs inside a Docker container, a virtual machine,
|
||||
or a cloud instance, which can impose its own limits on CPU usage and be
|
||||
terminated "from orbit" if it fails to complete.
|
||||
|
||||
Temporary storage requirements
|
||||
------------------------------
|
||||
|
||||
OCRmyPDF will use a large amount of temporary storage for its work,
|
||||
proportional to the total number of pixels needed to rasterize the PDF.
|
||||
The raster image of a 8.5×11" color page at 300 DPI takes 25 MB
|
||||
uncompressed; OCRmyPDF saves its intermediates as PNG, but that still
|
||||
means it requires about 9 MB per intermediate based on average
|
||||
compression ratios. Multiple intermediates per page are also required,
|
||||
depending on the command line given. A rule of thumb would be to allow
|
||||
100 MB of temporary storage per page in a file – meaning that a small
|
||||
cloud servers or small VM partitions should be provisioned with plenty
|
||||
of extra space, if say, a 500 page file might be sent.
|
||||
|
||||
To change the temporary directory, see :ref:`tmpdir`.
|
||||
|
||||
On Amazon Web Services or other cloud vendors, consider setting your
|
||||
temporary directory to `empheral
|
||||
storage <https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html>`__.
|
||||
|
||||
Timeouts
|
||||
--------
|
||||
|
||||
To prevent excessively long OCR jobs consider setting
|
||||
``--tesseract-timeout`` and/or ``--skip-big`` arguments. ``--skip-big``
|
||||
is particularly helpful if your PDFs include documents such as reports
|
||||
on standard page sizes with large images attached - often large images
|
||||
are not worth OCR'ing anyway.
|
||||
|
||||
Document management systems
|
||||
---------------------------
|
||||
|
||||
If you are looking for a full document management system, consider
|
||||
`paperless-ngx <https://github.com/paperless-ngx/paperless-ngx>`__,
|
||||
which is a web application that uses OCRmyPDF to automatically OCR and
|
||||
archive documents.
|
||||
|
||||
Commercial OCR alternatives
|
||||
---------------------------
|
||||
|
||||
The author also provides professional services that include OCR and
|
||||
building databases around PDFs, and is happy to provide consultation.
|
||||
|
||||
Abbyy Cloud OCR is viable commercial alternative with a web services
|
||||
API. Amazon Textract, Google Cloud Vision, and Microsoft Azure
|
||||
Computer Vision provide advanced OCR but have less PDF rendering capability.
|
||||
@@ -0,0 +1,30 @@
|
||||
% SPDX-FileCopyrightText: 2023 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
# Design notes
|
||||
|
||||
## Why doesn\'t OCRmyPDF use PyTesseract?
|
||||
|
||||
PyTesseract is a Python wrapper around the Tesseract OCR engine. When
|
||||
OCRmyPDF was first written, PyTesseract used ABI bindings to call the
|
||||
Tesseract library. This was not a good fit for OCRmyPDF because ABI
|
||||
bindings can be fragile.
|
||||
|
||||
PyTesseract has since evolved calling the Tesseract executable,
|
||||
abandoning the ABI approach and using the CLI instead, just like
|
||||
OCRmyPDF does. If it were written from scratch today, OCRmyPDF might use
|
||||
PyTesseract.
|
||||
|
||||
PyTesseract has more features don\'t particularly need PDF output, but
|
||||
less features than OCRmyPDF\'s API for creating PDFs.
|
||||
|
||||
## What is `executor()`?
|
||||
|
||||
OCRmyPDF uses a custom concurrent executor which can support either
|
||||
threads or processes with the same interface. This is useful because
|
||||
OCRmyPDF can use either threads or processes to parallelize work,
|
||||
whichever is more appropriate for the task at hand.
|
||||
|
||||
The interface is currently private and subject to change. In particular,
|
||||
if experiments with asyncio and anyio are successful, the interface will
|
||||
change.
|
||||
@@ -1,32 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2023 James R. Barlow
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
============
|
||||
Design notes
|
||||
============
|
||||
|
||||
Why doesn't OCRmyPDF use PyTesseract?
|
||||
=====================================
|
||||
|
||||
PyTesseract is a Python wrapper around the Tesseract OCR engine. When OCRmyPDF was
|
||||
first written, PyTesseract used ABI bindings to call the Tesseract library. This
|
||||
was not a good fit for OCRmyPDF because ABI bindings can be fragile.
|
||||
|
||||
PyTesseract has since evolved calling the Tesseract executable, abandoning the ABI
|
||||
approach and using the CLI instead, just like OCRmyPDF does. If it were written from
|
||||
scratch today, OCRmyPDF might use PyTesseract.
|
||||
|
||||
PyTesseract has more features don't particularly need PDF output, but less features
|
||||
than OCRmyPDF's API for creating PDFs.
|
||||
|
||||
What is ``executor()``?
|
||||
=======================
|
||||
|
||||
OCRmyPDF uses a custom concurrent executor which can support either threads or
|
||||
processes with the same interface. This is useful because OCRmyPDF can use
|
||||
either threads or processes to parallelize work, whichever is more appropriate
|
||||
for the task at hand.
|
||||
|
||||
The interface is currently private and subject to change. In particular, if
|
||||
experiments with asyncio and anyio are successful, the interface will change.
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
% SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
# Common error messages
|
||||
|
||||
## Page already has text
|
||||
|
||||
:::{code}
|
||||
ERROR - 1: page already has text! – aborting (use --force-ocr to force OCR)
|
||||
:::
|
||||
|
||||
You ran ocrmypdf on a file that already contains printable text or a
|
||||
hidden OCR text layer (it can\'t quite tell the difference). You
|
||||
probably don\'t want to do this, because the file is already searchable.
|
||||
|
||||
As the error message suggests, your options are:
|
||||
|
||||
- `ocrmypdf --force-ocr` to
|
||||
`rasterize <raster-vector>`{.interpreted-text role="ref"} all vector
|
||||
content and run OCR on the images. This is useful if a previous OCR
|
||||
program failed, or if the document contains a text watermark.
|
||||
- `ocrmypdf --skip-text` to skip OCR and other processing on any pages
|
||||
that contain text. Text pages will be copied into the output PDF
|
||||
without modification.
|
||||
- `ocrmypdf --redo-ocr` to scan the file for any existing OCR
|
||||
(non-printing text), remove it, and do OCR again. This is one way to
|
||||
take advantage of improvements in OCR accuracy. Printable vector
|
||||
text is excluded from OCR, so this can be used on files that contain
|
||||
a mix of digital and scanned files.
|
||||
|
||||
## Input file \'filename\' is not a valid PDF
|
||||
|
||||
OCRmyPDF checks files with pikepdf, a library that in turn uses libqpdf
|
||||
to fixes errors in PDFs, before it tries to work on them. In most cases
|
||||
this happens because the PDF is corrupt and truncated (incomplete file
|
||||
copying) and not much can be done.
|
||||
|
||||
You can try rewriting the file with Ghostscript:
|
||||
|
||||
:::{code} bash
|
||||
gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf
|
||||
:::
|
||||
|
||||
`pdftk` can also rewrite PDFs:
|
||||
|
||||
:::{code} bash
|
||||
pdftk input.pdf cat output output.pdf
|
||||
:::
|
||||
|
||||
Sometimes Acrobat can repair PDFs with its [Preflight
|
||||
tool](https://helpx.adobe.com/acrobat/using/correcting-problem-areas-preflight-tool.html).
|
||||
@@ -1,57 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
=====================
|
||||
Common error messages
|
||||
=====================
|
||||
|
||||
Page already has text
|
||||
=====================
|
||||
|
||||
.. code-block::
|
||||
|
||||
ERROR - 1: page already has text! – aborting (use --force-ocr to force OCR)
|
||||
|
||||
You ran ocrmypdf on a file that already contains printable text or a
|
||||
hidden OCR text layer (it can't quite tell the difference). You probably
|
||||
don't want to do this, because the file is already searchable.
|
||||
|
||||
As the error message suggests, your options are:
|
||||
|
||||
- ``ocrmypdf --force-ocr`` to :ref:`rasterize <raster-vector>` all
|
||||
vector content and run OCR on the images. This is useful if a
|
||||
previous OCR program failed, or if the document contains a text
|
||||
watermark.
|
||||
- ``ocrmypdf --skip-text`` to skip OCR and other processing on any
|
||||
pages that contain text. Text pages will be copied into the output
|
||||
PDF without modification.
|
||||
- ``ocrmypdf --redo-ocr`` to scan the file for any existing OCR
|
||||
(non-printing text), remove it, and do OCR again. This is one way
|
||||
to take advantage of improvements in OCR accuracy. Printable vector
|
||||
text is excluded from OCR, so this can be used on files that contain
|
||||
a mix of digital and scanned files.
|
||||
|
||||
|
||||
Input file 'filename' is not a valid PDF
|
||||
========================================
|
||||
|
||||
OCRmyPDF checks files with pikepdf, a library that in turn uses libqpdf to fixes
|
||||
errors in PDFs, before it tries to work on them. In most cases this happens
|
||||
because the PDF is corrupt and truncated (incomplete file copying) and not much
|
||||
can be done.
|
||||
|
||||
You can try rewriting the file with Ghostscript:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf
|
||||
|
||||
``pdftk`` can also rewrite PDFs:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
pdftk input.pdf cat output output.pdf
|
||||
|
||||
Sometimes Acrobat can repair PDFs with its `Preflight
|
||||
tool <https://helpx.adobe.com/acrobat/using/correcting-problem-areas-preflight-tool.html>`__.
|
||||
@@ -0,0 +1,74 @@
|
||||
% SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
{#jbig2}
|
||||
|
||||
# Installing the JBIG2 encoder
|
||||
|
||||
Most Linux distributions do not include a JBIG2 encoder since JBIG2
|
||||
encoding was patented for a long time. All known JBIG2 US patents have
|
||||
expired as of 2017, but it is possible that unknown patents exist.
|
||||
|
||||
JBIG2 encoding is recommended for OCRmyPDF and is used to losslessly
|
||||
create smaller PDFs. If JBIG2 encoding is not available, lower quality
|
||||
CCITT encoding will be used for monochrome images.
|
||||
|
||||
JBIG2 decoding is not patented and is performed automatically by most
|
||||
PDF viewers. It is widely supported and has been part of the PDF
|
||||
specification since 2001.
|
||||
|
||||
JBIG encoding is automatically provided by these OCRmyPDF packages: -
|
||||
Docker image (both Ubuntu and Alpine) - Snap package - ArchLinux AUR
|
||||
package - Alpine Linux package - Homebrew on macOS
|
||||
|
||||
For all other platforms, you would need to build the JBIG2 encoder from
|
||||
source:
|
||||
|
||||
:::{code} bash
|
||||
git clone https://github.com/agl/jbig2enc
|
||||
cd jbig2enc
|
||||
./autogen.sh
|
||||
./configure && make
|
||||
[sudo] make install
|
||||
:::
|
||||
|
||||
Dependencies include libtoolize and libleptonica, which on Ubuntu
|
||||
systems are packaged as libtool and libleptonica-dev. On Fedora (35)
|
||||
they are packaged as libtool and leptonica-devel. For this to work,
|
||||
please make sure to install `autotools`, `automake`, `libtool` and
|
||||
`leptonica` first if not already installed.
|
||||
|
||||
:::{code} bash
|
||||
[sudo] apt install autotools-dev automake libtool libleptonica-dev
|
||||
:::
|
||||
|
||||
{#jbig2-lossy}
|
||||
|
||||
## Lossy mode JBIG2
|
||||
|
||||
OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially
|
||||
dangerous feature. Users should [review the technical concerns with
|
||||
JBIG2 in lossy mode](https://en.wikipedia.org/wiki/JBIG2#Disadvantages)
|
||||
and decide if this feature is acceptable for their use case. In general,
|
||||
this mode should not be used for archival purposes, should not be used
|
||||
when the original document is not available or will be destroyed, and
|
||||
should not be used when numbers present in the document are important,
|
||||
because there is a risk of 6/8 and 8/6 substitution errors.
|
||||
|
||||
JBIG2 lossy mode does achieve higher compression ratios than any other
|
||||
monochrome (bitonal) compression technology; for large text documents
|
||||
the savings are considerable. JBIG2 lossless still gives great
|
||||
compression ratios and is a major improvement over the older CCITT G4
|
||||
standard.
|
||||
|
||||
To turn on JBIG2 lossy mode, add the argument `--jbig2-lossy`.
|
||||
`--optimize {1,2,3}` are necessary for the argument to take effect also
|
||||
required. Also, a JBIG2 encoder must be installed as described in the
|
||||
previous section.
|
||||
|
||||
You can adjust the threshold for JBIG2 compression with the
|
||||
`--jbig2-threshold`. The default is 0.85, meaning that if two symbols
|
||||
are 85% similar, they will be compressed together.
|
||||
|
||||
*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by
|
||||
default.*
|
||||
@@ -1,81 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
.. _jbig2:
|
||||
|
||||
============================
|
||||
Installing the JBIG2 encoder
|
||||
============================
|
||||
|
||||
Most Linux distributions do not include a JBIG2 encoder since JBIG2
|
||||
encoding was patented for a long time. All known JBIG2 US patents have
|
||||
expired as of 2017, but it is possible that unknown patents exist.
|
||||
|
||||
JBIG2 encoding is recommended for OCRmyPDF and is used to losslessly
|
||||
create smaller PDFs. If JBIG2 encoding is not available, lower quality
|
||||
CCITT encoding will be used for monochrome images.
|
||||
|
||||
JBIG2 decoding is not patented and is performed automatically by most
|
||||
PDF viewers. It is widely supported and has been part of the PDF
|
||||
specification since 2001.
|
||||
|
||||
JBIG encoding is automatically provided by these OCRmyPDF packages:
|
||||
- Docker image (both Ubuntu and Alpine)
|
||||
- Snap package
|
||||
- ArchLinux AUR package
|
||||
- Alpine Linux package
|
||||
- Homebrew on macOS
|
||||
|
||||
For all other platforms, you would need to build the JBIG2 encoder from source:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
git clone https://github.com/agl/jbig2enc
|
||||
cd jbig2enc
|
||||
./autogen.sh
|
||||
./configure && make
|
||||
[sudo] make install
|
||||
|
||||
Dependencies include libtoolize and libleptonica, which on Ubuntu systems
|
||||
are packaged as libtool and libleptonica-dev. On Fedora (35) they are packaged
|
||||
as libtool and leptonica-devel. For this to work, please make sure to install
|
||||
``autotools``, ``automake``, ``libtool`` and ``leptonica`` first if not already
|
||||
installed.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[sudo] apt install autotools-dev automake libtool libleptonica-dev
|
||||
|
||||
.. _jbig2-lossy:
|
||||
|
||||
Lossy mode JBIG2
|
||||
================
|
||||
|
||||
OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially dangerous
|
||||
feature. Users should
|
||||
`review the technical concerns with JBIG2 in lossy
|
||||
mode <https://en.wikipedia.org/wiki/JBIG2#Disadvantages>`__
|
||||
and decide if this feature is acceptable for their use case. In general,
|
||||
this mode should not be used for archival purposes, should not be used when
|
||||
the original document is not available or will be destroyed, and should
|
||||
not be used when numbers present in the document are important, because
|
||||
there is a risk of 6/8 and 8/6 substitution errors.
|
||||
|
||||
JBIG2 lossy mode does achieve higher compression ratios than any other
|
||||
monochrome (bitonal) compression technology; for large text documents
|
||||
the savings are considerable. JBIG2 lossless still gives great
|
||||
compression ratios and is a major improvement over the older CCITT G4
|
||||
standard.
|
||||
|
||||
To turn on JBIG2 lossy mode, add the argument ``--jbig2-lossy``.
|
||||
``--optimize {1,2,3}`` are necessary for the argument to take effect
|
||||
also required. Also, a JBIG2 encoder must be installed as described in
|
||||
the previous section.
|
||||
|
||||
You can adjust the threshold for JBIG2 compression with the
|
||||
``--jbig2-threshold``. The default is 0.85, meaning that if two symbols
|
||||
are 85% similar, they will be compressed together.
|
||||
|
||||
*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by
|
||||
default.*
|
||||
@@ -0,0 +1,71 @@
|
||||
% SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
# Maintainer notes
|
||||
|
||||
This is for those who package OCRmyPDF for downstream use. (Thank you
|
||||
for your hard work.)
|
||||
|
||||
## Known ports/packagers
|
||||
|
||||
OCRmyPDF has been ported to many platforms already. If you are
|
||||
interesting in porting to a new platform, check with
|
||||
[Repology](https://repology.org/projects/?search=ocrmypdf) to see the
|
||||
status of that platform.
|
||||
|
||||
### Make sure you can package pikepdf
|
||||
|
||||
pikepdf, created by the same author, is a mixed Python and C++14 package
|
||||
with much stiffer build requirements. If you want to use OCRmyPDF on
|
||||
some novel platform or distribution, first make sure you can package
|
||||
pikepdf.
|
||||
|
||||
### Non-Python dependencies
|
||||
|
||||
Note that we have non-Python dependencies. In particular, OCRmyPDF
|
||||
requires Ghostscript and Tesseract OCR to be installed and needs to be
|
||||
able to locate their binaries on the system PATH. On Windows, OCRmyPDF
|
||||
will also check the registry for their locations.
|
||||
|
||||
Tesseract OCR relies on SIMD for performance and only has proper support
|
||||
for this on ARM and x86\_64. Performance may be poor on other processor
|
||||
architectures.
|
||||
|
||||
### Versioning scheme
|
||||
|
||||
OCRmyPDF uses hatch-vcs for versioning, which derives the version from
|
||||
Git as a single source of truth. This may be unsuitable for some
|
||||
distributions, e.g. to indicate that your distribution modifies OCRmyPDF
|
||||
in some way.
|
||||
|
||||
You can patch the `__version__` variable in `src/ocrmypdf/_version.py`
|
||||
if necessary, or set the environment variable
|
||||
`SETUPTOOLS_SCM_PRETEND_VERSION` to the required version, if you need to
|
||||
override versioning for some reason.
|
||||
|
||||
### jbig2enc
|
||||
|
||||
OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some
|
||||
distributions have shied away from packaging JBIG2 because it contains
|
||||
patented algorithms, but all patents have expired since 2017. If
|
||||
possible, consider packaging it too to improve OCRmyPDF\'s compression.
|
||||
|
||||
### Command line completions
|
||||
|
||||
Please ensure that command line completions are installed, as described
|
||||
in the installation documentation.
|
||||
|
||||
### 32-bit Linux support
|
||||
|
||||
If you maintain a Linux distribution that supports 32-bit x86 or ARM,
|
||||
OCRmyPDF should continue to work as long as all of its dependencies
|
||||
continue to be available in 32-bit form. Please note we do not test on
|
||||
32-bit platforms.
|
||||
|
||||
### HEIF/HEIC
|
||||
|
||||
OCRmyPDF defaults to installing the pi-heif PyPI package, which supports
|
||||
converting HEIF (High Efficiency Image File Format) images to PDF from
|
||||
the command line. If your distribution does not have this library
|
||||
available, you can exclude it and OCRmyPDF will gracefully degrade
|
||||
automatically, losing only support for this feature.
|
||||
@@ -1,77 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
================
|
||||
Maintainer notes
|
||||
================
|
||||
|
||||
This is for those who package OCRmyPDF for downstream use. (Thank you
|
||||
for your hard work.)
|
||||
|
||||
Known ports/packagers
|
||||
=====================
|
||||
|
||||
OCRmyPDF has been ported to many platforms already. If you are interesting in
|
||||
porting to a new platform, check with
|
||||
`Repology <https://repology.org/projects/?search=ocrmypdf>`__ to see the status
|
||||
of that platform.
|
||||
|
||||
Make sure you can package pikepdf
|
||||
---------------------------------
|
||||
|
||||
pikepdf, created by the same author, is a mixed Python and C++14 package with
|
||||
much stiffer build requirements. If you want to use OCRmyPDF on some novel platform
|
||||
or distribution, first make sure you can package pikepdf.
|
||||
|
||||
Non-Python dependencies
|
||||
-----------------------
|
||||
|
||||
Note that we have non-Python dependencies. In particular, OCRmyPDF requires
|
||||
Ghostscript and Tesseract OCR to be installed and needs to be able to locate their
|
||||
binaries on the system PATH. On Windows, OCRmyPDF will also check the registry
|
||||
for their locations.
|
||||
|
||||
Tesseract OCR relies on SIMD for performance and only has proper support for this
|
||||
on ARM and x86_64. Performance may be poor on other processor architectures.
|
||||
|
||||
Versioning scheme
|
||||
-----------------
|
||||
|
||||
OCRmyPDF uses hatch-vcs for versioning, which derives the version from
|
||||
Git as a single source of truth. This may be unsuitable for some distributions, e.g.
|
||||
to indicate that your distribution modifies OCRmyPDF in some way.
|
||||
|
||||
You can patch the ``__version__`` variable in ``src/ocrmypdf/_version.py`` if
|
||||
necessary, or set the environment variable ``SETUPTOOLS_SCM_PRETEND_VERSION``
|
||||
to the required version, if you need to override versioning for some reason.
|
||||
|
||||
jbig2enc
|
||||
--------
|
||||
|
||||
OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some distributions
|
||||
have shied away from packaging JBIG2 because it contains patented algorithms, but
|
||||
all patents have expired since 2017. If possible, consider packaging it too to
|
||||
improve OCRmyPDF's compression.
|
||||
|
||||
Command line completions
|
||||
------------------------
|
||||
|
||||
Please ensure that command line completions are installed, as described in the
|
||||
installation documentation.
|
||||
|
||||
32-bit Linux support
|
||||
--------------------
|
||||
|
||||
If you maintain a Linux distribution that supports 32-bit x86 or ARM, OCRmyPDF
|
||||
should continue to work as long as all of its dependencies continue to be
|
||||
available in 32-bit form. Please note we do not test on 32-bit platforms.
|
||||
|
||||
HEIF/HEIC
|
||||
---------
|
||||
|
||||
OCRmyPDF defaults to installing the pi-heif PyPI package, which supports converting
|
||||
HEIF (High Efficiency Image File Format) images to PDF from the command line.
|
||||
If your distribution does not have this library available, you can exclude it and
|
||||
OCRmyPDF will gracefully degrade automatically, losing only support for this
|
||||
feature.
|
||||
@@ -0,0 +1,100 @@
|
||||
% SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
% SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
# PDF optimization
|
||||
|
||||
OCRmyPDF includes an image-oriented PDF optimizer. By default, the
|
||||
optimizer runs with safe settings with the goal of improving compression
|
||||
at no loss of quality. At higher optimization levels, lossy
|
||||
optimizations may be applied and tuned. Optimization occurs after OCR,
|
||||
and only if OCR succeeded. It does not perform other possible
|
||||
optimizations such as deduplicating resources, consolidating fonts,
|
||||
simplifying vector drawings, or anything of that nature.
|
||||
|
||||
:::{list-table} OCRmyPDF optimization settings
|
||||
---
|
||||
widths: 33 6 60
|
||||
header-rows: 1
|
||||
---
|
||||
|
||||
* - Optimization level
|
||||
- Shorthand
|
||||
- Description
|
||||
* - ``--optimize 0``
|
||||
- ``-O0``
|
||||
- Disable most optimizations.
|
||||
* - ``--optimize 1`` (default)
|
||||
- ``-O1``
|
||||
- Safe and lossless optimizations.
|
||||
* - ``--optimize 2``
|
||||
- ``-O2``
|
||||
- Safe and lossy optimizations.
|
||||
* - ``--optimize 3``
|
||||
- ``-O3``
|
||||
- Aggressive lossy optimizations.
|
||||
:::
|
||||
|
||||
The exact type of optimizations performed will vary over time, and
|
||||
depend on the availability of third-party tools.
|
||||
|
||||
Despite optimizations, OCRmyPDF might still increase the overall file
|
||||
size, since it must embed information about the recognized text, and
|
||||
depending on the settings chosen, may not be able to represent the
|
||||
output file as compactly as the input file.
|
||||
|
||||
## Optimizations that always occurs
|
||||
|
||||
OCRmyPDF will automatically replace obsolete or inferior compression
|
||||
schemes such as RLE or LZW with superior schemes such as Deflate, and
|
||||
convert monochrome images to CCITT G4. Since this is lossless, it always
|
||||
occurs and there is no way to disable it. Other non-image compressed
|
||||
objects are compressed as well.
|
||||
|
||||
## Fast web view
|
||||
|
||||
OCRmyPDF automatically optimizes PDFs for \"fast web view\" in Adobe
|
||||
Acrobat\'s parlance, or equivalently, linearizes PDFs so that the
|
||||
resources they reference are presented in the order a viewer needs them
|
||||
for sequential display. This reduces the latency of viewing a PDF both
|
||||
online and from local storage, in exchange for a slight increase in file
|
||||
size.
|
||||
|
||||
To disable this optimization and all others, use
|
||||
`ocrmypdf --optimize 0 ...` or the shorthand `-O0`.
|
||||
|
||||
Adobe Acrobat might not report the file as being \"fast web view\".
|
||||
|
||||
## Lossless optimizations
|
||||
|
||||
At optimization level `-O1` (the default), OCRmyPDF will also attempt
|
||||
lossless image optimization.
|
||||
|
||||
If a JBIG2 encoder is available, then monochrome images will be
|
||||
converted to JBIG2, with the potential for huge savings on large black
|
||||
and white images, since JBIG2 is far more efficient than any other
|
||||
monochrome (bi-level) compression. (All known US patents related to
|
||||
JBIG2 have probably expired, but it remains the responsibility of the
|
||||
user to supply a JBIG2 encoder such as
|
||||
[jbig2enc](https://github.com/agl/jbig2enc). OCRmyPDF does not implement
|
||||
JBIG2 encoding on its own.)
|
||||
|
||||
OCRmyPDF currently does not attempt to recompress losslessly compressed
|
||||
objects more aggressively.
|
||||
|
||||
## Lossy optimizations
|
||||
|
||||
At optimization level `-O2` and `-O3`, OCRmyPDF will some attempt lossy
|
||||
image optimization.
|
||||
|
||||
If `pngquant` is installed, OCRmyPDF will use it to perform quantize
|
||||
paletted images to reduce their size.
|
||||
|
||||
The quality of JPEGs may be lowered, on the assumption that a lower
|
||||
quality image may be suitable for storage after OCR.
|
||||
|
||||
It is not possible to optimize all image types. Uncommon image types may
|
||||
be skipped by the optimizer.
|
||||
|
||||
OCRmyPDF provides `lossy mode JBIG2 <jbig2-lossy>`{.interpreted-text
|
||||
role="ref"} as an advanced feature that additional requires the argument
|
||||
`--jbig2-lossy`.
|
||||
@@ -1,100 +0,0 @@
|
||||
.. SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
|
||||
================
|
||||
PDF optimization
|
||||
================
|
||||
|
||||
OCRmyPDF includes an image-oriented PDF optimizer. By default, the optimizer
|
||||
runs with safe settings with the goal of improving compression at no loss of
|
||||
quality. At higher optimization levels, lossy optimizations may be applied and
|
||||
tuned. Optimization occurs after OCR, and only if OCR succeeded. It does not
|
||||
perform other possible optimizations such as deduplicating resources,
|
||||
consolidating fonts, simplifying vector drawings, or anything of that nature.
|
||||
|
||||
.. list-table:: Title
|
||||
:widths: 33 6 60
|
||||
:header-rows: 1
|
||||
|
||||
* - Optimization level
|
||||
- Shorthand
|
||||
- Description
|
||||
* - ``--optimize 0``
|
||||
- ``-O0``
|
||||
- Disable most optimizations.
|
||||
* - ``--optimize 1`` (default)
|
||||
- ``-O1``
|
||||
- Safe and lossless optimizations.
|
||||
* - ``--optimize 2``
|
||||
- ``-O2``
|
||||
- Safe and lossy optimizations.
|
||||
* - ``--optimize 3``
|
||||
- ``-O3``
|
||||
- Aggressive lossy optimizations.
|
||||
|
||||
The exact type of optimizations performed will vary over time, and depend on
|
||||
the availability of third-party tools.
|
||||
|
||||
Despite optimizations, OCRmyPDF might still increase the overall file size,
|
||||
since it must embed information about the recognized text, and depending on the
|
||||
settings chosen, may not be able to represent the output file as compactly as
|
||||
the input file.
|
||||
|
||||
Optimizations that always occurs
|
||||
================================
|
||||
|
||||
OCRmyPDF will automatically replace obsolete or inferior compression schemes
|
||||
such as RLE or LZW with superior schemes such as Deflate, and convert
|
||||
monochrome images to CCITT G4. Since this is lossless, it always occurs and there
|
||||
is no way to disable it. Other non-image compressed objects are compressed as
|
||||
well.
|
||||
|
||||
Fast web view
|
||||
=============
|
||||
|
||||
OCRmyPDF automatically optimizes PDFs for "fast web view" in Adobe Acrobat's
|
||||
parlance, or equivalently, linearizes PDFs so that the resources they reference
|
||||
are presented in the order a viewer needs them for sequential display. This
|
||||
reduces the latency of viewing a PDF both online and from local storage, in
|
||||
exchange for a slight increase in file size.
|
||||
|
||||
To disable this optimization and all others, use ``ocrmypdf --optimize 0 ...``
|
||||
or the shorthand ``-O0``.
|
||||
|
||||
Adobe Acrobat might not report the file as being "fast web view".
|
||||
|
||||
Lossless optimizations
|
||||
======================
|
||||
|
||||
At optimization level ``-O1`` (the default), OCRmyPDF will also attempt lossless
|
||||
image optimization.
|
||||
|
||||
If a JBIG2 encoder is available, then monochrome images will be converted to
|
||||
JBIG2, with the potential for huge savings on large black and white images,
|
||||
since JBIG2 is far more efficient than any other monochrome (bi-level)
|
||||
compression. (All known US patents related to JBIG2 have probably expired, but
|
||||
it remains the responsibility of the user to supply a JBIG2 encoder such as
|
||||
`jbig2enc <https://github.com/agl/jbig2enc>`__. OCRmyPDF does not implement
|
||||
JBIG2 encoding on its own.)
|
||||
|
||||
OCRmyPDF currently does not attempt to recompress losslessly compressed objects
|
||||
more aggressively.
|
||||
|
||||
Lossy optimizations
|
||||
===================
|
||||
|
||||
At optimization level ``-O2`` and ``-O3``, OCRmyPDF will some attempt lossy
|
||||
image optimization.
|
||||
|
||||
If ``pngquant`` is installed, OCRmyPDF will use it to perform quantize paletted
|
||||
images to reduce their size.
|
||||
|
||||
The quality of JPEGs may be lowered, on the assumption that a lower quality
|
||||
image may be suitable for storage after OCR.
|
||||
|
||||
It is not possible to optimize all image types. Uncommon image types may be
|
||||
skipped by the optimizer.
|
||||
|
||||
OCRmyPDF provides :ref:`lossy mode JBIG2 <jbig2-lossy>` as an advanced feature
|
||||
that additional requires the argument ``--jbig2-lossy``.
|
||||
@@ -1,15 +1,9 @@
|
||||
.. SPDX-FileCopyrightText: 2022 James R. Barlow
|
||||
..
|
||||
.. SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
{#security}
|
||||
|
||||
.. _security:
|
||||
# PDF security issues
|
||||
|
||||
===================
|
||||
PDF security issues
|
||||
===================
|
||||
|
||||
OCRmyPDF should only be used on PDFs you trust. It is not designed to
|
||||
protect you against malware.
|
||||
> OCRmyPDF should only be used on PDFs you trust. It is not designed to
|
||||
> protect you against malware.
|
||||
|
||||
Recognizing that many users have an interest in handling PDFs and
|
||||
applying OCR to PDFs they did not generate themselves, this article
|
||||
@@ -18,89 +12,84 @@ themselves.
|
||||
|
||||
The disclaimer applies: this software has no warranties of any kind.
|
||||
|
||||
PDFs may contain malware
|
||||
========================
|
||||
## PDFs may contain malware
|
||||
|
||||
PDF is a rich, complex file format. The official PDF 1.7 specification,
|
||||
ISO 32000:2008, is hundreds of pages long and references several annexes
|
||||
each of which are similar in length. PDFs can contain video, audio, XML,
|
||||
JavaScript and other programming, and forms. In some cases, they can
|
||||
open internet connections to pre-selected URLs. All of these are possible
|
||||
attack vectors.
|
||||
open internet connections to pre-selected URLs. All of these are
|
||||
possible attack vectors.
|
||||
|
||||
In short, PDFs `may contain
|
||||
viruses <https://security.stackexchange.com/questions/64052/can-a-pdf-file-contain-a-virus>`__.
|
||||
In short, PDFs [may contain
|
||||
viruses](https://security.stackexchange.com/questions/64052/can-a-pdf-file-contain-a-virus).
|
||||
|
||||
If you do not trust a PDF or its source, do not open it or use OCRmyPDF
|
||||
on it. Consider using a Docker container or virtual machine to isolate
|
||||
an untrusted PDF from your system.
|
||||
|
||||
How OCRmyPDF processes PDFs
|
||||
===========================
|
||||
## How OCRmyPDF processes PDFs
|
||||
|
||||
OCRmyPDF must open and interpret your PDF in order to insert an OCR
|
||||
layer. First, it runs all PDFs through
|
||||
`pikepdf <https://github.com/pikepdf/pikepdf>`__, a library based on
|
||||
`QPDF <https://github.com/qpdf/qpdf>`__, a program that repairs PDFs
|
||||
with syntax errors. This is done because, in the author's experience, a
|
||||
[pikepdf](https://github.com/pikepdf/pikepdf), a library based on
|
||||
[QPDF](https://github.com/qpdf/qpdf), a program that repairs PDFs with
|
||||
syntax errors. This is done because, in the author\'s experience, a
|
||||
significant number of PDFs in the wild, especially those created by
|
||||
scanners, are not well-formed files. QPDF makes it more likely that
|
||||
OCRmyPDF will succeed, but offers no security guarantees. QPDF is also
|
||||
used to split the PDF into single page PDFs.
|
||||
|
||||
Finally, OCRmyPDF rasterizes each page of the PDF using
|
||||
`Ghostscript <http://ghostscript.com/>`__ in ``-dSAFER`` mode.
|
||||
[Ghostscript](http://ghostscript.com/) in `-dSAFER` mode.
|
||||
|
||||
Depending on the options specified, OCRmyPDF may graft the OCR layer
|
||||
into the existing PDF or it may essentially reconstruct ("re-fry") a
|
||||
into the existing PDF or it may essentially reconstruct (\"re-fry\") a
|
||||
visually identical PDF that may be quite different at the binary level.
|
||||
That said, OCRmyPDF is not a tool designed for sanitizing PDFs.
|
||||
|
||||
Password protected PDFs
|
||||
=======================
|
||||
## Password protected PDFs
|
||||
|
||||
Password protected PDFs usually have two passwords, and owner and user
|
||||
password. When the user password is set to empty, PDF readers will open
|
||||
the file automatically and mark it as "(SECURED)". Password security can
|
||||
also request certain restrictions on the PDF, but anyone can remove these
|
||||
restrictions if they have either the owner *or* user password. Passwords
|
||||
mainly present a barrier for casual users.
|
||||
the file automatically and mark it as \"(SECURED)\". Password security
|
||||
can also request certain restrictions on the PDF, but anyone can remove
|
||||
these restrictions if they have either the owner *or* user password.
|
||||
Passwords mainly present a barrier for casual users.
|
||||
|
||||
OCRmyPDF cannot remove passwords from PDFs. If you want to remove a
|
||||
password from a PDF, you must use other software, such as ``qpdf``.
|
||||
password from a PDF, you must use other software, such as `qpdf`.
|
||||
|
||||
If the owner and user password are set, a
|
||||
password is required for ``qpdf``. If only the owner password is set, then the
|
||||
password can be stripped, even if one does not have the owner password. To
|
||||
remove the password from a using QPDF, use:
|
||||
If the owner and user password are set, a password is required for
|
||||
`qpdf`. If only the owner password is set, then the password can be
|
||||
stripped, even if one does not have the owner password. To remove the
|
||||
password from a using QPDF, use:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
qpdf --decrypt --password='abc123' input.pdf no_password.pdf
|
||||
:::{code} bash
|
||||
qpdf --decrypt --password='abc123' input.pdf no_password.pdf
|
||||
:::
|
||||
|
||||
Then you can run OCRmyPDF on the file.
|
||||
|
||||
In its default mode, OCRmyPDF generates PDF/A. Passwords may not be set on PDF/A
|
||||
documents. If you want to set a password on the output PDF, you must
|
||||
specify ``--output-type pdf``.
|
||||
In its default mode, OCRmyPDF generates PDF/A. Passwords may not be set
|
||||
on PDF/A documents. If you want to set a password on the output PDF, you
|
||||
must specify `--output-type pdf`.
|
||||
|
||||
Signature images
|
||||
================
|
||||
## Signature images
|
||||
|
||||
Many programs exist which are capable of inserting an image of someone's
|
||||
signature. On its own, this offers no security guarantees. It is trivial
|
||||
to remove the signature image and apply it to other files. This practice
|
||||
offers no real security.
|
||||
Many programs exist which are capable of inserting an image of
|
||||
someone\'s signature. On its own, this offers no security guarantees. It
|
||||
is trivial to remove the signature image and apply it to other files.
|
||||
This practice offers no real security.
|
||||
|
||||
Digital signatures
|
||||
==================
|
||||
## Digital signatures
|
||||
|
||||
Important documents can be digitally signed and certified to attest to
|
||||
their authorship, approval or execution of a legal agreement. OCRmyPDF
|
||||
will detect signed PDFs and will not modify them, unless the
|
||||
``--invalidate-digital-signatures`` option is used, which will
|
||||
invalidate any signatures. (The signature may still be present in the PDF
|
||||
if opened, but PDF readers will not validate it.)
|
||||
`--invalidate-digital-signatures` option is used, which will invalidate
|
||||
any signatures. (The signature may still be present in the PDF if
|
||||
opened, but PDF readers will not validate it.)
|
||||
|
||||
A digital signature adds a cryptographic hash of the document to the
|
||||
document, so tamper protection is provided. That also precludes OCRmyPDF
|
||||
@@ -108,20 +97,19 @@ from modifying the document and preserving the signature.
|
||||
|
||||
Digital signatures are not the same as a signature image. A digital
|
||||
signature is a cryptographic hash of the document that is encrypted with
|
||||
the author's private key. The signature is decrypted with the author's
|
||||
the author\'s private key. The signature is decrypted with the author\'s
|
||||
public key. The public key is usually distributed by a certificate
|
||||
authority. The signature is then verified by the PDF reader. If the
|
||||
document is modified, the signature will be invalidated.
|
||||
|
||||
Certificate-encrypted PDFs
|
||||
==========================
|
||||
## Certificate-encrypted PDFs
|
||||
|
||||
PDFs can be encrypted with a certificate. This is a more secure form of
|
||||
encryption than a password. The certificate is usually issued by a
|
||||
certificate authority. A certificate is used to encrypt the document using
|
||||
the public key for the benefit of a specific recipient who possesses
|
||||
the private key.
|
||||
certificate authority. A certificate is used to encrypt the document
|
||||
using the public key for the benefit of a specific recipient who
|
||||
possesses the private key.
|
||||
|
||||
OCRmyPDF cannot open certificate-encrypted PDFs. If you have the
|
||||
certificate, you can use other PDF software, such as Acrobat, to
|
||||
decrypt the PDF.
|
||||
certificate, you can use other PDF software, such as Acrobat, to decrypt
|
||||
the PDF.
|
||||
Reference in New Issue
Block a user