From 3b9367fc69f563050550aa78a39caae47a4d5d3e Mon Sep 17 00:00:00 2001 From: "James R. Barlow" Date: Thu, 17 Apr 2025 02:27:59 -0700 Subject: [PATCH] Continuing rst -> md --- docs/cloud.md | 84 ++++++++++++++++++ docs/cloud.rst | 92 -------------------- docs/design_notes.md | 30 +++++++ docs/design_notes.rst | 32 ------- docs/errors.md | 51 +++++++++++ docs/errors.rst | 57 ------------ docs/jbig2.md | 74 ++++++++++++++++ docs/jbig2.rst | 81 ----------------- docs/maintainers.md | 71 +++++++++++++++ docs/maintainers.rst | 77 ---------------- docs/optimizer.md | 100 +++++++++++++++++++++ docs/optimizer.rst | 100 --------------------- docs/{pdfsecurity.rst => pdfsecurity.md} | 106 ++++++++++------------- 13 files changed, 457 insertions(+), 498 deletions(-) create mode 100644 docs/cloud.md delete mode 100644 docs/cloud.rst create mode 100644 docs/design_notes.md delete mode 100644 docs/design_notes.rst create mode 100644 docs/errors.md delete mode 100644 docs/errors.rst create mode 100644 docs/jbig2.md delete mode 100644 docs/jbig2.rst create mode 100644 docs/maintainers.md delete mode 100644 docs/maintainers.rst create mode 100644 docs/optimizer.md delete mode 100644 docs/optimizer.rst rename docs/{pdfsecurity.rst => pdfsecurity.md} (53%) diff --git a/docs/cloud.md b/docs/cloud.md new file mode 100644 index 00000000..520df176 --- /dev/null +++ b/docs/cloud.md @@ -0,0 +1,84 @@ +% SPDX-FileCopyrightText: 2025 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +{#ocr-service} + +# Online deployments + +OCRmyPDF is designed to be used as a command line tool, but it can be +used in a web service. This document describes some considerations for +doing so. + +A basic web service implementation is provided in the source code +repository, as `misc/webservice.py`. It is only demonstration quality +and is not intended for production use. + +OCRmyPDF is not designed for use as a public web service where a +malicious user could upload a chosen PDF. In particular, it is not +necessarily secure against PDF malware or PDFs that cause denial of +service. For further discussino of security, see +[security](security). + +OCRmyPDF relies on Ghostscript, and therefore, if deployed online one +should be prepared to comply with Ghostscript\'s Affero GPL license, and +any other licenses. + +Setting aside these concerns, a side effect of OCRmyPDF is that it may +incidentally sanitize PDFs containing certain types of malware. It +repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF +structures that are part of an attack. When PDF/A output is selected +(the default), the input PDF is partially reconstructed by Ghostscript. +When `--force-ocr` is used, all pages are rasterized and reconverted to +PDF, which could remove malware in embedded images. + +## Limiting CPU usage + +OCRmyPDF will attempt to use all available CPUs and storage, so +executing `nice ocrmypdf` or limiting the number of jobs with the +`--jobs` argument may ensure the server remains responsive. Another +option would be to run OCRmyPDF jobs inside a Docker container, a +virtual machine, or a cloud instance, which can impose its own limits on +CPU usage and be terminated \"from orbit\" if it fails to complete. + +## Temporary storage requirements + +OCRmyPDF will use a large amount of temporary storage for its work, +proportional to the total number of pixels needed to rasterize the PDF. +The raster image of a 8.5×11\" color page at 300 DPI takes 25 MB +uncompressed; OCRmyPDF saves its intermediates as PNG, but that still +means it requires about 9 MB per intermediate based on average +compression ratios. Multiple intermediates per page are also required, +depending on the command line given. A rule of thumb would be to allow +100 MB of temporary storage per page in a file -- meaning that a small +cloud servers or small VM partitions should be provisioned with plenty +of extra space, if say, a 500 page file might be sent. + +To change the temporary directory, see [tmpdir](advanced#tmpdir). + +On Amazon Web Services or other cloud vendors, consider setting your +temporary directory to [empheral +storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html). + +## Timeouts + +To prevent excessively long OCR jobs consider setting +`--tesseract-timeout` and/or `--skip-big` arguments. `--skip-big` is +particularly helpful if your PDFs include documents such as reports on +standard page sizes with large images attached - often large images are +not worth OCR\'ing anyway. + +## Document management systems + +If you are looking for a full document management system, consider +[paperless-ngx](https://github.com/paperless-ngx/paperless-ngx), which +is a web application that uses OCRmyPDF to automatically OCR and archive +documents. + +## Commercial OCR alternatives + +The author also provides professional services that include OCR and +building databases around PDFs, and is happy to provide consultation. + +Abbyy Cloud OCR is viable commercial alternative with a web services +API. Amazon Textract, Google Cloud Vision, and Microsoft Azure Computer +Vision provide advanced OCR but have less PDF rendering capability. diff --git a/docs/cloud.rst b/docs/cloud.rst deleted file mode 100644 index 2798c672..00000000 --- a/docs/cloud.rst +++ /dev/null @@ -1,92 +0,0 @@ -.. SPDX-FileCopyrightText: 2023 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 - - -.. _ocr-service: - -================== -Online deployments -================== - -OCRmyPDF is designed to be used as a command line tool, but it can be -used in a web service. This document describes some considerations for -doing so. - -A basic web service implementation is provided in the source code -repository, as ``misc/webservice.py``. It is only demonstration quality -and is not intended for production use. - -OCRmyPDF is not designed for use as a public web service where a -malicious user could upload a chosen PDF. In particular, it is not -necessarily secure against PDF malware or PDFs that cause denial of -service. For further discussino of security, see :ref:`security`. - -OCRmyPDF relies on Ghostscript, and therefore, if deployed -online one should be prepared to comply with Ghostscript's Affero GPL -license, and any other licenses. - -Setting aside these concerns, a side effect of OCRmyPDF is that it may -incidentally sanitize PDFs containing certain types of malware. It -repairs the PDF with pikepdf/libqpdf, which could correct malformed PDF -structures that are part of an attack. When PDF/A output is selected -(the default), the input PDF is partially reconstructed by Ghostscript. -When ``--force-ocr`` is used, all pages are rasterized and reconverted -to PDF, which could remove malware in embedded images. - -Limiting CPU usage ------------------- - -OCRmyPDF will attempt to use all available CPUs and storage, so -executing ``nice ocrmypdf`` or limiting the number of jobs with the -``--jobs`` argument may ensure the server remains responsive. Another option -would be to run OCRmyPDF jobs inside a Docker container, a virtual machine, -or a cloud instance, which can impose its own limits on CPU usage and be -terminated "from orbit" if it fails to complete. - -Temporary storage requirements ------------------------------- - -OCRmyPDF will use a large amount of temporary storage for its work, -proportional to the total number of pixels needed to rasterize the PDF. -The raster image of a 8.5×11" color page at 300 DPI takes 25 MB -uncompressed; OCRmyPDF saves its intermediates as PNG, but that still -means it requires about 9 MB per intermediate based on average -compression ratios. Multiple intermediates per page are also required, -depending on the command line given. A rule of thumb would be to allow -100 MB of temporary storage per page in a file – meaning that a small -cloud servers or small VM partitions should be provisioned with plenty -of extra space, if say, a 500 page file might be sent. - -To change the temporary directory, see :ref:`tmpdir`. - -On Amazon Web Services or other cloud vendors, consider setting your -temporary directory to `empheral -storage `__. - -Timeouts --------- - -To prevent excessively long OCR jobs consider setting -``--tesseract-timeout`` and/or ``--skip-big`` arguments. ``--skip-big`` -is particularly helpful if your PDFs include documents such as reports -on standard page sizes with large images attached - often large images -are not worth OCR'ing anyway. - -Document management systems ---------------------------- - -If you are looking for a full document management system, consider -`paperless-ngx `__, -which is a web application that uses OCRmyPDF to automatically OCR and -archive documents. - -Commercial OCR alternatives ---------------------------- - -The author also provides professional services that include OCR and -building databases around PDFs, and is happy to provide consultation. - -Abbyy Cloud OCR is viable commercial alternative with a web services -API. Amazon Textract, Google Cloud Vision, and Microsoft Azure -Computer Vision provide advanced OCR but have less PDF rendering capability. \ No newline at end of file diff --git a/docs/design_notes.md b/docs/design_notes.md new file mode 100644 index 00000000..4d8b761a --- /dev/null +++ b/docs/design_notes.md @@ -0,0 +1,30 @@ +% SPDX-FileCopyrightText: 2023 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +# Design notes + +## Why doesn\'t OCRmyPDF use PyTesseract? + +PyTesseract is a Python wrapper around the Tesseract OCR engine. When +OCRmyPDF was first written, PyTesseract used ABI bindings to call the +Tesseract library. This was not a good fit for OCRmyPDF because ABI +bindings can be fragile. + +PyTesseract has since evolved calling the Tesseract executable, +abandoning the ABI approach and using the CLI instead, just like +OCRmyPDF does. If it were written from scratch today, OCRmyPDF might use +PyTesseract. + +PyTesseract has more features don\'t particularly need PDF output, but +less features than OCRmyPDF\'s API for creating PDFs. + +## What is `executor()`? + +OCRmyPDF uses a custom concurrent executor which can support either +threads or processes with the same interface. This is useful because +OCRmyPDF can use either threads or processes to parallelize work, +whichever is more appropriate for the task at hand. + +The interface is currently private and subject to change. In particular, +if experiments with asyncio and anyio are successful, the interface will +change. diff --git a/docs/design_notes.rst b/docs/design_notes.rst deleted file mode 100644 index d4dbd307..00000000 --- a/docs/design_notes.rst +++ /dev/null @@ -1,32 +0,0 @@ -.. SPDX-FileCopyrightText: 2023 James R. Barlow -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -============ -Design notes -============ - -Why doesn't OCRmyPDF use PyTesseract? -===================================== - -PyTesseract is a Python wrapper around the Tesseract OCR engine. When OCRmyPDF was -first written, PyTesseract used ABI bindings to call the Tesseract library. This -was not a good fit for OCRmyPDF because ABI bindings can be fragile. - -PyTesseract has since evolved calling the Tesseract executable, abandoning the ABI -approach and using the CLI instead, just like OCRmyPDF does. If it were written from -scratch today, OCRmyPDF might use PyTesseract. - -PyTesseract has more features don't particularly need PDF output, but less features -than OCRmyPDF's API for creating PDFs. - -What is ``executor()``? -======================= - -OCRmyPDF uses a custom concurrent executor which can support either threads or -processes with the same interface. This is useful because OCRmyPDF can use -either threads or processes to parallelize work, whichever is more appropriate -for the task at hand. - -The interface is currently private and subject to change. In particular, if -experiments with asyncio and anyio are successful, the interface will change. - diff --git a/docs/errors.md b/docs/errors.md new file mode 100644 index 00000000..01b51b6c --- /dev/null +++ b/docs/errors.md @@ -0,0 +1,51 @@ +% SPDX-FileCopyrightText: 2022 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +# Common error messages + +## Page already has text + +:::{code} +ERROR - 1: page already has text! – aborting (use --force-ocr to force OCR) +::: + +You ran ocrmypdf on a file that already contains printable text or a +hidden OCR text layer (it can\'t quite tell the difference). You +probably don\'t want to do this, because the file is already searchable. + +As the error message suggests, your options are: + +- `ocrmypdf --force-ocr` to + `rasterize `{.interpreted-text role="ref"} all vector + content and run OCR on the images. This is useful if a previous OCR + program failed, or if the document contains a text watermark. +- `ocrmypdf --skip-text` to skip OCR and other processing on any pages + that contain text. Text pages will be copied into the output PDF + without modification. +- `ocrmypdf --redo-ocr` to scan the file for any existing OCR + (non-printing text), remove it, and do OCR again. This is one way to + take advantage of improvements in OCR accuracy. Printable vector + text is excluded from OCR, so this can be used on files that contain + a mix of digital and scanned files. + +## Input file \'filename\' is not a valid PDF + +OCRmyPDF checks files with pikepdf, a library that in turn uses libqpdf +to fixes errors in PDFs, before it tries to work on them. In most cases +this happens because the PDF is corrupt and truncated (incomplete file +copying) and not much can be done. + +You can try rewriting the file with Ghostscript: + +:::{code} bash +gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf +::: + +`pdftk` can also rewrite PDFs: + +:::{code} bash +pdftk input.pdf cat output output.pdf +::: + +Sometimes Acrobat can repair PDFs with its [Preflight +tool](https://helpx.adobe.com/acrobat/using/correcting-problem-areas-preflight-tool.html). diff --git a/docs/errors.rst b/docs/errors.rst deleted file mode 100644 index d6017f10..00000000 --- a/docs/errors.rst +++ /dev/null @@ -1,57 +0,0 @@ -.. SPDX-FileCopyrightText: 2022 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -===================== -Common error messages -===================== - -Page already has text -===================== - -.. code-block:: - - ERROR - 1: page already has text! – aborting (use --force-ocr to force OCR) - -You ran ocrmypdf on a file that already contains printable text or a -hidden OCR text layer (it can't quite tell the difference). You probably -don't want to do this, because the file is already searchable. - -As the error message suggests, your options are: - -- ``ocrmypdf --force-ocr`` to :ref:`rasterize ` all - vector content and run OCR on the images. This is useful if a - previous OCR program failed, or if the document contains a text - watermark. -- ``ocrmypdf --skip-text`` to skip OCR and other processing on any - pages that contain text. Text pages will be copied into the output - PDF without modification. -- ``ocrmypdf --redo-ocr`` to scan the file for any existing OCR - (non-printing text), remove it, and do OCR again. This is one way - to take advantage of improvements in OCR accuracy. Printable vector - text is excluded from OCR, so this can be used on files that contain - a mix of digital and scanned files. - - -Input file 'filename' is not a valid PDF -======================================== - -OCRmyPDF checks files with pikepdf, a library that in turn uses libqpdf to fixes -errors in PDFs, before it tries to work on them. In most cases this happens -because the PDF is corrupt and truncated (incomplete file copying) and not much -can be done. - -You can try rewriting the file with Ghostscript: - -.. code-block:: bash - - gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf - -``pdftk`` can also rewrite PDFs: - -.. code-block:: bash - - pdftk input.pdf cat output output.pdf - -Sometimes Acrobat can repair PDFs with its `Preflight -tool `__. diff --git a/docs/jbig2.md b/docs/jbig2.md new file mode 100644 index 00000000..4af41e7b --- /dev/null +++ b/docs/jbig2.md @@ -0,0 +1,74 @@ +% SPDX-FileCopyrightText: 2022 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +{#jbig2} + +# Installing the JBIG2 encoder + +Most Linux distributions do not include a JBIG2 encoder since JBIG2 +encoding was patented for a long time. All known JBIG2 US patents have +expired as of 2017, but it is possible that unknown patents exist. + +JBIG2 encoding is recommended for OCRmyPDF and is used to losslessly +create smaller PDFs. If JBIG2 encoding is not available, lower quality +CCITT encoding will be used for monochrome images. + +JBIG2 decoding is not patented and is performed automatically by most +PDF viewers. It is widely supported and has been part of the PDF +specification since 2001. + +JBIG encoding is automatically provided by these OCRmyPDF packages: - +Docker image (both Ubuntu and Alpine) - Snap package - ArchLinux AUR +package - Alpine Linux package - Homebrew on macOS + +For all other platforms, you would need to build the JBIG2 encoder from +source: + +:::{code} bash +git clone https://github.com/agl/jbig2enc +cd jbig2enc +./autogen.sh +./configure && make +[sudo] make install +::: + +Dependencies include libtoolize and libleptonica, which on Ubuntu +systems are packaged as libtool and libleptonica-dev. On Fedora (35) +they are packaged as libtool and leptonica-devel. For this to work, +please make sure to install `autotools`, `automake`, `libtool` and +`leptonica` first if not already installed. + +:::{code} bash +[sudo] apt install autotools-dev automake libtool libleptonica-dev +::: + +{#jbig2-lossy} + +## Lossy mode JBIG2 + +OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially +dangerous feature. Users should [review the technical concerns with +JBIG2 in lossy mode](https://en.wikipedia.org/wiki/JBIG2#Disadvantages) +and decide if this feature is acceptable for their use case. In general, +this mode should not be used for archival purposes, should not be used +when the original document is not available or will be destroyed, and +should not be used when numbers present in the document are important, +because there is a risk of 6/8 and 8/6 substitution errors. + +JBIG2 lossy mode does achieve higher compression ratios than any other +monochrome (bitonal) compression technology; for large text documents +the savings are considerable. JBIG2 lossless still gives great +compression ratios and is a major improvement over the older CCITT G4 +standard. + +To turn on JBIG2 lossy mode, add the argument `--jbig2-lossy`. +`--optimize {1,2,3}` are necessary for the argument to take effect also +required. Also, a JBIG2 encoder must be installed as described in the +previous section. + +You can adjust the threshold for JBIG2 compression with the +`--jbig2-threshold`. The default is 0.85, meaning that if two symbols +are 85% similar, they will be compressed together. + +*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by +default.* diff --git a/docs/jbig2.rst b/docs/jbig2.rst deleted file mode 100644 index 41bdee87..00000000 --- a/docs/jbig2.rst +++ /dev/null @@ -1,81 +0,0 @@ -.. SPDX-FileCopyrightText: 2022 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -.. _jbig2: - -============================ -Installing the JBIG2 encoder -============================ - -Most Linux distributions do not include a JBIG2 encoder since JBIG2 -encoding was patented for a long time. All known JBIG2 US patents have -expired as of 2017, but it is possible that unknown patents exist. - -JBIG2 encoding is recommended for OCRmyPDF and is used to losslessly -create smaller PDFs. If JBIG2 encoding is not available, lower quality -CCITT encoding will be used for monochrome images. - -JBIG2 decoding is not patented and is performed automatically by most -PDF viewers. It is widely supported and has been part of the PDF -specification since 2001. - -JBIG encoding is automatically provided by these OCRmyPDF packages: -- Docker image (both Ubuntu and Alpine) -- Snap package -- ArchLinux AUR package -- Alpine Linux package -- Homebrew on macOS - -For all other platforms, you would need to build the JBIG2 encoder from source: - -.. code-block:: bash - - git clone https://github.com/agl/jbig2enc - cd jbig2enc - ./autogen.sh - ./configure && make - [sudo] make install - -Dependencies include libtoolize and libleptonica, which on Ubuntu systems -are packaged as libtool and libleptonica-dev. On Fedora (35) they are packaged -as libtool and leptonica-devel. For this to work, please make sure to install -``autotools``, ``automake``, ``libtool`` and ``leptonica`` first if not already -installed. - -.. code-block:: bash - - [sudo] apt install autotools-dev automake libtool libleptonica-dev - -.. _jbig2-lossy: - -Lossy mode JBIG2 -================ - -OCRmyPDF provides lossy mode JBIG2 as an advanced and potentially dangerous -feature. Users should -`review the technical concerns with JBIG2 in lossy -mode `__ -and decide if this feature is acceptable for their use case. In general, -this mode should not be used for archival purposes, should not be used when -the original document is not available or will be destroyed, and should -not be used when numbers present in the document are important, because -there is a risk of 6/8 and 8/6 substitution errors. - -JBIG2 lossy mode does achieve higher compression ratios than any other -monochrome (bitonal) compression technology; for large text documents -the savings are considerable. JBIG2 lossless still gives great -compression ratios and is a major improvement over the older CCITT G4 -standard. - -To turn on JBIG2 lossy mode, add the argument ``--jbig2-lossy``. -``--optimize {1,2,3}`` are necessary for the argument to take effect -also required. Also, a JBIG2 encoder must be installed as described in -the previous section. - -You can adjust the threshold for JBIG2 compression with the -``--jbig2-threshold``. The default is 0.85, meaning that if two symbols -are 85% similar, they will be compressed together. - -*Due to an oversight, ocrmypdf v7.0 and v7.1 used lossy mode by -default.* diff --git a/docs/maintainers.md b/docs/maintainers.md new file mode 100644 index 00000000..0a2b3fd4 --- /dev/null +++ b/docs/maintainers.md @@ -0,0 +1,71 @@ +% SPDX-FileCopyrightText: 2022 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +# Maintainer notes + +This is for those who package OCRmyPDF for downstream use. (Thank you +for your hard work.) + +## Known ports/packagers + +OCRmyPDF has been ported to many platforms already. If you are +interesting in porting to a new platform, check with +[Repology](https://repology.org/projects/?search=ocrmypdf) to see the +status of that platform. + +### Make sure you can package pikepdf + +pikepdf, created by the same author, is a mixed Python and C++14 package +with much stiffer build requirements. If you want to use OCRmyPDF on +some novel platform or distribution, first make sure you can package +pikepdf. + +### Non-Python dependencies + +Note that we have non-Python dependencies. In particular, OCRmyPDF +requires Ghostscript and Tesseract OCR to be installed and needs to be +able to locate their binaries on the system PATH. On Windows, OCRmyPDF +will also check the registry for their locations. + +Tesseract OCR relies on SIMD for performance and only has proper support +for this on ARM and x86\_64. Performance may be poor on other processor +architectures. + +### Versioning scheme + +OCRmyPDF uses hatch-vcs for versioning, which derives the version from +Git as a single source of truth. This may be unsuitable for some +distributions, e.g. to indicate that your distribution modifies OCRmyPDF +in some way. + +You can patch the `__version__` variable in `src/ocrmypdf/_version.py` +if necessary, or set the environment variable +`SETUPTOOLS_SCM_PRETEND_VERSION` to the required version, if you need to +override versioning for some reason. + +### jbig2enc + +OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some +distributions have shied away from packaging JBIG2 because it contains +patented algorithms, but all patents have expired since 2017. If +possible, consider packaging it too to improve OCRmyPDF\'s compression. + +### Command line completions + +Please ensure that command line completions are installed, as described +in the installation documentation. + +### 32-bit Linux support + +If you maintain a Linux distribution that supports 32-bit x86 or ARM, +OCRmyPDF should continue to work as long as all of its dependencies +continue to be available in 32-bit form. Please note we do not test on +32-bit platforms. + +### HEIF/HEIC + +OCRmyPDF defaults to installing the pi-heif PyPI package, which supports +converting HEIF (High Efficiency Image File Format) images to PDF from +the command line. If your distribution does not have this library +available, you can exclude it and OCRmyPDF will gracefully degrade +automatically, losing only support for this feature. diff --git a/docs/maintainers.rst b/docs/maintainers.rst deleted file mode 100644 index 33c77cb8..00000000 --- a/docs/maintainers.rst +++ /dev/null @@ -1,77 +0,0 @@ -.. SPDX-FileCopyrightText: 2022 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -================ -Maintainer notes -================ - -This is for those who package OCRmyPDF for downstream use. (Thank you -for your hard work.) - -Known ports/packagers -===================== - -OCRmyPDF has been ported to many platforms already. If you are interesting in -porting to a new platform, check with -`Repology `__ to see the status -of that platform. - -Make sure you can package pikepdf ---------------------------------- - -pikepdf, created by the same author, is a mixed Python and C++14 package with -much stiffer build requirements. If you want to use OCRmyPDF on some novel platform -or distribution, first make sure you can package pikepdf. - -Non-Python dependencies ------------------------ - -Note that we have non-Python dependencies. In particular, OCRmyPDF requires -Ghostscript and Tesseract OCR to be installed and needs to be able to locate their -binaries on the system PATH. On Windows, OCRmyPDF will also check the registry -for their locations. - -Tesseract OCR relies on SIMD for performance and only has proper support for this -on ARM and x86_64. Performance may be poor on other processor architectures. - -Versioning scheme ------------------ - -OCRmyPDF uses hatch-vcs for versioning, which derives the version from -Git as a single source of truth. This may be unsuitable for some distributions, e.g. -to indicate that your distribution modifies OCRmyPDF in some way. - -You can patch the ``__version__`` variable in ``src/ocrmypdf/_version.py`` if -necessary, or set the environment variable ``SETUPTOOLS_SCM_PRETEND_VERSION`` -to the required version, if you need to override versioning for some reason. - -jbig2enc --------- - -OCRmyPDF will use jbig2enc, a JBIG2 encoder, if one can be found. Some distributions -have shied away from packaging JBIG2 because it contains patented algorithms, but -all patents have expired since 2017. If possible, consider packaging it too to -improve OCRmyPDF's compression. - -Command line completions ------------------------- - -Please ensure that command line completions are installed, as described in the -installation documentation. - -32-bit Linux support --------------------- - -If you maintain a Linux distribution that supports 32-bit x86 or ARM, OCRmyPDF -should continue to work as long as all of its dependencies continue to be -available in 32-bit form. Please note we do not test on 32-bit platforms. - -HEIF/HEIC ---------- - -OCRmyPDF defaults to installing the pi-heif PyPI package, which supports converting -HEIF (High Efficiency Image File Format) images to PDF from the command line. -If your distribution does not have this library available, you can exclude it and -OCRmyPDF will gracefully degrade automatically, losing only support for this -feature. \ No newline at end of file diff --git a/docs/optimizer.md b/docs/optimizer.md new file mode 100644 index 00000000..aeda78a8 --- /dev/null +++ b/docs/optimizer.md @@ -0,0 +1,100 @@ +% SPDX-FileCopyrightText: 2022 James R. Barlow +% SPDX-License-Identifier: CC-BY-SA-4.0 + +# PDF optimization + +OCRmyPDF includes an image-oriented PDF optimizer. By default, the +optimizer runs with safe settings with the goal of improving compression +at no loss of quality. At higher optimization levels, lossy +optimizations may be applied and tuned. Optimization occurs after OCR, +and only if OCR succeeded. It does not perform other possible +optimizations such as deduplicating resources, consolidating fonts, +simplifying vector drawings, or anything of that nature. + +:::{list-table} OCRmyPDF optimization settings +--- +widths: 33 6 60 +header-rows: 1 +--- + +* - Optimization level + - Shorthand + - Description +* - ``--optimize 0`` + - ``-O0`` + - Disable most optimizations. +* - ``--optimize 1`` (default) + - ``-O1`` + - Safe and lossless optimizations. +* - ``--optimize 2`` + - ``-O2`` + - Safe and lossy optimizations. +* - ``--optimize 3`` + - ``-O3`` + - Aggressive lossy optimizations. +::: + +The exact type of optimizations performed will vary over time, and +depend on the availability of third-party tools. + +Despite optimizations, OCRmyPDF might still increase the overall file +size, since it must embed information about the recognized text, and +depending on the settings chosen, may not be able to represent the +output file as compactly as the input file. + +## Optimizations that always occurs + +OCRmyPDF will automatically replace obsolete or inferior compression +schemes such as RLE or LZW with superior schemes such as Deflate, and +convert monochrome images to CCITT G4. Since this is lossless, it always +occurs and there is no way to disable it. Other non-image compressed +objects are compressed as well. + +## Fast web view + +OCRmyPDF automatically optimizes PDFs for \"fast web view\" in Adobe +Acrobat\'s parlance, or equivalently, linearizes PDFs so that the +resources they reference are presented in the order a viewer needs them +for sequential display. This reduces the latency of viewing a PDF both +online and from local storage, in exchange for a slight increase in file +size. + +To disable this optimization and all others, use +`ocrmypdf --optimize 0 ...` or the shorthand `-O0`. + +Adobe Acrobat might not report the file as being \"fast web view\". + +## Lossless optimizations + +At optimization level `-O1` (the default), OCRmyPDF will also attempt +lossless image optimization. + +If a JBIG2 encoder is available, then monochrome images will be +converted to JBIG2, with the potential for huge savings on large black +and white images, since JBIG2 is far more efficient than any other +monochrome (bi-level) compression. (All known US patents related to +JBIG2 have probably expired, but it remains the responsibility of the +user to supply a JBIG2 encoder such as +[jbig2enc](https://github.com/agl/jbig2enc). OCRmyPDF does not implement +JBIG2 encoding on its own.) + +OCRmyPDF currently does not attempt to recompress losslessly compressed +objects more aggressively. + +## Lossy optimizations + +At optimization level `-O2` and `-O3`, OCRmyPDF will some attempt lossy +image optimization. + +If `pngquant` is installed, OCRmyPDF will use it to perform quantize +paletted images to reduce their size. + +The quality of JPEGs may be lowered, on the assumption that a lower +quality image may be suitable for storage after OCR. + +It is not possible to optimize all image types. Uncommon image types may +be skipped by the optimizer. + +OCRmyPDF provides `lossy mode JBIG2 `{.interpreted-text +role="ref"} as an advanced feature that additional requires the argument +`--jbig2-lossy`. diff --git a/docs/optimizer.rst b/docs/optimizer.rst deleted file mode 100644 index 8d298c85..00000000 --- a/docs/optimizer.rst +++ /dev/null @@ -1,100 +0,0 @@ -.. SPDX-FileCopyrightText: 2022 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 - -================ -PDF optimization -================ - -OCRmyPDF includes an image-oriented PDF optimizer. By default, the optimizer -runs with safe settings with the goal of improving compression at no loss of -quality. At higher optimization levels, lossy optimizations may be applied and -tuned. Optimization occurs after OCR, and only if OCR succeeded. It does not -perform other possible optimizations such as deduplicating resources, -consolidating fonts, simplifying vector drawings, or anything of that nature. - -.. list-table:: Title - :widths: 33 6 60 - :header-rows: 1 - - * - Optimization level - - Shorthand - - Description - * - ``--optimize 0`` - - ``-O0`` - - Disable most optimizations. - * - ``--optimize 1`` (default) - - ``-O1`` - - Safe and lossless optimizations. - * - ``--optimize 2`` - - ``-O2`` - - Safe and lossy optimizations. - * - ``--optimize 3`` - - ``-O3`` - - Aggressive lossy optimizations. - -The exact type of optimizations performed will vary over time, and depend on -the availability of third-party tools. - -Despite optimizations, OCRmyPDF might still increase the overall file size, -since it must embed information about the recognized text, and depending on the -settings chosen, may not be able to represent the output file as compactly as -the input file. - -Optimizations that always occurs -================================ - -OCRmyPDF will automatically replace obsolete or inferior compression schemes -such as RLE or LZW with superior schemes such as Deflate, and convert -monochrome images to CCITT G4. Since this is lossless, it always occurs and there -is no way to disable it. Other non-image compressed objects are compressed as -well. - -Fast web view -============= - -OCRmyPDF automatically optimizes PDFs for "fast web view" in Adobe Acrobat's -parlance, or equivalently, linearizes PDFs so that the resources they reference -are presented in the order a viewer needs them for sequential display. This -reduces the latency of viewing a PDF both online and from local storage, in -exchange for a slight increase in file size. - -To disable this optimization and all others, use ``ocrmypdf --optimize 0 ...`` -or the shorthand ``-O0``. - -Adobe Acrobat might not report the file as being "fast web view". - -Lossless optimizations -====================== - -At optimization level ``-O1`` (the default), OCRmyPDF will also attempt lossless -image optimization. - -If a JBIG2 encoder is available, then monochrome images will be converted to -JBIG2, with the potential for huge savings on large black and white images, -since JBIG2 is far more efficient than any other monochrome (bi-level) -compression. (All known US patents related to JBIG2 have probably expired, but -it remains the responsibility of the user to supply a JBIG2 encoder such as -`jbig2enc `__. OCRmyPDF does not implement -JBIG2 encoding on its own.) - -OCRmyPDF currently does not attempt to recompress losslessly compressed objects -more aggressively. - -Lossy optimizations -=================== - -At optimization level ``-O2`` and ``-O3``, OCRmyPDF will some attempt lossy -image optimization. - -If ``pngquant`` is installed, OCRmyPDF will use it to perform quantize paletted -images to reduce their size. - -The quality of JPEGs may be lowered, on the assumption that a lower quality -image may be suitable for storage after OCR. - -It is not possible to optimize all image types. Uncommon image types may be -skipped by the optimizer. - -OCRmyPDF provides :ref:`lossy mode JBIG2 ` as an advanced feature -that additional requires the argument ``--jbig2-lossy``. diff --git a/docs/pdfsecurity.rst b/docs/pdfsecurity.md similarity index 53% rename from docs/pdfsecurity.rst rename to docs/pdfsecurity.md index ef4056e9..73eb628e 100644 --- a/docs/pdfsecurity.rst +++ b/docs/pdfsecurity.md @@ -1,15 +1,9 @@ -.. SPDX-FileCopyrightText: 2022 James R. Barlow -.. -.. SPDX-License-Identifier: CC-BY-SA-4.0 +{#security} -.. _security: +# PDF security issues -=================== -PDF security issues -=================== - - OCRmyPDF should only be used on PDFs you trust. It is not designed to - protect you against malware. +> OCRmyPDF should only be used on PDFs you trust. It is not designed to +> protect you against malware. Recognizing that many users have an interest in handling PDFs and applying OCR to PDFs they did not generate themselves, this article @@ -18,89 +12,84 @@ themselves. The disclaimer applies: this software has no warranties of any kind. -PDFs may contain malware -======================== +## PDFs may contain malware PDF is a rich, complex file format. The official PDF 1.7 specification, ISO 32000:2008, is hundreds of pages long and references several annexes each of which are similar in length. PDFs can contain video, audio, XML, JavaScript and other programming, and forms. In some cases, they can -open internet connections to pre-selected URLs. All of these are possible -attack vectors. +open internet connections to pre-selected URLs. All of these are +possible attack vectors. -In short, PDFs `may contain -viruses `__. +In short, PDFs [may contain +viruses](https://security.stackexchange.com/questions/64052/can-a-pdf-file-contain-a-virus). If you do not trust a PDF or its source, do not open it or use OCRmyPDF on it. Consider using a Docker container or virtual machine to isolate an untrusted PDF from your system. -How OCRmyPDF processes PDFs -=========================== +## How OCRmyPDF processes PDFs OCRmyPDF must open and interpret your PDF in order to insert an OCR layer. First, it runs all PDFs through -`pikepdf `__, a library based on -`QPDF `__, a program that repairs PDFs -with syntax errors. This is done because, in the author's experience, a +[pikepdf](https://github.com/pikepdf/pikepdf), a library based on +[QPDF](https://github.com/qpdf/qpdf), a program that repairs PDFs with +syntax errors. This is done because, in the author\'s experience, a significant number of PDFs in the wild, especially those created by scanners, are not well-formed files. QPDF makes it more likely that OCRmyPDF will succeed, but offers no security guarantees. QPDF is also used to split the PDF into single page PDFs. Finally, OCRmyPDF rasterizes each page of the PDF using -`Ghostscript `__ in ``-dSAFER`` mode. +[Ghostscript](http://ghostscript.com/) in `-dSAFER` mode. Depending on the options specified, OCRmyPDF may graft the OCR layer -into the existing PDF or it may essentially reconstruct ("re-fry") a +into the existing PDF or it may essentially reconstruct (\"re-fry\") a visually identical PDF that may be quite different at the binary level. That said, OCRmyPDF is not a tool designed for sanitizing PDFs. -Password protected PDFs -======================= +## Password protected PDFs Password protected PDFs usually have two passwords, and owner and user password. When the user password is set to empty, PDF readers will open -the file automatically and mark it as "(SECURED)". Password security can -also request certain restrictions on the PDF, but anyone can remove these -restrictions if they have either the owner *or* user password. Passwords -mainly present a barrier for casual users. +the file automatically and mark it as \"(SECURED)\". Password security +can also request certain restrictions on the PDF, but anyone can remove +these restrictions if they have either the owner *or* user password. +Passwords mainly present a barrier for casual users. OCRmyPDF cannot remove passwords from PDFs. If you want to remove a -password from a PDF, you must use other software, such as ``qpdf``. +password from a PDF, you must use other software, such as `qpdf`. -If the owner and user password are set, a -password is required for ``qpdf``. If only the owner password is set, then the -password can be stripped, even if one does not have the owner password. To -remove the password from a using QPDF, use: +If the owner and user password are set, a password is required for +`qpdf`. If only the owner password is set, then the password can be +stripped, even if one does not have the owner password. To remove the +password from a using QPDF, use: -.. code-block:: bash - - qpdf --decrypt --password='abc123' input.pdf no_password.pdf +:::{code} bash +qpdf --decrypt --password='abc123' input.pdf no_password.pdf +::: Then you can run OCRmyPDF on the file. -In its default mode, OCRmyPDF generates PDF/A. Passwords may not be set on PDF/A -documents. If you want to set a password on the output PDF, you must -specify ``--output-type pdf``. +In its default mode, OCRmyPDF generates PDF/A. Passwords may not be set +on PDF/A documents. If you want to set a password on the output PDF, you +must specify `--output-type pdf`. -Signature images -================ +## Signature images -Many programs exist which are capable of inserting an image of someone's -signature. On its own, this offers no security guarantees. It is trivial -to remove the signature image and apply it to other files. This practice -offers no real security. +Many programs exist which are capable of inserting an image of +someone\'s signature. On its own, this offers no security guarantees. It +is trivial to remove the signature image and apply it to other files. +This practice offers no real security. -Digital signatures -================== +## Digital signatures Important documents can be digitally signed and certified to attest to their authorship, approval or execution of a legal agreement. OCRmyPDF will detect signed PDFs and will not modify them, unless the -``--invalidate-digital-signatures`` option is used, which will -invalidate any signatures. (The signature may still be present in the PDF -if opened, but PDF readers will not validate it.) +`--invalidate-digital-signatures` option is used, which will invalidate +any signatures. (The signature may still be present in the PDF if +opened, but PDF readers will not validate it.) A digital signature adds a cryptographic hash of the document to the document, so tamper protection is provided. That also precludes OCRmyPDF @@ -108,20 +97,19 @@ from modifying the document and preserving the signature. Digital signatures are not the same as a signature image. A digital signature is a cryptographic hash of the document that is encrypted with -the author's private key. The signature is decrypted with the author's +the author\'s private key. The signature is decrypted with the author\'s public key. The public key is usually distributed by a certificate authority. The signature is then verified by the PDF reader. If the document is modified, the signature will be invalidated. -Certificate-encrypted PDFs -========================== +## Certificate-encrypted PDFs PDFs can be encrypted with a certificate. This is a more secure form of encryption than a password. The certificate is usually issued by a -certificate authority. A certificate is used to encrypt the document using -the public key for the benefit of a specific recipient who possesses -the private key. +certificate authority. A certificate is used to encrypt the document +using the public key for the benefit of a specific recipient who +possesses the private key. OCRmyPDF cannot open certificate-encrypted PDFs. If you have the -certificate, you can use other PDF software, such as Acrobat, to -decrypt the PDF. \ No newline at end of file +certificate, you can use other PDF software, such as Acrobat, to decrypt +the PDF.