v4.3.2 release notes

ghostscript: elide overprinting to fix PDF/A errors in GS 9.20
It looks like GS 9.19 can incorrectly set overprinting for the text layer even though this makes no sense in PDF/A, or at least someone produced PDFs that have this after a Tesseract PDF -> GS PDF/A conversion. GS 9.20 complains about this. Instead of aborting, elide the feature. See http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=094d5a1880f1cb9ed320ca9353eb69436e09b594 and issue #107. It looks like it is better to elide features and warn about elision rather than abort with an error.
2016-11-10 23:16:08 -08:00 · 2016-11-10 14:48:02 -08:00 · 2016-11-09 05:41:12 -08:00 · 2016-11-07 14:36:50 -08:00 · 2016-11-07 14:36:08 -08:00 · 2016-11-07 14:35:54 -08:00
35 changed files with 1655 additions and 565 deletions
@@ -3,6 +3,7 @@
 *.sublime-*
 venv-*/
 pyvenv.cfg
+tasks.py

 # Package building
 *.egg-info/
@@ -15,6 +16,9 @@ wheelhouse/
 # Automatically generated files
 ocrmypdf/lib/_*.py
 ocrmypdf/version.py
+docs/_build/
+docs/_static/
+docs/_templates/

 # Code coverage
 .coverage
@@ -27,11 +27,17 @@ recursive-exclude tests/cache *
 include *.rst
 include pipeline.svg
 recursive-exclude .github * 
+recursive-include docs *.py
+recursive-include docs *.rst
+recursive-include docs *.svg
+recursive-exclude docs/_build *
+

 # support files
 recursive-include ocrmypdf/data *
 recursive-include share *
 include *.py
+exclude tasks.py

 # code
 recursive-include ocrmypdf *.py
@@ -2,7 +2,20 @@ OCRmyPDF
 ========

 OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to
-be searched.
+be searched or copy-pasted.
+
+.. code-block:: bash
+
+   ocrmypdf                      # it's a scriptable command line program
+      -l eng+fra                 # it supports multiple languages
+      --rotate-pages             # it can fix pages that are misrotated
+      --deskew                   # it can deskew crooked PDFs!
+      --title "My PDF"           # it can change output metadata
+      --jobs 4                   # it uses multiple cores by default
+      --output-type pdfa         # it produces PDF/A by default
+      input_scanned.pdf          # takes PDF input (or images)
+      output_searchable.pdf      # produces validated PDF output
+

 Main features
 -------------
@@ -19,7 +32,7 @@ Main features
 -  Processes pages in parallel when more than one CPU core is
   available
 -  Uses `Tesseract OCR <https://github.com/tesseract-ocr/tesseract>`_ engine
-  Supports the `39 languages <https://code.google.com/p/tesseract-ocr/downloads/list>`_ recognized by Tesseract
+-  Supports more than `100 languages <https://github.com/tesseract-ocr/tessdata>`_ recognized by Tesseract
 -  Battle-tested on thousands of PDFs, a test suite and continuous integration

 For details: please consult the `release notes <RELEASE_NOTES.rst>`_.
@@ -38,286 +51,18 @@ Linux/UNIX: I found many, but none of them were really satisfying.
 -  Or they did not produce valid PDF files (even though they were readable with my current PDF reader)
 -  On top of that none of them produced PDF/A files (format dedicated for long time storage)

-... so I decided to develop my own tool (using various existing scripts
-as an inspiration)
+...so I decided to develop my own tool (using various existing scripts
+as an inspiration). 

 Installation
 ------------

-Download OCRmyPDF here: https://github.com/jbarlow83/OCRmyPDF/releases
-
-These steps describe how to install OCRmyPDF on your system.
-
- `Installing on Debian and Ubuntu`_ (Debian stretch and Ubuntu 16.10 or later)
- `Installing the Docker image`_
- `Installing on Mac OS X`_
- `Installing on Ubuntu 14.04 LTS`_
- Installing and running on `Windows`_ using the Docker image
-
-If you prefer to install from source or install OCRmyPDF to a Python virtual environment, see steps for `Installing HEAD revision from sources`_.
-
-.. _Windows: `Installing on Windows`_
-
-
-Installing on Debian and Ubuntu
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Linux, UNIX, and macOS are supported. Windows is not directly supported but there is a Docker image available that runs on Windows.

 Users of Debian 9 or later or Ubuntu 16.10 or later may simply
 ``apt-get install ocrmypdf``.

-.. _Docker:
-
-Installing the Docker image
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-For many users, installing the Docker image will be easier than installing all of OCRmyPDF's dependencies. For Windows, it is the only option.
-
-If you have `Docker <https://docs.docker.com/>`_ installed on your system, you can install
-a Docker image of the latest release.
-
-Follow the Docker installation instructions for your platform.  If you can run this command
-successfully, your system is ready to download and execute the image:
-
-.. code-block:: bash
-
-   docker run hello-world
-   
-OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker engine host, and then follow these optional steps to enable multiple CPUs:
-
-.. code-block:: bash
-
-   # Optional step for Mac OS X users
-   docker-machine stop "yourVM"
-   VBoxManage modifyvm "yourVM" --cpus 2  # or whatever number of core is desired
-   docker-machine start "yourVM"
-   eval $(docker-machine env "yourVM")
-
-Assuming you have a Docker engine running somewhere, you can run these commands to download
-the image:
-
-.. code-block:: bash
-
-   docker pull jbarlow83/ocrmypdf
-
-Then tag it to give a more convenient name, just ocrmypdf:
-
-.. code-block:: bash
-
-   docker tag jbarlow83/ocrmypdf ocrmypdf
-
-This image contains language packs for English, French, Spanish and German. The alternative "polyglot" image provides `all available language packs <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#languages>`_:
-
-.. code-block:: bash
-
-   # Alternative step: If you need all language packs
-   docker pull jbarlow83/ocrmypdf-polyglot
-   docker tag jbarlow83/ocrmypdf-polyglot ocrmypdf
-
-You can then run ocrmypdf using the command:
-
-.. code-block:: bash
-
-   docker run ocrmypdf --help
-  
-To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`_, such as this in this template:
-
-.. code-block:: bash
-
-   docker run -v "$(pwd):/home/docker" <other docker arguments>   ocrmypdf <your arguments to ocrmypdf>
-
-In this worked example, the current working directory contains an input file called ``test.pdf`` and the output will go to ``output.pdf``: 
-
-.. code-block:: bash
-
-   docker run -v "$(pwd):/home/docker"   ocrmypdf --skip-text test.pdf output.pdf
-
-Note that ``ocrmypdf`` has its own separate ``-v VERBOSITYLEVEL`` argument to control debug verbosity. All Docker arguments should before the ``ocrmypdf`` image name and all arguments to ``ocrmypdf`` should be listed after.
-
-
-Installing on Mac OS X
-~~~~~~~~~~~~~~~~~~~~~~
-
-These instructions probably work on all Mac OS X versions later than 10.7 (Lion). OCRmyPDF is known to work on Yosemite and El Capitan, and regularly tested on El Capitan.
-
-If it's not already present, `install Homebrew <http://brew.sh/>`_.
-
-Update Homebrew:
-
-.. code-block:: bash
-
-   brew update
-   
-Install or upgrade the required Homebrew packages, if any are missing:
-
-.. code-block:: bash
-
-   brew install libpng openjpeg jbig2dec libtiff     # image libraries
-   brew install qpdf
-   brew install ghostscript
-   brew install python3
-   brew install libxml2 libffi leptonica
-   brew install unpaper   # optional
-   
-Install the required Tesseract OCR engine with the language packs you plan to use:
-   
-.. code-block:: bash
-
-   brew install tesseract                       # Option 1: for English, French, German, Spanish
-   
-.. code-block:: bash
-   
-   brew install tesseract --with-all-languages  # Option 2: for all language packs 
-   
-Update the homebrew pip and install Pillow:
-
-.. code-block:: bash
-
-   pip3 install --upgrade pip
-   pip3 install --upgrade pillow
-
-You can then install OCRmyPDF from PyPI:
-
-.. code-block:: bash
-
-   pip3 install ocrmypdf
-
-The command line program should now be available:
-
-.. code-block:: bash
-
-   ocrmypdf --help
-
-Installing on Ubuntu 14.04 LTS
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some other options, because of bugs in Python package installation.
-
-Add new "apt" repositories needed for backports of Ghostscript 9.16 and libav-11, which supports unpaper 6.1. This will replace Ghostscript on your system.
-
-.. code-block:: bash
-
-   sudo add-apt-repository ppa:vshn/ghostscript -y 
-   sudo add-apt-repository ppa:heyarje/libav-11 -y
-
-Update apt-get:
-
-.. code-block:: bash
-
-   sudo apt-get update
-   sudo apt-get upgrade
-   
-Install system dependencies:
-
-.. code-block:: bash
-
-   sudo apt-get install \
-      zlib1g-dev \
-      libjpeg-dev \
-      libffi-dev \
-      libavformat56 libavcodec56 libavutil54 \
-      ghostscript \
-      tesseract-ocr \
-      qpdf \
-      python3-pip \
-      python3-pil \
-      python3-pytest \
-      python3-reportlab
-
-If you wish install OCRmyPDF to the system Python, then install as follows (note this installs new packages
-into your system Python, which could interfere with other programs):
-
-.. code-block:: bash
-
-   sudo pip3 install ocrmypdf
-   
-If you wish to install OCRmyPDF to a virtual environment to isolate the system Python, you can
-follow these steps.  This includes a workaround `for a known, unresolved issue in Ubuntu 14.04's ensurepip
-package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`_:
-
-.. code-block:: bash
-
-   sudo apt-get install python3-venv
-   python3 -m venv venv-ocrmypdf --without-pip
-   source venv-ocrmypdf/bin/activate
-   wget -O - -o /dev/null https://bootstrap.pypa.io/get-pip.py | python
-   deactivate
-   python3 -m venv --system-site-packages venv-ocrmypdf
-   source venv-ocrmypdf/bin/activate
-   pip install ocrmypdf
-
-These installation instructions omit the optional dependency ``unpaper``, which is only available at version 0.4.2 in Ubuntu 14.04. The author could not find a backport of ``unpaper`` and is not motivated to figure how to set up a Ubuntu PPA to distribute it. You can create a .deb package to do the job of installing unpaper 6.1 (for x86 64-bit only):
-
-.. code-block:: bash
-
-   wget -q https://dl.dropboxusercontent.com/u/28971240/unpaper_6.1-1.deb -O unpaper_6.1-1.deb
-   sudo dpkg -i unpaper_6.1-1.deb
-
-
-Installing on Windows
-~~~~~~~~~~~~~~~~~~~~~
-
-Direct installation on Windows is not possible.  Install the _`Docker` container as described above.  Ensure that your command prompt can run the docker "hello world" container.
-
-Running on Windows
-~~~~~~~~~~~~~~~~~~
-
-The command line syntax to run ocrmypdf from a command prompt will resemble:
-
-.. code-block:: bat
-
-   docker run -v /c/Users/sampleuser:/home/docker ocrmypdf --skip-text test.pdf output.pdf
-
-where /c/Users/sampleuser is a Unix representation of the Windows path C:\\Users\\sampleuser, assuming a user named "sampleuser" is running ocrmypdf on a file in their home directory, and the files "test.pdf" and "output.pdf" are in the sampleuser folder. The Windows user must have read and write permissions.
-      
-Installing HEAD revision from sources
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-If you have ``git`` and ``python3.4`` or ``python3.5`` installed, you can install from source. When the ``pip`` installer runs,
-it will alert you if dependencies are missing.
-
-To install the HEAD revision from sources in the current Python 3 environment:
-
-.. code-block:: bash
-
-   pip3 install git+https://github.com/jbarlow83/OCRmyPDF.git
-
-Or, to install in `development mode <https://pythonhosted.org/setuptools/setuptools.html#development-mode>`_,  allowing customization of OCRmyPDF, use the ``-e`` flag:
-
-.. code-block:: bash
-
-   pip3 install -e git+https://github.com/jbarlow83/OCRmyPDF.git
-   
-On certain Linux distributions such as Ubuntu, you may need to use 
-run the install command as superuser:
-
-.. code-block:: bash
-
-   sudo pip3 install [-e] git+https://github.com/jbarlow83/OCRmyPDF.git
-   
-Note that this will alter your system's Python distribution. If you prefer 
-to not install as superuser, you can install the package in a Python virtual environment:
-
-.. code-block:: bash
-
-   git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
-   python3 -m venv
-   source venv/bin/activate
-   cd OCRmyPDF
-   pip3 install .
-
-However, ``ocrmypdf`` will only be accessible on the system PATH after
-you activate the virtual environment.
-
-To run the program:
-
-.. code-block:: bash
-   
-   ocrmypdf --help
-
-If not yet installed, the script will notify you about dependencies that
-need to be installed. The script requires specific versions of the
-dependencies. Older version than the ones mentioned in the release notes
-are likely not to be compatible to OCRmyPDF.
+For everyone else, `see our documentation <https://ocrmypdf.readthedocs.io/en/latest/installation.html>`_ for installation steps.

 Languages
 ---------
@@ -331,13 +76,13 @@ you can often find packages that provide language packs:
   apt-cache search tesseract-ocr

   # Debian/Ubuntu users
-   sudo apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified language back
+   apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified language back
   
 You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple
 languages can be requested.

-Support
-------
+Documentation and support
+-------------------------

 Once ocrmypdf is installed, the built-in help which explains the command syntax and options can be accessed via:

@@ -345,7 +90,7 @@ Once ocrmypdf is installed, the built-in help which explains the command syntax

   ocrmypdf --help

-The `Wiki <https://github.com/jbarlow83/OCRmyPDF/wiki>`_ page also contains some tips and suggests.
+Our `documentation is served on Read the Docs <https://ocrmypdf.readthedocs.io/en/latest/index.html>`_.

 If you detect an issue, please:

@@ -3,6 +3,47 @@ RELEASE NOTES

 OCRmyPDF uses `semantic versioning <http://semver.org/>`_.

+v4.3.2:
+=======
+
+-  Fixed a PDF/A creation issue with Ghostscript 9.20 
+
+
+v4.3.1:
+=======
+
+-  Fixed an issue where pages produced by the "hocr" renderer after a Tesseract timeout would be rotated incorrectly if the input page was rotated with a /Rotate marker
+-  Fixed a file handle leak in LeptonicaErrorTrap that would cause a "too many open files" error for files around hundred pages of pages long when ``--deskew`` or ``--remove-background`` or other Leptonica based image processing features were in use, depending on the system value of ``ulimit -n``
+-  Ability to specify multiple languages for multilingual documents is now advertised in documentation
+-  Reduced the file sizes of some test resources
+-  Cleaned up debug output
+-  Tesseract caching in test cases is now more cautious about false cache hits and reproducing exact output, not that any problems were observed
+
+
+v4.3:
+=====
+
+-  New feature ``--remove-background`` to detect and erase the background of color and grayscale images
+-  Better documentation
+-  Fixed an issue with PDFs that draw images when the raster stack depth is zero 
+-  ocrmypdf can now redirect its output to stdout for use in a shell pipeline
+
+   +  This does not improve performance since temporary files are still used for buffering
+   +  Some output validation is disabled in this mode
+
+v4.2.5:
+=======
+
+-  Fixed an issue (#100) with PDFs that omit the optional /BitsPerComponent parameter on images
+-  Removed non-free file milk.pdf
+
+
+v4.2.4:
+=======
+
+-  Fixed an error (#90) caused by PDFs that use stencil masks properly
+-  Fixed handling of PDFs that try to draw images or stencil masks without properly setting up the graphics state (such images are now ignored for the purposes of calculating DPI)
+
 v4.2.3:
 =======

@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<svg xmlns="http://www.w3.org/2000/svg" width="100%" height="100%" viewBox="0 0 915 585" xmlns:xlink="http://www.w3.org/1999/xlink">
+<defs>
+<linearGradient id="b" y2="445" gradientUnits="userSpaceOnUse" y1="179" gradientTransform="translate(0 -2.06)" x2="-29.7" x1="322">
+<stop stop-color="#333" offset="0"/>
+<stop stop-color="#fff" stop-opacity="0" offset="1"/>
+</linearGradient>
+<linearGradient id="a" y2="414" gradientUnits="userSpaceOnUse" y1="159" x2="490" x1="815">
+<stop stop-color="#33f" offset="0"/>
+<stop stop-color="#3f3fff" stop-opacity="0" offset="1"/>
+</linearGradient>
+</defs>
+<path fill="url(#b)" d="m403 247c-12 115-135 122-368 123-4.3-1.07-7.32-7.33-6-41 76-37 151-124 167-236 93 123 201 40.9 207 154z"/>
+<g stroke-width="3.45" fill="none">
+<path stroke="#000" d="m11.8 11.8h411v411l-411 0.01v-411z"/>
+<path stroke="#448" d="m489 11.7h415v411h-415v-411z"/>
+</g>
+<path d="m876 244c-12 115-133 120-366 121-6-14-10-40-3-43 76-37.3 136-106 152-218 38 48 209 101 217 140z" fill="url(#a)"/>
+<g id="RasterLarge" transform="matrix(1.36 0 0 1.28 -161 -636)">
+<path fill="#999" d="m287 730h-30v-60h20v20h10v10h10v10h20v10h40v-10h20v-20h10v-20h-10v-10h-10v-10h-40v-10h-30v-10h-20v-10h-10v-10h-10v-60h10v-10h10v-10h10v-10h90v10h30v60h-20v-10h-10v-20h-10v-10h-20v-10h-30v10h-20v10h-10v20h10v20h30v10h30v10h30v10h10v10h10v10h10v60h-10v10h-10v10h-10v10h-100v-10z"/>
+<path fill="#555" d="m297 730h-30v-10h-10v-50h20v20h10v20h10 10v10h60v-10h10v-10h10v-40h-20v-10h-20v-10h-40v-10h-20v-10h-20v-20h-10v-50h10v-10h10v-10h20v-10h70v10h30v10h10v50h-20v-20h-10v-20h-20v-10h-50v10h-10v10h-10v30h10v10h20v10h30v10h30v10h20v10h10v10h10v50h-10v20h-10v10h-20v10h-80z"/>
+<path d="m307 730h-30v-10h-20v-50h20v30h10v10h10v10h70v-10h20v-50h-10v-10h-20v-10h-40v-10h-30v-10h-10v-10h-10v-20h-10v-30h10v-20h20v-10h20v-10h50v10h30v10h20v50h-20v-30h-10v-10h-10v-10h-70v10h-10v40h10v10h10v10h30v10h40v10h20v10h10v20h10v40h-10v20h-20v10h-30v10h-50v-10z"/>
+</g>
+<g font-size="40" font-family="sans-serif" text-anchor="middle">
+<g font-size="100">
+<text y="518" x="210">Raster</text>
+<text y="518" x="695" fill="#338">Vector</text>
+</g>
+<text y="563" x="210">.jpeg  .gif  .png</text>
+<text y="563" x="696" fill="#338">.svg</text>
+</g>
+<path id="VectorLarge" fill="#005" d="m661 294v-62.5l23.4 0.184c0.678 20.8 7.32 36.3 19.9 46.4 12.7 9.93 32.1 14.9 57.9 14.9 24.1 0 42.5-4.29 55.1-12.9 12.7-8.71 19.1-21.3 19.1-37.9 0-13.2-3.86-23.4-11.6-30.5-7.59-7.11-23.7-14-48.4-20.8l-40.1-10.9c-29-7.97-49.5-17.9-61.4-29.8-11.8-11.9-17.7-28.2-17.7-48.9 0-23.3 9.15-41.4 27.4-54.3 18.3-12.9 44-19.3 77.1-19.3 14.1 0 29.6 1.41 46.4 4.23 16.8 2.7 34.7 6.68 53.7 12v58.5h-23c-2.3-19.4-9.49-33.4-21.6-41.9-11.9-8.71-30.2-13.1-54.7-13.1-21.4 0-37.7 3.99-49 12-11.1 7.85-16.7 19.3-16.7 34.4 0 13.1 4.2 23.4 12.6 30.9 8.4 7.48 26.2 14.9 53.5 22.3l37.6 10.1c27.5 7.48 47.1 17 58.8 28.7 11.8 11.5 17.7 27 17.7 46.5 0 26.6-9.42 46.7-28.3 60.2s-46.9 20.2-84.2 20.2c-16.7 0-33.7-1.53-51-4.6-18-3-35-7-53-13z"/>
+<use xlink:href="#VectorLarge" transform="matrix(.17 0 0 .17 392 313)"/>
+<use xlink:href="#RasterLarge" transform="matrix(.173 0 0 .173 -8.25 314)" height="100%" width="100%" y="0" x="0"/>
+</svg>
@@ -0,0 +1,350 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#
+# ocrmypdf documentation build configuration file, created by
+# sphinx-quickstart on Sun Sep  4 14:29:43 2016.
+#
+# This file is execfile()d with the current directory set to its
+# containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+# import os
+# import sys
+# sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration ------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    # 'sphinx.ext.mathjax',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The encoding of source files.
+#
+# source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = 'ocrmypdf'
+copyright = '2016, James R. Barlow'
+author = 'James R. Barlow'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+from ocrmypdf import VERSION as OCRMYPDF_VERSION
+
+_version_parts = OCRMYPDF_VERSION.split('.')
+
+version = '.'.join(_version_parts[0:2])
+# The full version, including alpha/beta/rc tags.
+release = OCRMYPDF_VERSION
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#
+# today = ''
+#
+# Else, today_fmt is used as the format for a strftime call.
+#
+today_fmt = '%Y-%m-%d'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This patterns also effect to html_static_path and html_extra_path
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# The reST default role (used for this markup: `text`) to use for all
+# documents.
+#
+# default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#
+# add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#
+# add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#
+# show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+# modindex_common_prefix = []
+
+# If true, keep warnings as "system message" paragraphs in the built documents.
+# keep_warnings = False
+
+# If true, `todo` and `todoList` produce output, else they produce nothing.
+todo_include_todos = False
+
+
+# -- Options for HTML output ----------------------------------------------
+
+import sphinx_rtd_theme
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+html_theme_options = {
+    'display_version': False
+}
+
+# Add any paths that contain custom themes here, relative to this directory.
+# html_theme_path = []
+
+# The name for this set of Sphinx documents.
+# "<project> v<release> documentation" by default.
+#
+# html_title = 'ocrmypdf v4.2'
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#
+# html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#
+# html_logo = None
+
+# The name of an image file (relative to this directory) to use as a favicon of
+# the docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#
+# html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Add any extra paths that contain custom files (such as robots.txt or
+# .htaccess) here, relative to this directory. These files are copied
+# directly to the root of the documentation.
+#
+# html_extra_path = []
+
+# If not None, a 'Last updated on:' timestamp is inserted at every page
+# bottom, using the given strftime format.
+# The empty string is equivalent to '%b %d, %Y'.
+#
+# html_last_updated_fmt = None
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#
+# html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#
+# html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#
+# html_additional_pages = {}
+
+# If false, no module index is generated.
+#
+# html_domain_indices = True
+
+# If false, no index is generated.
+#
+# html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#
+# html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#
+# html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#
+# html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#
+# html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#
+# html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+# html_file_suffix = None
+
+# Language to be used for generating the HTML full-text search index.
+# Sphinx supports the following languages:
+#   'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja'
+#   'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr', 'zh'
+#
+# html_search_language = 'en'
+
+# A dictionary with options for the search language support, empty by default.
+# 'ja' uses this config value.
+# 'zh' user can custom change `jieba` dictionary path.
+#
+# html_search_options = {'type': 'default'}
+
+# The name of a javascript file (relative to the configuration directory) that
+# implements a search results scorer. If empty, the default will be used.
+#
+# html_search_scorer = 'scorer.js'
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'ocrmypdfdoc'
+
+# -- Options for LaTeX output ---------------------------------------------
+
+latex_elements = {
+     # The paper size ('letterpaper' or 'a4paper').
+     #
+     # 'papersize': 'letterpaper',
+
+     # The font size ('10pt', '11pt' or '12pt').
+     #
+     # 'pointsize': '10pt',
+
+     # Additional stuff for the LaTeX preamble.
+     #
+     # 'preamble': '',
+
+     # Latex figure (float) alignment
+     #
+     # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'ocrmypdf.tex', 'ocrmypdf Documentation',
+     'James R. Barlow', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#
+# latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#
+# latex_use_parts = False
+
+# If true, show page references after internal links.
+#
+# latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#
+# latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#
+# latex_appendices = []
+
+# It false, will not define \strong, \code, 	itleref, \crossref ... but only
+# \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added
+# packages.
+#
+# latex_keep_old_macro_names = True
+
+# If false, no module index is generated.
+#
+# latex_domain_indices = True
+
+
+# -- Options for manual page output ---------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'ocrmypdf', 'ocrmypdf Documentation',
+     [author], 1)
+]
+
+# If true, show URL addresses after external links.
+#
+# man_show_urls = False
+
+
+# -- Options for Texinfo output -------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'ocrmypdf', 'ocrmypdf Documentation',
+     author, 'ocrmypdf', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#
+# texinfo_appendices = []
+
+# If false, no module index is generated.
+#
+# texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#
+# texinfo_show_urls = 'footnote'
+
+# If true, do not generate a @detailmenu in the "Top" node's menu.
+#
+# texinfo_no_detailmenu = False
@@ -0,0 +1,153 @@
+Cookbook
+========
+
+Basic examples
+--------------
+
+Help!
+"""""
+
+ocrmypdf has built-in help.
+
+.. code-block:: bash
+
+	ocrmypdf --help
+
+
+Add an OCR layer and convert to PDF/A
+"""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+	ocrmypdf input.pdf output.pdf
+
+Add an OCR layer and output a standard PDF
+""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+	ocrmypdf --output-type pdf input.pdf output.pdf
+
+Modify a file in place
+""""""""""""""""""""""
+
+The file will only be overwritten if OCRmyPDF is successful.
+
+.. code-block:: bash
+
+	ocrmypdf myfile.pdf myfile.pdf
+
+Correct page rotation
+"""""""""""""""""""""
+
+OCR will attempt to automatic correct the rotation of each page. This can help fix a scanning job that contains a mix of landscape and portrait pages.
+
+.. code-block:: bash
+
+	ocrmypdf --rotate-pages myfile.pdf myfile.pdf
+
+You can increase (decrease) the parameter ``--rotate-pages-threshold`` to make page rotation more (less) aggressive.
+
+
+OCR languages other than English
+""""""""""""""""""""""""""""""""
+
+By default OCRmyPDF assumes the document is English. 
+
+.. code-block:: bash
+
+	ocrmypdf -l fre LeParisien.pdf LeParisien.pdf
+	ocrmypdf -l eng+fre Bilingual-English-French.pdf Bilingual-English-French.pdf
+
+Language packs must be installed for all languages specified. See :ref:`Installing additional language packs <lang-packs>`.
+
+
+OCR images, not PDFs
+--------------------
+
+Use a program like `img2pdf <https://gitlab.mister-muffin.de/josch/img2pdf>`_ to convert your images to PDFs, and then pipe the results to run ocrmypdf:
+
+.. code-block:: bash
+
+	img2pdf my-images*.jpg | ocrmypdf - myfile.pdf
+
+If given a single image as input, OCRmyPDF will try converting it to a PDF on its own.  This feature may be removed at some point, because OCRmyPDF does not specialize in converting images to PDFs.
+
+You can also use Tesseract 3.04+ directly to convert single page images or multi-page TIFFs to PDF:
+
+.. code-block:: bash
+
+	tesseract my-image.jpg output-prefix pdf 
+
+Image processing
+----------------
+
+OCRmyPDF perform some image processing on each page of a PDF, if desired.  The same processing is applied to each page.  It is suggested that the user review files after image processing as these commands might remove desirable content, especially from poor quality scans.
+
+* ``--rotate-pages`` attempts to determine the correct orientation for each page and rotates the page if necessary.
+
+* ``--remove-background`` attempts to detect and remove a noisy background from grayscale or color images.  Monochrome images are ignored. This should not be used on documents that contain color photos as it may remove them.
+
+* ``--deskew`` will correct pages were scanned at a skewed angle by rotating them back into place.  Skew determination and correction is performed using `Postl's variance of line sums <http://www.leptonica.com/skew-measurement.html>`_ algorithm as implemented in `Leptonica <http://www.leptonica.com/index.html>`_.
+  
+* ``--clean`` uses `unpaper <https://www.flameeyes.eu/projects/unpaper>`_ to clean up pages before OCR, but does not alter the final output.  This makes it less likely that OCR will try to find text in background noise.
+
+* ``--clean-final`` uses unpaper to clean up pages before OCR and inserts the page into the final output.  You will want to review each page to ensure that unpaper did not remove something important.
+
+
+OCR and correct document skew (crooked scan)
+""""""""""""""""""""""""""""""""""""""""""""
+
+.. code-block:: bash
+
+	ocrmypdf --deskew input.pdf output.pdf
+
+
+Hot (watched) folders
+---------------------
+
+To set up a "hot folder" that will trigger an OCR operation for every file inserted, use a program like Python `watchdog <https://pypi.python.org/pypi/watchdog>`_ (supports all major OS).
+
+.. code-block:: bash
+
+	pip install watchdog
+
+watchdog installs the command line program ``watchmedo``, which can be told to run ``ocrmypdf`` on any .pdf added to the current directory (``.``) and place the result in the previously created ``out/`` folder.
+
+.. code-block:: bash
+
+	cd hot-folder
+	mkdir out
+	watchmedo shell-command \
+		--patterns="*.pdf" \
+		--ignore-directories \
+		--command='ocrmypdf "${watch_src_path}" "out/${watch_src_path}" ' \
+		.  # don't forget the final dot
+
+For more complex behavior you can write a Python script around to use the watchdog API.
+
+On file servers, you could configure watchmedo as a system service so it will run all the time.
+
+Caveats
+"""""""
+
+* ``watchmedo`` may not work properly on a networked file system, depending on the capabilities of the file system client and server.
+* This simple recipe does not filter for the type of file system event, so file copies, deletes and moves, and directory operations, will all be sent to ocrmypdf, producing errors in several cases. Disable your watched folder if you are doing anything other than copying files to it.
+* If the source and destination directory are the same, watchmedo may create an infinite loop.
+
+
+Batch jobs
+----------
+
+Consider using the excellent `GNU Parallel <https://www.gnu.org/software/parallel/>`_ to apply OCRmyPDF to multiple files at once.
+
+Both ``parallel`` and ``ocrmypdf`` will try to use all available processors. To maximize parallelism without overloading your system with processes, consider using ``parallel -j 2`` to limit parallel to running two jobs at once.
+
+This command will run all ocrmypdf all files named ``*.pdf`` in the current directory and write them to the previous created ``output/`` folder.
+
+.. code-block:: bash
+
+	parallel -j 2 ocrmypdf '{}' 'output/{}' ::: *.pdf
+
+If you have thousands of files to work with, contact the author.
+ 
@@ -0,0 +1,33 @@
+Common error messages
+=====================
+
+Page already has text
+---------------------
+
+.. code::
+
+	ERROR -    1: page already has text! – aborting (use --force-ocr to force OCR)
+
+You ran ocrmypdf on a file that already contains printable text or a hidden OCR text layer (it can't quite tell the difference). You probably don't want to do this, because the file is already searchable.
+
+As the error message suggests, your options are:
+
+- ``ocrmypdf --force-ocr`` to :ref:`rasterize <raster-vector>` all vector content and run OCR on the images. This is useful if a previous OCR program failed, or if the document contains a text watermark.
+
+- ``ocrmypdf --skip-text`` to skip OCR and other processing on any pages that contain text. Text pages will be copied into the output PDF without modification.
+
+
+Input file 'filename' is not a valid PDF
+----------------------------------------
+
+OCRmyPDF passes files through qpdf, a program that fixes errors in PDFs, before it tries to work on them. In most cases this happens because the PDF is corrupt and
+truncated (incomplete file copying) and not much can be done.
+
+You can try rewriting the file with Ghostscript or pdftk:
+
+- ``gs -o output.pdf -dSAFER -sDEVICE=pdfwrite input.pdf``
+
+- ``pdftk input.pdf cat output output.pdf``
+
+Sometimes Acrobat can repair PDFs with its `Preflight tool <https://helpx.adobe.com/acrobat/using/correcting-problem-areas-preflight-tool.html>`_.
+
@@ -0,0 +1,33 @@
+.. ocrmypdf documentation master file, created by
+   sphinx-quickstart on Sun Sep  4 14:29:43 2016.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+OCRmyPDF documentation
+======================
+
+OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to
+be searched.
+
+PDFs are the best format for scanned documents.  Unfortunately, PDFs can be difficult to work with. OCRmyPDF makes it easy to apply image processing and OCR to existing PDFs.
+
+Contents:
+
+.. toctree::
+   :maxdepth: 2
+
+   introduction
+   installation
+   languages
+   cookbook
+   errors
+
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`
+
@@ -0,0 +1,269 @@
+Installation
+============
+
+Installing on Debian and Ubuntu
+-------------------------------
+
+Users of Debian 9 ("stretch") or later or Ubuntu 16.10 or later may simply
+
+.. code-block:: bash
+
+	apt-get install ocrmypdf
+
+.. _Docker:
+
+Installing the Docker image
+---------------------------
+
+For many users, installing the Docker image will be easier than installing all of OCRmyPDF's dependencies. For Windows, it is the only option.
+
+If you have `Docker <https://docs.docker.com/>`_ installed on your system, you can install
+a Docker image of the latest release.
+
+Follow the Docker installation instructions for your platform.  If you can run this command
+successfully, your system is ready to download and execute the image:
+
+.. code-block:: bash
+
+   docker run hello-world
+   
+OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker engine host, and then follow these optional steps to enable multiple CPUs:
+
+.. code-block:: bash
+
+   # Optional step for Mac OS X users
+   docker-machine stop "yourVM"
+   VBoxManage modifyvm "yourVM" --cpus 2  # or whatever number of core is desired
+   docker-machine start "yourVM"
+   eval $(docker-machine env "yourVM")
+
+Assuming you have a Docker engine running somewhere, you can run these commands to download
+the image:
+
+.. code-block:: bash
+
+   docker pull jbarlow83/ocrmypdf
+
+Then tag it to give a more convenient name, just ocrmypdf:
+
+.. code-block:: bash
+
+   docker tag jbarlow83/ocrmypdf ocrmypdf
+
+.. _docker-polyglot:
+
+This image contains language packs for English, French, Spanish and German. The alternative "polyglot" image provides `all available language packs <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#languages>`_:
+
+.. code-block:: bash
+
+   # Alternative step: If you need all language packs
+   docker pull jbarlow83/ocrmypdf-polyglot
+   docker tag jbarlow83/ocrmypdf-polyglot ocrmypdf
+
+You can then run ocrmypdf using the command:
+
+.. code-block:: bash
+
+   docker run --rm ocrmypdf --help
+  
+To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`_, and both the input and output file must be inside the writable volume.  This example command uses the current working directory as the writable volume:
+
+.. code-block:: bash
+
+   docker run --rm -v "$(pwd):/home/docker" <other docker arguments>   ocrmypdf <your arguments to ocrmypdf>
+
+In this worked example, the current working directory contains an input file called ``test.pdf`` and the output will go to ``output.pdf``: 
+
+.. code-block:: bash
+
+   docker run --rm -v "$(pwd):/home/docker"   ocrmypdf --skip-text test.pdf output.pdf
+
+Note that ``ocrmypdf`` has its own separate ``-v VERBOSITYLEVEL`` argument to control debug verbosity. All Docker arguments should before the ``ocrmypdf`` image name and all arguments to ``ocrmypdf`` should be listed after.
+
+
+Installing on macOS (formerly Mac OS X)
+---------------------------------------
+
+These instructions probably work on all macOS supported by Homebrew. OCRmyPDF is known to work on Yosemite and El Capitan, and regularly tested on El Capitan.
+
+If it's not already present, `install Homebrew <http://brew.sh/>`_.
+
+Update Homebrew:
+
+.. code-block:: bash
+
+   brew update
+   
+Install or upgrade the required Homebrew packages, if any are missing:
+
+.. code-block:: bash
+
+   brew install libpng openjpeg jbig2dec libtiff     # image libraries
+   brew install qpdf
+   brew install ghostscript
+   brew install python3
+   brew install libxml2 libffi leptonica
+   brew install unpaper   # optional
+   
+Install the required Tesseract OCR engine with the language packs you plan to use:
+   
+.. code-block:: bash
+
+   brew install tesseract                       # Option 1: for English, French, German, Spanish
+
+.. _macos-all-languages:
+
+.. code-block:: bash
+   
+   brew install tesseract --with-all-languages  # Option 2: for all language packs 
+   
+Update the homebrew pip and install Pillow:
+
+.. code-block:: bash
+
+   pip3 install --upgrade pip
+   pip3 install --upgrade pillow
+
+You can then install OCRmyPDF from PyPI:
+
+.. code-block:: bash
+
+   pip3 install ocrmypdf
+
+The command line program should now be available:
+
+.. code-block:: bash
+
+   ocrmypdf --help
+
+Installing on Ubuntu 14.04 LTS
+------------------------------
+
+Installing on Ubuntu 14.04 LTS (trusty) is more difficult than some other options, because of bugs in Python package installation.
+
+Add new "apt" repositories needed for backports of Ghostscript 9.16 and libav-11, which supports unpaper 6.1. This will replace Ghostscript on your system.
+
+.. code-block:: bash
+
+   sudo add-apt-repository ppa:vshn/ghostscript -y 
+   sudo add-apt-repository ppa:heyarje/libav-11 -y
+
+Update apt-get:
+
+.. code-block:: bash
+
+   sudo apt-get update
+   sudo apt-get upgrade
+   
+Install system dependencies:
+
+.. code-block:: bash
+
+   sudo apt-get install \
+      zlib1g-dev \
+      libjpeg-dev \
+      libffi-dev \
+      libavformat56 libavcodec56 libavutil54 \
+      ghostscript \
+      tesseract-ocr \
+      qpdf \
+      python3-pip \
+      python3-pil \
+      python3-pytest \
+      python3-reportlab
+
+If you wish install OCRmyPDF to the system Python, then install as follows (note this installs new packages
+into your system Python, which could interfere with other programs):
+
+.. code-block:: bash
+
+   sudo pip3 install ocrmypdf
+   
+If you wish to install OCRmyPDF to a virtual environment to isolate the system Python, you can
+follow these steps.  This includes a workaround `for a known, unresolved issue in Ubuntu 14.04's ensurepip
+package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`_:
+
+.. code-block:: bash
+
+   sudo apt-get install python3-venv
+   python3 -m venv venv-ocrmypdf --without-pip
+   source venv-ocrmypdf/bin/activate
+   wget -O - -o /dev/null https://bootstrap.pypa.io/get-pip.py | python
+   deactivate
+   python3 -m venv --system-site-packages venv-ocrmypdf
+   source venv-ocrmypdf/bin/activate
+   pip install ocrmypdf
+
+These installation instructions omit the optional dependency ``unpaper``, which is only available at version 0.4.2 in Ubuntu 14.04. The author could not find a backport of ``unpaper``, and created a .deb package to do the job of installing unpaper 6.1 (for x86 64-bit only):
+
+.. code-block:: bash
+
+   wget -q https://dl.dropboxusercontent.com/u/28971240/unpaper_6.1-1.deb -O unpaper_6.1-1.deb
+   sudo dpkg -i unpaper_6.1-1.deb
+
+
+Installing on Windows
+---------------------
+
+Direct installation on Windows is not possible.  Install the _`Docker` container as described above.  Ensure that your command prompt can run the docker "hello world" container.
+
+Running on Windows
+~~~~~~~~~~~~~~~~~~
+
+The command line syntax to run ocrmypdf from a command prompt will resemble:
+
+.. code-block:: bat
+
+   docker run -v /c/Users/sampleuser:/home/docker ocrmypdf --skip-text test.pdf output.pdf
+
+where /c/Users/sampleuser is a Unix representation of the Windows path C:\\Users\\sampleuser, assuming a user named "sampleuser" is running ocrmypdf on a file in their home directory, and the files "test.pdf" and "output.pdf" are in the sampleuser folder. The Windows user must have read and write permissions.
+      
+Installing HEAD revision from sources
+-------------------------------------
+
+If you have ``git`` and ``python3.4`` or ``python3.5`` installed, you can install from source. When the ``pip`` installer runs,
+it will alert you if dependencies are missing.
+
+To install the HEAD revision from sources in the current Python 3 environment:
+
+.. code-block:: bash
+
+   pip3 install git+https://github.com/jbarlow83/OCRmyPDF.git
+
+Or, to install in `development mode <https://pythonhosted.org/setuptools/setuptools.html#development-mode>`_,  allowing customization of OCRmyPDF, use the ``-e`` flag:
+
+.. code-block:: bash
+
+   pip3 install -e git+https://github.com/jbarlow83/OCRmyPDF.git
+   
+On certain Linux distributions such as Ubuntu, you may need to use 
+run the install command as superuser:
+
+.. code-block:: bash
+
+   sudo pip3 install [-e] git+https://github.com/jbarlow83/OCRmyPDF.git
+   
+Note that this will alter your system's Python distribution. If you prefer 
+to not install as superuser, you can install the package in a Python virtual environment:
+
+.. code-block:: bash
+
+   git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
+   python3 -m venv
+   source venv/bin/activate
+   cd OCRmyPDF
+   pip3 install .
+
+However, ``ocrmypdf`` will only be accessible on the system PATH after
+you activate the virtual environment.
+
+To run the program:
+
+.. code-block:: bash
+   
+   ocrmypdf --help
+
+If not yet installed, the script will notify you about dependencies that
+need to be installed. The script requires specific versions of the
+dependencies. Older version than the ones mentioned in the release notes
+are likely not to be compatible to OCRmyPDF.
@@ -0,0 +1,76 @@
+Introduction
+============
+
+OCRmyPDF is a Python 3 package that adds OCR layers to PDFs.
+
+
+About OCR
+---------
+
+`Optical character recognition <https://en.wikipedia.org/wiki/Optical_character_recognition>`_ is technology that converts images of typed or handwritten text, such as in a scanned document, to computer text that can be searched and copied.
+
+OCRmyPDF uses `Tesseract <https://github.com/tesseract-ocr/tesseract>`_, the best available open source OCR engine, to perform OCR.
+
+.. _raster-vector:
+
+About PDFs
+----------
+
+PDFs are page description files that attempts to preserve a layout exactly. They can contain `vector graphic files <http://vector-conversions.com/vectorizing/raster_vs_vector.html>`_ that can contain raster objects such as scanned images. Because PDFs can contain multiple pages (unlike many image formats) and can contain fonts and text, it is a good formats for exchanging scanned documents.
+
+.. image:: bitmap_vs_svg.svg
+
+A PDF page might contain multiple images, even if it only appears to have one image.  Some scanners or scanning software will segment pages into monochromatic text and color regions for example, to improve the compression ratio and appearance of the page.
+
+Rasterizing a PDF is the process of generating an image suitable for display or analyzing with an OCR engine.  OCR engines like Tesseract work with images, not vector objects.
+
+
+About PDF/A
+-----------
+
+`PDF/A <https://en.wikipedia.org/wiki/PDF/A>`_ is an ISO-standardized subset of the full PDF specification that is designed for archiving (the 'A' stands for Archive).  PDF/A differs from PDF primarily by omitting features that would make it difficult to read the file in the future, such as embedded Javascript, video, audio and references to external fonts.  All fonts and resources needed to interpret the PDF must be contained within it. Because PDF/A disables Javascript and other types of embedded content, it is probably more secure.
+
+There are various conformance levels and versions, such as "PDF/A-2b".
+
+Generally speaking, the best format for scanned documents is PDF/A. Some governments and jurisdictions, US Courts in particular, `mandate the use of PDF/A <https://pdfblog.com/2012/02/13/what-is-pdfa/>`_ for scanned documents.
+
+Since most people who scan documents are interested in reading them indefinitely into the future, OCRmyPDF generates PDF/A-2b by default.
+
+PDF/A has a few drawbacks.  Some PDF viewers include an alert that the file is a PDF/A, which may confuse some users.  It also tends to produce larger files than PDF, because it embeds certain resources even if they are commonly available. PDF/A files can be digitally signed, but may not be encrypted, to ensure they can be read in the future.  Fortunately, converting from PDF/A to a regular PDF is trivial, and any PDF viewer can view PDF/A.
+
+
+What OCRmyPDF does
+------------------
+
+OCRmyPDF analyzes each page of a PDF to determine the colorspace and resolution (DPI) needed to capture all of the information on that page without losing content.  It uses `Ghostscript <http://ghostscript.com/>`_ to rasterize the page, and then performs on OCR on the rasterized image.  It is not enough to simply extract the images from each page and run OCR on them individually.  Of course one could use Ghostscript or another PDF rasterizer and then pass the image to Tesseract.  OCRmyPDF automates this process and produces a minimally changed output file that contains the same information, colorspace and resolution.
+
+The Tesseract OCR engine can output 'hOCR' files, which are XML files that contain a description of the text it found on the page.  OCRmyPDF will render a new PDF that contains only the hidden text layer, and merge this with the original page.
+
+Alternately, OCRmyPDF can use the Tesseract OCR engine to directly output PDFs for each page, then merge them.
+
+By default, OCRmyPDF will convert the file to a PDF/A.  This behavior can be disabled with the ``--output-type pdf`` argument.
+
+Depending on the settings selected, OCRmyPDF may "graft" the OCR layer into the existing PDF, or reconstruct a visually equivalent new PDF.
+
+
+Limitations
+-----------
+
+OCRmyPDF is limited by the Tesseract OCR engine.  As such it experiences these limitations, as do any other programs that rely on Tesseract:
+
+* The OCR is not as accurate as commercial solutions such as Abbyy.
+* It is not capable of recognizing handwriting.
+* It may find gibberish and report this as OCR output.
+* If a document contains languages outside of those given in the ``-l LANG`` arguments, results may be poor.
+* It is not always good at analyzing the natural reading order of documents. For example, it may fail to recognize that a document contains two columns and join text across the columns.
+* Poor quality scans may produce poor quality OCR. Garbage in, garbage out.
+  
+OCRmyPDF is also limited by the PDF specification:
+
+* PDF encodes the position of text glyphs but does not encode document structure.  There is no markup that divides a document in sections, paragraphs, sentences, or even words (since blank spaces are not represented). As such all elements of document structure including the spaces between words must be derived heuristically.  Some PDF viewers do a better job of this than others.
+
+Ghostscript also imposes some limitations:
+
+* PDFs containing JBIG2-encoded content will be converted to CCITT Group4 encoding, which has lower compression ratios, if Ghostscript PDF/A is enabled.
+  
+OCRmyPDF is currently not designed to be used as a Python API; it is designed to be run as a command line tool. ``import ocrmypf`` currently attempts to process the command line on ``sys.argv`` at import time so it has side effects that will interfere with its use as a package. The API it presents should not be considered stable.
@@ -0,0 +1,48 @@
+.. _lang-packs:
+
+Installing additional language packs
+====================================
+
+OCRmyPDF uses Tesseract for OCR, and relies on its language packs for languages other than English. 
+
+Tesseract supports `most languages <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#languages>`_.
+
+For Linux users, you can often find packages that provide language packs:
+
+Debian and Ubuntu users
+-----------------------
+
+.. code-block:: bash
+
+   # Display a list of all Tesseract language packs
+   apt-cache search tesseract-ocr
+
+   # Debian/Ubuntu users
+   apt-get install tesseract-ocr-chi-sim  # Example: Install Chinese Simplified language back
+   
+You can then pass the ``-l LANG`` argument to OCRmyPDF to give a hint as to what languages it should search for. Multiple
+languages can be requested using either ``-l eng+fre`` (English and French) or ``-l eng -l fre``.
+
+Mac OS X (macOS) users
+----------------------
+
+You can install additional language packs by :ref:`installing Tesseract using Homebrew with all language packs <macos-all-languages>`.
+
+Docker users
+------------
+
+Users of the Docker image may use the alternative :ref:`"polyglot" container <docker-polyglot>` which includes all languages.
+
+Known limitations
+-----------------
+
+As of v4.2, users of ocrmypdf working languages outside the Latin alphabet should use the following syntax:
+
+.. code-block:: bash
+
+	ocrmypdf -l eng+gre --output-type pdf --pdf-renderer tesseract
+
+The reasons for this are:
+
+* The latest version of Ghostscript (9.19 as of this writing) has unfixed bugs in Unicode handling that generate invalid character maps, so Ghostscript cannot be used for PDF/A conversion
+* The default "hocr" PDF renderer does not handle Asian fonts properly
@@ -1,6 +1,9 @@
 from enum import IntEnum
 import os
 from collections.abc import Iterable
+import pkg_resources
+
+VERSION = pkg_resources.get_distribution('ocrmypdf').version


 class ExitCode(IntEnum):
@@ -28,3 +31,4 @@ def page_number(input_file):

 def is_iterable_notstr(thing):
    return isinstance(thing, Iterable) and not isinstance(thing, str)
+
@@ -21,7 +21,7 @@ from PIL import Image
 from functools import partial

 from ruffus import transform, suffix, merge, active_if, regex, jobs_limit, \
-    formatter, follows, split, collate, check_if_uptodate, graphviz
+    formatter, follows, split, collate, check_if_uptodate, graphviz, posttask
 import ruffus.ruffus_exceptions as ruffus_exceptions
 import ruffus.cmdline as cmdline
 import ruffus.proxy_logger as proxy_logger
@@ -32,13 +32,10 @@ from .pdfa import generate_pdfa_def, file_claims_pdfa
 from . import ghostscript
 from . import tesseract
 from . import qpdf
-from . import ExitCode, page_number, is_iterable_notstr
+from . import leptonica
+from . import ExitCode, page_number, is_iterable_notstr, VERSION
 from collections.abc import Sequence

-import pkg_resources
-
-VERSION = pkg_resources.get_distribution('ocrmypdf').version
-
 warnings.simplefilter('ignore', pypdf.utils.PdfReadWarning)


@@ -155,6 +152,9 @@ ocrmypdf so it is already installed.

    img2pdf --pagesize A4 page*.png | ocrmypdf - myfile.pdf

+Online documentation is located at:
+    https://ocrmypdf.readthedocs.io/en/latest/introduction.html
+
 """)

 parser.add_argument(
@@ -163,11 +163,13 @@ parser.add_argument(
         "standard input)")
 parser.add_argument(
    'output_file',
-    help="output searchable PDF file")
+    help="output searchable PDF file (or '-' to write to standard output)")
 parser.add_argument(
    '-l', '--language', action='append',
-    help="languages of the file to be OCRed (see tesseract --list-langs for "
-         "all language packs installed in your system)")
+    help="Language(s) of the file to be OCRed (see tesseract --list-langs for "
+         "all language packs installed in your system). To specify multiple "
+         "languages, join them with '+' or issue this argument once for each "
+         "language.")
 parser.add_argument(
    '-j', '--jobs', metavar='N', type=int,
    help="Use up to N CPU cores simultaneously (default: use all)")
@@ -204,6 +206,10 @@ preprocessing = parser.add_argument_group(
 preprocessing.add_argument(
    '-r', '--rotate-pages', action='store_true',
    help="automatically rotate pages based on detected text orientation")
+preprocessing.add_argument(
+    '--remove-background', action='store_true',
+    help="attempt to remove background from gray or color pages, setting it "
+         "to white ")
 preprocessing.add_argument(
    '-d', '--deskew', action='store_true',
    help="deskew each page before performing OCR")
@@ -300,6 +306,8 @@ if not set(options.language).issubset(tesseract.languages()):
 # ----------
 # Arguments

+options.verbose_abbreviated_path = 1
+
 if options.pdf_renderer == 'auto':
    options.pdf_renderer = 'hocr'

@@ -350,7 +358,8 @@ if set(options.language) & {'chi_sim', 'chi_tra'} \

 lossless_reconstruction = False
 if options.pdf_renderer == 'hocr':
-    if not options.deskew and not options.clean_final and not options.force_ocr:
+    if not options.deskew and not options.clean_final and \
+            not options.force_ocr and not options.remove_background:
        lossless_reconstruction = True


@@ -455,11 +464,16 @@ _pdfinfo_lock = manager.Lock()
 work_folder = mkdtemp(prefix="com.github.ocrmypdf.")


+def done_task(caller):
+    "Useful as debug hook"
+    pass
+
+
@atexit.register
 def cleanup_working_files(*args):
    if options.keep_temporary_files:
-        print("Temporary working files saved at:")
-        print(work_folder)
+        print("Temporary working files saved at:\n{0}".format(work_folder),
+              file=sys.stderr)
    else:
        with suppress(FileNotFoundError):
            shutil.rmtree(work_folder)
@@ -528,6 +542,7 @@ def triage_image_file(input_file, output_file, log):
        sys.exit(ExitCode.input_file)


+@posttask(partial(done_task, 'triage'))
@transform(
    input=os.path.join(work_folder, 'origin'),
    filter=formatter('(?i)'),
@@ -550,6 +565,7 @@ def triage(
    triage_image_file(input_file, output_file, log)


+@posttask(partial(done_task, 'repair_pdf'))
@transform(
    input=triage,
    filter=suffix('.pdf'),
@@ -643,6 +659,7 @@ def is_ocr_required(pageinfo, log):
    return ocr_required


+@posttask(partial(done_task, 'split_pages'))
@split(
    repair_pdf,
    os.path.join(work_folder, '*.page.pdf'),
@@ -685,6 +702,7 @@ def split_pages(
                os.path.basename(filename)[0:6] + alt_suffix))


+@posttask(partial(done_task, 'rasterize_preview'))
@active_if(options.rotate_pages)
@transform(
    input=split_pages,
@@ -707,6 +725,7 @@ def rasterize_preview(
        log=log)


+@posttask(partial(done_task, 'orient_page'))
@collate(
    input=[split_pages, rasterize_preview],
    filter=regex(r".*/(\d{6})(\.ocr|\.skip)(?:\.page\.pdf|\.preview\.jpg)"),
@@ -781,6 +800,7 @@ def orient_page(
            pdfinfo[pageno] = pageinfo


+@posttask(partial(done_task, 'rasterize_with_ghostscript'))
@transform(
    input=orient_page,
    filter=suffix('.ocr.oriented.pdf'),
@@ -817,9 +837,37 @@ def rasterize_with_ghostscript(
        log=log)


+@posttask(partial(done_task, 'preprocess_remove_background'))
@transform(
    input=rasterize_with_ghostscript,
    filter=suffix(".page.png"),
+    output=".pp-background.png",
+    extras=[_log, _pdfinfo, _pdfinfo_lock])
+def preprocess_remove_background(
+        input_file,
+        output_file,
+        log,
+        pdfinfo,
+        pdfinfo_lock):
+
+    if not options.remove_background:
+        re_symlink(input_file, output_file, log)
+        return
+
+    pageinfo = get_pageinfo(input_file, pdfinfo, pdfinfo_lock)
+
+    if any(image['bpc'] > 1 for image in pageinfo['images']):
+        leptonica.remove_background(input_file, output_file)
+    else:
+        log.info("{0:4d}: background removal skipped on mono page".format(
+            pageinfo['pageno']))
+        re_symlink(input_file, output_file, log)
+
+
+@posttask(partial(done_task, 'preprocess_deskew'))
+@transform(
+    input=preprocess_remove_background,
+    filter=suffix(".pp-background.png"),
    output=".pp-deskew.png",
    extras=[_log, _pdfinfo, _pdfinfo_lock])
 def preprocess_deskew(
@@ -836,10 +884,10 @@ def preprocess_deskew(
    pageinfo = get_pageinfo(input_file, pdfinfo, pdfinfo_lock)
    dpi = get_page_square_dpi(pageinfo)

-    from . import leptonica
    leptonica.deskew(input_file, output_file, dpi)


+@posttask(partial(done_task, 'preprocess_clean'))
@transform(
    input=preprocess_deskew,
    filter=suffix(".pp-deskew.png"),
@@ -862,6 +910,7 @@ def preprocess_clean(
    unpaper.clean(input_file, output_file, dpi, log)


+@posttask(partial(done_task, 'ocr_tesseract_hocr'))
@active_if(options.pdf_renderer == 'hocr')
@transform(
    input=preprocess_clean,
@@ -889,9 +938,11 @@ def ocr_tesseract_hocr(
        )


+@posttask(partial(done_task, 'select_image_for_pdf'))
@collate(
-    input=[rasterize_with_ghostscript, preprocess_deskew, preprocess_clean],
-    filter=regex(r".*/(\d{6})(?:\.page|\.pp-deskew|\.pp-clean)\.png"),
+    input=[rasterize_with_ghostscript, preprocess_remove_background,
+           preprocess_deskew, preprocess_clean],
+    filter=regex(r".*/(\d{6})(?:\.page|\.pp-.*)\.png"),
    output=os.path.join(work_folder, r'\1.image'),
    extras=[_log, _pdfinfo, _pdfinfo_lock])
@graphviz(shape='diamond')
@@ -905,6 +956,8 @@ def select_image_for_pdf(
        image_suffix = '.pp-clean.png'
    elif options.deskew:
        image_suffix = '.pp-deskew.png'
+    elif options.remove_background:
+        image_suffix = '.pp-background.png'
    else:
        image_suffix = '.page.png'
    image = next(ii for ii in infiles if ii.endswith(image_suffix))
@@ -929,6 +982,7 @@ def select_image_for_pdf(
        re_symlink(image, output_file)


+@posttask(partial(done_task, 'select_image_layer'))
@active_if(options.pdf_renderer == 'hocr')
@collate(
    input=[select_image_for_pdf, orient_page],
@@ -959,11 +1013,14 @@ def select_image_layer(
        with open(image, 'rb') as imfile, \
                open(output_file, 'wb') as pdf:
            rawdata = imfile.read()
+            log.debug('{:4d}: convert'.format(page_number(page_pdf)))
            img2pdf.convert(
                rawdata, with_pdfrw=False,
                layout_fun=layout_fun, outputstream=pdf)
+            log.debug('{:4d}: convert done'.format(page_number(page_pdf)))


+@posttask(partial(done_task, 'render_hocr_page'))
@active_if(options.pdf_renderer == 'hocr')
@transform(
    input=ocr_tesseract_hocr,
@@ -986,6 +1043,7 @@ def render_hocr_page(
                         showBoundingboxes=False, invisibleText=True)


+@posttask(partial(done_task, 'render_hocr_debug_page'))
@active_if(options.pdf_renderer == 'hocr')
@active_if(options.debug_rendering)
@collate(
@@ -1015,6 +1073,7 @@ class PdfMergeFailedError(Exception):
    pass


+@posttask(partial(done_task, 'add_text_layer'))
@active_if(options.pdf_renderer == 'hocr')
@collate(
    input=[render_hocr_page, select_image_layer],
@@ -1093,6 +1152,7 @@ def add_text_layer(
        pdf_output.write(out)


+@posttask(partial(done_task, 'tesseract_ocr_and_render_pdf'))
@active_if(options.pdf_renderer == 'tesseract')
@collate(
    input=[select_image_for_pdf, orient_page],
@@ -1158,6 +1218,7 @@ def get_pdfmark(base_pdf):
    return pdfmark


+@posttask(partial(done_task, 'generate_postscript_stub'))
@active_if(options.output_type == 'pdfa')
@transform(
    input=repair_pdf,
@@ -1174,6 +1235,7 @@ def generate_postscript_stub(
    generate_pdfa_def(output_file, pdfmark)


+@posttask(partial(done_task, 'skip_page'))
@transform(
    input=orient_page,
    filter=suffix('.skip.oriented.pdf'),
@@ -1191,6 +1253,7 @@ def skip_page(
    re_symlink(input_file, output_file, log)


+@posttask(partial(done_task, 'merge_pages_ghostscript'))
@active_if(options.output_type == 'pdfa')
@merge(
    input=[add_text_layer, render_hocr_debug_page, skip_page,
@@ -1222,6 +1285,7 @@ def merge_pages_ghostscript(
    ghostscript.generate_pdfa(pdf_pages, output_file, options.jobs or 1)


+@posttask(partial(done_task, 'merge_pages_qpdf'))
@active_if(options.output_type == 'pdf')
@merge(
    input=[add_text_layer, render_hocr_debug_page, skip_page,
@@ -1268,6 +1332,7 @@ def merge_pages_qpdf(
    qpdf.merge(pdf_pages, output_file)


+@posttask(partial(done_task, 'copy_final'))
@merge(
    input=[merge_pages_ghostscript, merge_pages_qpdf],
    output=options.output_file,
@@ -1279,7 +1344,13 @@ def copy_final(
        pdfinfo,
        pdfinfo_lock):
    input_file = next((ii for ii in input_files if ii.endswith('.pdf')))
-    shutil.copy(input_file, output_file)
+
+    if output_file == '-':
+        from shutil import copyfileobj
+        with open(input_file, 'rb') as input_stream:
+            copyfileobj(input_stream, sys.stdout.buffer)
+    else:
+        shutil.copy(input_file, output_file)


 def available_cpu_count():
@@ -1405,6 +1476,14 @@ def run_pipeline():
                _log.error("File not found - " + options.input_file)
                return ExitCode.input_file

+        if options.output_file == '-':
+            if sys.stdout.isatty():
+                _log.error(textwrap.dedent("""\
+                    Output was set to stdout '-' but it looks like stdout
+                    is connected to a terminal.  Please redirect stdout to a
+                    file."""))
+                return ExitCode.bad_args
+
        cmdline.run(options)
    except ruffus_exceptions.RethrownJobError as e:
        if options.verbose:
@@ -1436,23 +1515,28 @@ def run_pipeline():
        _log.error(e)
        return ExitCode.other_error

-    if options.output_type == 'pdfa':
-        pdfa_info = file_claims_pdfa(options.output_file)
-        if pdfa_info['pass']:
-            msg = 'Output file is a {} (as expected)'
-            _log.info(msg.format(pdfa_info['conformance']))
-        else:
-            msg = 'Output file was generated but is not PDF/A (seems to be {})'
-            _log.warning(msg.format(pdfa_info['conformance']))
+    if options.output_file != '-':
+        if options.output_type == 'pdfa':
+            pdfa_info = file_claims_pdfa(options.output_file)
+            if pdfa_info['pass']:
+                msg = 'Output file is a {} (as expected)'
+                _log.info(msg.format(pdfa_info['conformance']))
+            else:
+                msg = 'Output file is okay but is not PDF/A (seems to be {})'
+                _log.warning(msg.format(pdfa_info['conformance']))

+                return ExitCode.invalid_output_pdf
+        if not qpdf.check(options.output_file, _log):
+            _log.warning('Output file: The generated PDF is INVALID')
            return ExitCode.invalid_output_pdf
-
-    if not qpdf.check(options.output_file, _log):
-        _log.warning('Output file: The generated PDF is INVALID')
-        return ExitCode.invalid_output_pdf
+    else:
+        _log.info("Output sent to stdout")

    with _pdfinfo_lock:
-        _log.debug(_pdfinfo)
+        if options.verbose:
+            from pprint import pformat
+            referent = _pdfinfo._getvalue()  # get the real list out of proxy
+            _log.debug(pformat(referent))
        direction = {0: 'n', 90: 'e',
                     180: 's', 270: 'w'}
        orientations = []
@@ -21,7 +21,7 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
            '-dFirstPage=%i' % pageno,
            '-dLastPage=%i' % pageno,
            '-o', tmp.name,
-            '-r{0}x{1}'.format(str(xres), str(yres)),
+            '-r{0}x{1}'.format(str(round(xres)), str(round(yres))),
            input_file
        ]

@@ -29,7 +29,10 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log,
                  universal_newlines=True)
        stdout, stderr = p.communicate()
        if stdout:
-            log.debug(stdout)
+            if 'error' in stdout:
+                log.error(stdout)  # Ghostscript puts errors in stdout
+            else:
+                log.debug(stdout)
        if stderr:
            log.error(stderr)

@@ -53,7 +56,7 @@ def generate_pdfa(pdf_pages, output_file, threads=1):
            "-sProcessColorModel=DeviceRGB",
            "-dJPEGQ=95",
            "-dPDFA=2",
-            "-sPDFACompatibilityPolicy=2",
+            "-sPDFACompatibilityPolicy=1",
            "-sOutputFile=" + gs_pdf.name,
        ]
        args_gs.extend(pdf_pages)
@@ -1,14 +1,10 @@
-#!/usr/bin/env python2
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #
-# © 2013-15: jbarlow83 from Github (https://github.com/jbarlow83)
+# © 2013-16: jbarlow83 from Github (https://github.com/jbarlow83)
 #
-#
-# Use Leptonica to detect find and remove page skew.  Leptonica uses the method
-# of differential square sums, which its author claim is faster and more robust
-# than the Hough transform used by ImageMagick.
+# Python FFI wrapper for Leptonica library

-from __future__ import print_function, absolute_import, division
 import argparse
 import sys
 import os
@@ -44,22 +40,34 @@ class LeptonicaErrorTrap(object):

    """
    def __enter__(self):
+        from io import UnsupportedOperation
        self.tmpfile = TemporaryFile()

        # Save the old stderr, and redirect stderr to temporary file
-        self.old_stderr_fileno = os.dup(sys.stderr.fileno())
-        os.dup2(self.tmpfile.fileno(), sys.stderr.fileno())
+        sys.stderr.flush()
+        try:
+            self.copy_of_stderr = os.dup(sys.stderr.fileno())
+            os.dup2(self.tmpfile.fileno(), sys.stderr.fileno(),
+                    inheritable=False)
+        except UnsupportedOperation:
+            self.copy_of_stderr = None
        return

    def __exit__(self, exc_type, exc_value, traceback):
        # Restore old stderr
-        os.dup2(self.old_stderr_fileno, sys.stderr.fileno())
+        sys.stderr.flush()
+        if self.copy_of_stderr is not None:
+            os.dup2(self.copy_of_stderr, sys.stderr.fileno())
+            os.close(self.copy_of_stderr)

        # Get data from tmpfile (in with block to ensure it is closed)
        with self.tmpfile as tmpfile:
            tmpfile.seek(0)  # Cursor will be at end, so move back to beginning
            leptonica_output = tmpfile.read().decode(errors='replace')

+        assert self.tmpfile.closed
+        assert not sys.stderr.closed
+
        # If there are Python errors, let them bubble up
        if exc_type:
            logger.warning(leptonica_output)
@@ -104,6 +112,12 @@ class Pix:

    Leptonica's reference counting is not threadsafe. This class can be used
    in a threadsafe manner if a Python threading.Lock protects the data.
+
+    This class treats Pix objects as immutable.  All methods return new
+    modified objects.  This allows convenient chaining:
+
+    >>>   Pix.read('filename.jpg').scale((0.5, 0.5)).deskew().show()
+
    """

    def __init__(self, pix):
@@ -117,6 +131,22 @@ class Pix:
        else:
            return "<leptonica.Pix image NULL>"

+    def _repr_png_(self):
+        """iPython display hook
+
+        returns png version of image
+        """
+
+        data = ffi.new('l_uint8 **')
+        size = ffi.new('size_t *')
+
+        err = lept.pixWriteMemPng(data, size, self._pix, 0)
+        if err != 0:
+            raise LeptonicaIOError("pixWriteMemPng")
+
+        char_data = ffi.cast('char *', data[0])
+        return ffi.buffer(char_data, size[0])[:]
+
    def __getstate__(self):
        data = ffi.new('l_uint32 **')
        size = ffi.new('size_t *')
@@ -142,6 +172,9 @@ class Pix:
            cdata_uint32, len(state['data']))
        Pix.__init__(self, pix)

+    def __eq__(self, other):
+        return self.__getstate__() == other.__getstate__()
+
    @property
    def width(self):
        return self._pix.w
@@ -201,18 +234,29 @@ class Pix:
        "Returns a PIL.Image version of this Pix"
        from PIL import Image

-        with LeptonicaErrorTrap():
-            pix_swapped = Pix(lept.pixEndianByteSwapNew(self._pix))
+        # Leptonica manages data in words, so it implicitly does an endian
+        # swap.  Tell Pillow about this when it reads the data.
+        pix = self
+        if sys.byteorder == 'little':
+            if self.mode == 'RGB':
+                raw_mode = 'XBGR'
+            elif self.mode == 'RGBA':
+                raw_mode = 'ABGR'
+            elif self.mode == '1':
+                raw_mode = '1;I'
+                pix = Pix(lept.pixEndianByteSwapNew(pix._pix))
+            else:
+                raw_mode = self.mode
+                pix = Pix(lept.pixEndianByteSwapNew(pix._pix))
+        else:
+            raw_mode = self.mode  # no endian swap needed

-        size = (pix_swapped._pix.wpl * 4, pix_swapped._pix.h)
-        buf = ffi.buffer(pix_swapped._pix.data, size[0] * size[1])
+        size = (pix._pix.w, pix._pix.h)
+        bytecount = pix._pix.wpl * 4 * pix._pix.h
+        buf = ffi.buffer(pix._pix.data, bytecount)
+        stride = pix._pix.wpl * 4

-        im_raw = Image.frombytes(self.mode, size, buf, 'raw')
-
-        # Leptonica stores images in 32-bit words
-        # Need to crop the any trailing amount
-        box = (0, 0, self.width, self.height)
-        im = im_raw.crop(box)
+        im = Image.frombytes(self.mode, size, buf, 'raw', raw_mode, stride)

        return im

@@ -231,10 +275,10 @@ class Pix:
        with LeptonicaErrorTrap():
            return Pix(lept.pixDeskew(self._pix, reduction_factor))

-    def scale(self, scalex, scaley):
+    def scale(self, scale_xy):
        "Returns the pix object rescaled according to the proportions given."
        with LeptonicaErrorTrap():
-            return Pix(lept.pixScale(self._pix, scalex, scaley))
+            return Pix(lept.pixScale(self._pix, scale_xy[0], scale_xy[1]))

    def rotate180(self):
        with LeptonicaErrorTrap():
@@ -352,6 +396,33 @@ class Pix:
                black,
                white))

+    def gamma_trc(self, gamma=1.0, minval=0, maxval=255):
+        with LeptonicaErrorTrap():
+            return Pix(lept.pixGammaTRC(
+                ffi.NULL,
+                self._pix,
+                gamma,
+                minval,
+                maxval
+                ))
+
+    def background_norm(
+            self, mask=None, grayscale=None, tile_size=(10, 15), fg_threshold=60,
+            min_count=40, bg_val=200, smooth_kernel=(2, 1)):
+        with LeptonicaErrorTrap():
+            return Pix(lept.pixBackgroundNorm(
+                self._pix,
+                mask or ffi.NULL,
+                grayscale or ffi.NULL,
+                tile_size[0],
+                tile_size[1],
+                fg_threshold,
+                min_count,
+                bg_val,
+                smooth_kernel[0],
+                smooth_kernel[1]
+                ))
+
    @staticmethod
    @lru_cache(maxsize=1)
    def make_pixel_sum_tab8():
@@ -462,6 +533,22 @@ def deskew(infile, outfile, dpi):
        raise LeptonicaIOError("Failed to open destination file: %s" % outfile)


+def remove_background(infile, outfile, tile_size=(40, 60), gamma=1.0,
+                      black_threshold=70, white_threshold=190):
+    try:
+        pix = Pix.read(infile)
+    except LeptonicaIOError:
+        raise LeptonicaIOError("Failed to open file: %s" % infile)
+
+    pix = pix.background_norm(tile_size=tile_size).gamma_trc(
+            gamma, black_threshold, white_threshold)
+
+    try:
+        pix.write_implied_format(outfile)
+    except LeptonicaIOError:
+        raise LeptonicaIOError("Failed to open destination file: %s" % outfile)
+
+
 if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description="Python wrapper to access Leptonica")
@@ -61,6 +61,12 @@ PIX * pixRead ( const char *filename );
 PIX * pixScale ( PIX *pixs, l_float32 scalex, l_float32 scaley );
 l_int32 pixFindSkew ( PIX *pixs, l_float32 *pangle, l_float32 *pconf );
 l_int32 pixWriteImpliedFormat ( const char *filename, PIX *pix, l_int32 quality, l_int32 progressive );
+l_int32
+pixWriteMemPng(l_uint8  **pdata,
+               size_t    *psize,
+               PIX       *pix,
+               l_float32  gamma);
+
 void pixDestroy ( PIX **ppix );

 PIX *
@@ -131,6 +137,50 @@ pixClipRectangle(PIX   *pixs,
                 BOX   *box,
                 BOX  **pboxc);

+PIX *
+pixBackgroundNorm(PIX     *pixs,
+                  PIX     *pixim,
+                  PIX     *pixg,
+                  l_int32  sx,
+                  l_int32  sy,
+                  l_int32  thresh,
+                  l_int32  mincount,
+                  l_int32  bgval,
+                  l_int32  smoothx,
+                  l_int32  smoothy);
+
+PIX *
+pixGammaTRC(PIX       *pixd,
+            PIX       *pixs,
+            l_float32  gamma,
+            l_int32    minval,
+            l_int32    maxval);
+
+
+l_int32
+pixNumSignificantGrayColors(PIX       *pixs,
+                            l_int32    darkthresh,
+                            l_int32    lightthresh,
+                            l_float32  minfract,
+                            l_int32    factor,
+                            l_int32   *pncolors);
+
+l_int32
+pixColorFraction(PIX        *pixs,
+                 l_int32     darkthresh,
+                 l_int32     lightthresh,
+                 l_int32     diffthresh,
+                 l_int32     factor,
+                 l_float32  *ppixfract,
+                 l_float32  *pcolorfract);
+
+PIX *
+pixColorMagnitude(PIX     *pixs,
+                  l_int32  rwhite,
+                  l_int32  gwhite,
+                  l_int32  bwhite,
+                  l_int32  type);
+
 void
 boxDestroy(BOX  **pbox);

@@ -9,6 +9,14 @@ import sys
 import PyPDF2 as pypdf
 from collections import namedtuple

+try:
+    from math import isclose
+except ImportError:
+    def isclose(a, b, rel_tol=1e-9):
+        "Python 3.4 does not have math.isclose()"
+        diff = abs(b - a)
+        return diff <= abs(rel_tol * b) or diff <= abs(rel_tol * a)
+
 matrix_mult = pypdf.pdf.utils.matrixMultiply

 FRIENDLY_COLORSPACE = {
@@ -52,6 +60,9 @@ FRIENDLY_COMP = {
 }


+UNIT_SQUARE = (1.0, 0.0, 0.0, 1.0, 0.0, 0.0)
+
+
 def _matrix_from_shorthand(shorthand):
    """Convert from PDF matrix shorthand to full matrix

@@ -73,13 +84,46 @@ def _shorthand_from_matrix(matrix):
    return tuple(map(float, (a, b, c, d, e, f)))


+def _is_unit_square(shorthand):
+    values = map(float, shorthand)
+    pairwise = zip(values, UNIT_SQUARE)
+    return all([isclose(a, b, rel_tol=1e-3) for a, b in pairwise])
+
+RasterSettings = namedtuple('RasterSettings',
+    ['name', 'shorthand', 'stack_depth'])
+
+InlineSettings = namedtuple('InlineSettings',
+    ['settings', 'shorthand', 'stack_depth'])
+
 ContentsInfo = namedtuple('ContentsInfo', ['raster_settings', 'inline_images'])


 def _interpret_contents(contentstream):
+    """Interpret the PDF content stream
+
+    The stack represents the state of the PDF graphics stack.  We are only
+    interested in the current transformation matrix (CTM) so we only track
+    this object; a full implementation would need to track many other items.
+
+    The CTM is initialized to the mapping from user space to device space.
+    PDF units are 1/72".  In a PDF viewer or printer this matrix is initialized
+    to the transformation to device space.  For example if set to
+    (1/72, 0, 0, 1/72, 0, 0) then all units would be calculated in inches.
+
+    Images are always considered to be (0, 0) -> (1, 1).  Before drawing an
+    image there should be a 'cm' that sets up an image coordinate system
+    where drawing from (0, 0) -> (1, 1) will draw on the desired area of the
+    page.
+
+    PDF units suit our needs so we initialize ctm to the identity matrix.
+
+    PyPDF2 replaces inline images with a fake "INLINE IMAGE" operator.
+
+    """
+
    operations = contentstream.operations
    stack = []
-    ctm = _matrix_from_shorthand((1, 0, 0, 1, 0, 0))
+    ctm = _matrix_from_shorthand(UNIT_SQUARE)
    image_raster_settings = []
    inline_images = []

@@ -96,12 +140,16 @@ def _interpret_contents(contentstream):
                _matrix_from_shorthand(operands), ctm)
        elif command == b'Do':
            image_name = operands[0]
-            image_raster_settings.append(
-                (image_name, _shorthand_from_matrix(ctm)))
+            raster = RasterSettings(
+                name=image_name, shorthand=_shorthand_from_matrix(ctm),
+                stack_depth=len(stack))
+            image_raster_settings.append(raster)
        elif command == b'INLINE IMAGE':
            settings = operands['settings']
-            inline_images.append(
-                (settings, _shorthand_from_matrix(ctm)))
+            inline = InlineSettings(
+                settings=settings, shorthand=_shorthand_from_matrix(ctm),
+                stack_depth=len(stack))
+            inline_images.append(inline)

    return ContentsInfo(
        raster_settings=image_raster_settings,
@@ -175,24 +223,27 @@ def _get_dpi(ctm_shorthand, image_size):
 def _find_page_inline_images(page, pageinfo, contentsinfo):
    "Find inline images on the page"

-    for n, im in enumerate(contentsinfo.inline_images):
-        settings, shorthand = im
+    for n, inline in enumerate(contentsinfo.inline_images):
        image = {}
        image['name'] = str('inline-%02d' % n)
-        image['width'] = settings['/W']
-        image['height'] = settings['/H']
-        image['bpc'] = settings['/BPC']
-        image['color'] = FRIENDLY_COLORSPACE.get(settings['/CS'], '-')
+        image['width'] = inline.settings['/W']
+        image['height'] = inline.settings['/H']
+        if '/BPC' in inline.settings:
+            image['bpc'] = inline.settings['/BPC']
+        else:
+            image['bpc'] = 8
+        image['color'] = FRIENDLY_COLORSPACE.get(inline.settings['/CS'], '-')
        image['comp'] = FRIENDLY_COMP.get(image['color'], '?')
-        if '/F' in settings:
-            filter_ = settings['/F']
+        if '/F' in inline.settings:
+            filter_ = inline.settings['/F']
            if isinstance(filter_, pypdf.generic.ArrayObject):
                filter_ = filter_[0]
            image['enc'] = FRIENDLY_ENCODING.get(filter_, 'image')
        else:
            image['enc'] = 'image'

-        dpi_w, dpi_h = _get_dpi(shorthand, (image['width'], image['height']))
+        dpi_w, dpi_h = _get_dpi(
+            inline.shorthand, (image['width'], image['height']))
        image['dpi_w'], image['dpi_h'] = Decimal(dpi_w), Decimal(dpi_h)
        yield image

@@ -213,7 +264,10 @@ def _find_page_regular_images(page, pageinfo, contentsinfo):
        image['name'] = str(xobj)
        image['width'] = pdfimage['/Width']
        image['height'] = pdfimage['/Height']
-        image['bpc'] = pdfimage['/BitsPerComponent']
+        if '/BitsPerComponent' in pdfimage:
+            image['bpc'] = pdfimage['/BitsPerComponent']
+        else:
+            image['bpc'] = 8

        # Fixme: this is incorrectly treats explicit masks as stencil masks,
        # but good enough for now. Explicit masks have /ImageMask true but are
@@ -253,35 +307,30 @@ def _find_page_regular_images(page, pageinfo, contentsinfo):

        for raster in contentsinfo.raster_settings:
            # Loop in case the same image is display multiple times on a page
-            if raster[0] != image['name']:
+            if raster.name != image['name']:
                continue
-            shorthand = raster[1]

-            if image['type'] == 'stencil':
-                # Stencil masks are implicitly scaled over the whole page
-                # Images that are used in explicit masks are not drawn directly
-                # but drawn by the image they mask over, so they will never
-                # be called for in raster settings
-                if shorthand != (1, 0, 0, 1, 0, 0):
-                    raise NotImplementedError(
-                        "Don't know how to handle "
-                        "stencil masks when graphics stack depth > 0.")
-                page_w = float(pageinfo['width_inches']) * 72.0
-                page_h = float(pageinfo['height_inches']) * 72.0
-                shorthand = (page_w, 0.0, 0.0,
-                             page_h, 0.0, 0.0)
+            if raster.stack_depth == 0 and _is_unit_square(raster.shorthand):
+                # At least one PDF in the wild (and test suite) draws an image
+                # when the graphics stack depth is 0, meaning that the image
+                # gets drawn into a square of 1x1 PDF units (or 1/72",
+                # or 0.35 mm).  The equivalent DPI will be >100,000.  Exclude
+                # these from our DPI calculation for the page.
+                continue

            dpi_w, dpi_h = _get_dpi(
-                shorthand, (image['width'], image['height']))
+                raster.shorthand, (image['width'], image['height']))

            # When image is used multiple times take the highest DPI it is
            # rendered at
            image['dpi_w'] = max(dpi_w, image.get('dpi_w', 0))
            image['dpi_h'] = max(dpi_h, image.get('dpi_h', 0))

-        image['dpi_w'] = Decimal(image['dpi_w'])
-        image['dpi_h'] = Decimal(image['dpi_h'])
-        image['dpi'] = (image['dpi_w'] * image['dpi_h']) ** Decimal(0.5)
+        DPI_PREC = Decimal('1.000')
+        image['dpi_w'] = Decimal(image['dpi_w']).quantize(DPI_PREC)
+        image['dpi_h'] = Decimal(image['dpi_h']).quantize(DPI_PREC)
+        dpi = Decimal(image['dpi_w'] * image['dpi_h']).sqrt()
+        image['dpi'] = dpi.quantize(DPI_PREC)
        yield image


@@ -327,6 +376,11 @@ def _pdf_get_pageinfo(infile, pageno: int):
    pageinfo['width_inches'] = width_pt / Decimal(72.0)
    pageinfo['height_inches'] = height_pt / Decimal(72.0)

+    try:
+        pageinfo['rotate'] = int(page['/Rotate'])
+    except KeyError:
+        pageinfo['rotate'] = 0
+
    try:
        contentstream = pypdf.pdf.ContentStream(page.getContents(), pdf)
    except AttributeError as e:
@@ -70,13 +70,14 @@ def encode_text_string(s: str) -> str:

    From PDF 32000-1:2008 a string object may be included in hexademical form
    if it is enclosed in angle brackets.  For general Unicode the string should
-    be UTF-16 (big endian) with byte order marks.  A non-hexademical
-    representation is doable but this is preferable since it allows the output
-    Postscript file to be completely ASCII and no escaping of Postscript
-    characters is necessary.
+    be UTF-16 (big endian) with byte order marks.  Many strings including all
+    ASCII strings fall could be encoded as PdfDocEncoding literals provided
+    that certain Postscript sequences are escaped.  But it's far simpler to
+    encode everything as UTF-16.
    '''

-    # Sometimes lazy C programmer leave their NULs at the end of strings
+    # Sometimes lazy C programmers leave their NULs at the end of strings they
+    # insert into PDFs
    # tests/resources/aspect.pdf is one example (created by ImageMagick)
    s = s.replace('\x00', '')

@@ -139,21 +140,20 @@ def file_claims_pdfa(filename):
                'conformance': 'No XMP metadata'}

    pdfa_dict = {attr.localName: attr.value for attr in pdfa_nodes}
-    pdfa_dict['pass'] = False
-    pdfa_dict['output'] = 'pdf'
-    if pdfa_dict:
-        part_conformance = pdfa_dict['part'] + pdfa_dict['conformance']
-        valid_part_conforms = {'1A', '1B', '2A', '2B', '2U', '3A', '3B', '3U'}
+    if not pdfa_dict:
+        return {'pass': False, 'output': 'pdf',
+                'conformance': 'No XMP metadata'}

-        conformance = 'PDF/A-{}'.format(
-            part_conformance)
+    part_conformance = pdfa_dict['part'] + pdfa_dict['conformance']
+    valid_part_conforms = {'1A', '1B', '2A', '2B', '2U', '3A', '3B', '3U'}

-        if part_conformance in valid_part_conforms:
-            pdfa_dict['pass'] = True
-            pdfa_dict['output'] = 'pdfa'
-        pdfa_dict['conformance'] = conformance
-    else:
-        pdfa_dict['conformance'] = 'PDF'
+    conformance = 'PDF/A-{}'.format(
+        part_conformance)
+
+    if part_conformance in valid_part_conforms:
+        pdfa_dict['pass'] = True
+        pdfa_dict['output'] = 'pdfa'
+    pdfa_dict['conformance'] = conformance

    return pdfa_dict

@@ -147,11 +147,16 @@ def page_timedout(log, input_file):
    log.warning(prefix + " took too long to OCR - skipping")


-def _generate_null_hocr(output_hocr, pageinfo):
+def _generate_null_hocr(output_hocr, image):
+    """Produce a .hocr file that reports no text detected on a page that is
+    the same size as the input image."""
+    from PIL import Image
+
+    im = Image.open(image)
+    w, h = im.size
+
    with open(output_hocr, 'w', encoding="utf-8") as f:
-        f.write(HOCR_TEMPLATE.format(
-            pageinfo['width_pixels'],
-            pageinfo['height_pixels']))
+        f.write(HOCR_TEMPLATE.format(w, h))


 def generate_hocr(input_file, output_hocr, language: list, tessconfig: list,
@@ -181,11 +186,11 @@ def generate_hocr(input_file, output_hocr, language: list, tessconfig: list,
        # Temporary workaround to hocrTransform not being able to function if
        # it does not have a valid hOCR file.
        page_timedout(log, input_file)
-        _generate_null_hocr(output_hocr, pageinfo_getter())
+        _generate_null_hocr(output_hocr, input_file)
    except CalledProcessError as e:
        tesseract_log_output(log, e.output, input_file)
        if 'Image too large' in e.output:
-            _generate_null_hocr(output_hocr, pageinfo_getter())
+            _generate_null_hocr(output_hocr, input_file)
            return

        raise e from e
@@ -1,2 +1,2 @@
 [pytest]
-norecursedirs = lib .pc
+norecursedirs = lib .pc .git
@@ -148,9 +148,10 @@ def check_external_program(


 command = next((arg for arg in sys.argv[1:] if not arg.startswith('-')), '')
+forced = ('--force' in sys.argv)


-if command.startswith('install') or \
+if not forced and command.startswith('install') or \
        command in ['check', 'test', 'nosetests', 'easy_install']:
    check_external_program(
        program='tesseract',
@@ -174,6 +175,8 @@ if command.startswith('install') or \
        package='qpdf',
        version_check_args=['--version']
    )
+else:
+    print('Skipping external program tests because of --force')


 if 'upload' in sys.argv[1:]:
@@ -1,101 +0,0 @@
-#!/usr/bin/env python3
-# -*- coding: utf-8 -*-
-# © 2016 James R. Barlow: github.com/jbarlow83
-# Release sanity checking
-
-import argparse
-from subprocess import run, PIPE, DEVNULL, STDOUT, CalledProcessError
-from git import Repo, Remote
-import logging
-import re
-import sys
-import os
-
-
-def test_repo(repo):
-    assert not repo.is_dirty(), "Repository is dirty"
-    if repo.untracked_files:
-        logging.warning('Some files are untracked:')
-        logging.warning('\n' + '\n'.join(repo.untracked_files))
-    assert repo.active_branch.name == 'master', 'Not on branch master'
-
-
-def travis(args):
-    repo = Repo('.')
-    test_repo(repo)
-
-    git_describe = repo.git.describe()
-
-    try:
-        env = os.environ.copy()
-        env['SETUPTOOLS_SCM_PRETEND_VERSION'] = git_describe
-        proc = run(['check-manifest'], check=True, universal_newlines=True, stdout=PIPE, stderr=STDOUT, env=env)
-        logging.info(proc.stdout)
-    except CalledProcessError as e:
-        logging.error('MANIFEST.in error')
-        logging.error(e.stdout)
-        sys.exit(1)
-
-    run(['python3', 'setup.py', 'build'], check=True)
-
-    origin = Remote(repo, 'jbarlow')
-    result = origin.push(refspec='master:master')[0]
-    if result.flags & (1024|4|8|32|16):
-        logging.error(result.summary)
-    else:
-        logging.info(result.summary)
-
-    logging.info("Pushed to Travis CI")
-    logging.info("If this passes, git tag and release")
-
-
-def release(args):
-    repo = Repo('.')
-    test_repo(repo)
-
-    git_describe = repo.git.describe()
-
-    assert not git_describe.startswith('v') and not '-' in git_describe and not '+ng' in git_describe, \
-        "Not tagged properly for release: " + git_describe
-
-    plain_version = git_describe[1:]  # without 'v' prefix
-
-    with open('RELEASE_NOTES.rst') as f:
-        notes = f.read()
-        assert plain_version in notes, "Version not mentioned in release notes"
-
-    proc = run(['python3', 'setup.py', 'sdist', 'bdist_wheel'], universal_newlines=True, check=True, stdout=PIPE, stderr=STDOUT)
-    logging.info(proc.stdout)
-
-
-    origin = Remote(repo, 'jbarlow')
-    result = origin.push(refspec='master:master', tags=True)[0]
-    if result.flags & (1024|4|8|32|16):
-        logging.error(result.summary)
-    else:
-        logging.info(result.summary)
-
-    run(['twine', 'upload', '-r', 'pypitest',
-        'dist/ocrmypdf-{}.tar.gz'.format(plain_version),
-        'dist/ocrmypdf-{}-py34-none-any.whl'.format(plain_version)], check=True, universal_newlines=True, stdout=PIPE)
-
-
-parser = argparse.ArgumentParser(description="ocrmypdf release tasks")
-subparsers = parser.add_subparsers()
-
-push_travis = subparsers.add_parser(
-    'push-travis', description="Push master to travis for testing")
-push_travis.set_defaults(func=travis)
-
-release_parser = subparsers.add_parser(
-    'release', description="Release to PyPI etc")
-release_parser.set_defaults(func=release)
-
-
-def main():
-    args = parser.parse_args()
-    args.func(args)
-
-
-if __name__ == '__main__':
-    main()
@@ -9,20 +9,28 @@ Files derived from free sources
 These test resources come from free sources, under either public domain or Creative Commons licenses.
 In some cases they were converted from one image format to another without other changes.

-+---------------------+--------------------------------------------------------------------------------+
-| File                | Source                                                                         |
-+=====================+================================================================================+
-| c02-22.pdf          | `Project Gutenberg`_, Adventures of Huckleberry Finn, page 22                  |
-+---------------------+--------------------------------------------------------------------------------+
-| congress.jpg        | `US Congressional Records`_ (Public Domain)                                    |
-+---------------------+--------------------------------------------------------------------------------+
-| graph.pdf           | `Wikimedia: Pandas text analysis.png`_ (Public Domain)                         |
-+---------------------+--------------------------------------------------------------------------------+
-| lichtenstein.pdf    | `Wikimedia: JPEG2000 Lichtenstein`_ (Creative Commons BY-SA 3.0)               |
-+---------------------+--------------------------------------------------------------------------------+
-| LinnSequencer.jpg,  | `Wikimedia: LinnSequencer`_ (Creative Commons BY-SA 3.0)                       |
-| linn.pdf, linn.txt  |                                                                                |
-+---------------------+--------------------------------------------------------------------------------+
+.. list-table:: 
+    :widths: 20 50 30
+    :header-rows: 1
+
+    *   - File
+        - Source
+        - License
+    *   - c02-22.pdf
+        - `Project Gutenberg`_, Adventures of Huckleberry Finn, page 22
+        - Public Domain
+    *   - congress.jpg
+        - `US Congressional Records`_
+        - Public Domain
+    *   - graph.pdf
+        - `Wikimedia: Pandas text analysis.png`_
+        - Public Domain
+    *   - lichtenstein.pdf
+        - `Wikimedia: JPEG2000 Lichtenstein`_
+        - Creative Commons BY-SA 3.0
+    *   - LinnSequencer.jpg, linn.pdf, linn.txt
+        - `Wikimedia: LinnSequencer`_
+        - Creative Commons BY-SA 3.0


 Files generated for this project
@@ -31,44 +39,58 @@ Files generated for this project
 The following test resources were crafted specifically for this project, and can be used
 under the terms of the license in LICENSE.rst.

-+---------------------+-----------------------+---------------------------------------------------------+
-| File                | Contributor           | Purpose                                                 |
-+---------------------+-----------------------+---------------------------------------------------------+
-| aspect.pdf          | @jbarlow83            | test image with 200 x 100 DPI resolution                |
-+---------------------+-----------------------+---------------------------------------------------------+
-| blank.pdf           | @jbarlow83            | blank PDF                                               |
-+---------------------+-----------------------+---------------------------------------------------------+
-| cmyk.pdf            | @jbarlow83            | a CMYK image created in Photoshop                       |
-+---------------------+-----------------------+---------------------------------------------------------+
-| enormous.pdf        | @jbarlow83            | very large PDF page                                     |
-+---------------------+-----------------------+---------------------------------------------------------+
-| francais.pdf        | @jbarlow83            | a page containing French accents (diacritics)           |
-+---------------------+-----------------------+---------------------------------------------------------+
-| hugemono.pdf        | @jbarlow83            | large monochrome 35000x35000 image in JBIG2 encoding    |
-+---------------------+-----------------------+---------------------------------------------------------+
-| invalid.pdf         | @jbarlow83            | a PDF file header followed by EOF marker                |
-+---------------------+-----------------------+---------------------------------------------------------+
-| masks.pdf           | @supergrobi           | file containing stencil masks; printout of a German     |
-|                     |                       | Wikipedia article (Creative Commons BY-SA)              |
-+---------------------+-----------------------+---------------------------------------------------------+
-| milk.pdf            | @lowesjam             | linearized PDF containing some indirect objects         |
-+---------------------+-----------------------+---------------------------------------------------------+
-| missing_docinfo.pdf | @jbarlow83            | PDF file with no /DocumentInfo section                  |
-+---------------------+-----------------------+---------------------------------------------------------+
+.. list-table:: 
+    :widths: 20 20 60
+    :header-rows: 1
+
+    *   - File
+        - Contributor
+        - Purpose
+    *   - aspect.pdf
+        - @jbarlow83
+        - test image with 200 x 100 DPI resolution
+    *   - blank.pdf
+        - @jbarlow83
+        - blank PDF
+    *   - cmyk.pdf
+        - @jbarlow83
+        - a CMYK image created in Photoshop
+    *   - enormous.pdf
+        - @jbarlow83
+        - very large PDF page
+    *   - epson.pdf
+        - @lowesjam
+        - a linearized PDF containing some unusual indirect objects, created by an Epson printer; printout of a Wikipedia article (CC BY-SA)
+    *   - francais.pdf
+        - @jbarlow83
+        - a page containing French accents (diacritics)  
+    *   - hugemono.pdf
+        - @jbarlow83
+        - large monochrome 35000x35000 image in JBIG2 encoding 
+    *   - invalid.pdf
+        - @jbarlow83
+        - a PDF file header followed by EOF marker
+    *   - masks.pdf
+        - @supergrobi
+        - file containing explicit masks and a stencil mask drawn without a proper transformation matrix; printout of a German Wikipedia article (CC BY-SA)
+    *   - missing_docinfo.pdf
+        - @jbarlow83
+        - PDF file with no /DocumentInfo section 

 Assemblies
 ==========

 These test resources are assemblies from other previously mentioned files, released under the same license terms as their input files.

- cardinal.pdf (four cardinal directions, rotated copies of LinnSequencer.jpg)
+- cardinal.pdf (four cardinal directions, baked-in rotated copies of LinnSequencer.jpg)
 - ccitt.pdf (LinnSequencer.jpg, converted to CCITT encoding)
 - encrypted_algo4.pdf (congress.jpg, encrypted with algorithm 4 - not supported by PyPDF2)
 - graph_ocred.pdf (from graph.pdf)
 - jbig2.pdf (congress.jpg, converted to JBIG2 encoding)
 - multipage.pdf (from several other files)
 - palette.pdf (congress.jpg, converted to a 256-color palette)
- skew.pdf (from c02-22.pdf)
+- rotated_skew.pdf (a /Rotate'd and skewed document from LinnSequencer.jpg)
+- skew.pdf (from LinnSequencer.jpg, skew simulated by adjusting the transformation matrix)
 - skew-encrypted.pdf (skew.pdf with encryption - access supported by PyPDF2)


@@ -48,11 +48,17 @@ def main():

    m = hashlib.sha1()

-    version = subprocess.check_output(
+    tess_version = subprocess.check_output(
        ['tesseract', '--version'],
        stderr=subprocess.STDOUT)

-    m.update(version)
+    m.update(tess_version)
+
+    # Insert this source file into the hash function, to ensure that any
+    # changes to this file invalidate previous hashes
+    with open(__file__, 'rb') as f:
+        m.update(f.read())
+
    m.update(operation.encode())

    try:
@@ -93,10 +99,10 @@ def main():
            shutil.copy(cache_name, output_file)

        # Replicate output
-        with open(cache_name + '.stdout', 'r') as f:
-            print(f.read(), end='')
-        with open(cache_name + '.stderr', 'r') as f:
-            print(f.read(), end='', file=sys.stderr)
+        with open(cache_name + '.stdout', 'rb') as f:
+            sys.stdout.buffer.write(f.read())
+        with open(cache_name + '.stderr', 'rb') as f:
+            sys.stderr.buffer.write(f.read())
        sys.exit(0)

    # Cache miss
@@ -105,23 +111,22 @@ def main():
    # Call tesseract
    p = subprocess.Popen(
            ['tesseract'] + sys.argv[1:],
-            stdout=subprocess.PIPE, stderr=subprocess.PIPE,
-            universal_newlines=True)
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    stdout, stderr = p.communicate()

    if p.returncode != 0:
        # Do not cache errors or crashes
        print("Tesseract error", file=sys.stderr)
-        print(stdout, end='')
-        print(stderr, end='', file=sys.stderr)
+        sys.stdout.buffer.write(stdout)
+        sys.stderr.buffer.write(stderr)
        return p.returncode

-    with open(cache_name + '.stdout', 'w') as f:
+    with open(cache_name + '.stdout', 'wb') as f:
        f.write(stdout)
-    with open(cache_name + '.stderr', 'w') as f:
+    with open(cache_name + '.stderr', 'wb') as f:
        f.write(stderr)
-    print(stdout, end='')
-    print(stderr, end='', file=sys.stderr)
+    sys.stdout.buffer.write(stdout)
+    sys.stderr.buffer.write(stderr)

    # Insert file into cache
    if output_file != 'stdout':
@@ -68,6 +68,9 @@ def check_ocrmypdf(input_basename, output_basename, *args, env=None):
    assert p.returncode == 0
    assert os.path.exists(output_file), "Output file not created"
    assert os.stat(output_file).st_size > 100, "PDF too small or empty"
+    assert out == "", \
+        "The following was written to stdout and should not have been: \n" + \
+        "<stdout>\n" + out + "\n</stdout>"
    return output_file


@@ -124,8 +127,8 @@ def spoof_tesseract_big_image_error():
    return spoof('tesseract', 'tesseract_big_image_error.py')


-def test_quick(spoof_tesseract_noop):
-    check_ocrmypdf('c02-22.pdf', 'test_quick.pdf', env=spoof_tesseract_noop)
+def test_quick(spoof_tesseract_cache):
+    check_ocrmypdf('ccitt.pdf', 'test_quick.pdf', env=spoof_tesseract_cache)


 def test_deskew(spoof_tesseract_noop):
@@ -162,6 +165,38 @@ def test_clean(spoof_tesseract_noop):
                   env=spoof_tesseract_noop)


+def test_remove_background(spoof_tesseract_noop):
+    from PIL import Image
+
+    # Ensure the input image does not contain pure white/black
+    im = Image.open(_infile('congress.jpg'))
+    assert im.getextrema() != ((0, 255), (0, 255), (0, 255))
+
+    output_pdf = check_ocrmypdf(
+        'congress.jpg', 'test_remove_bg.pdf', '--remove-background',
+        '--image-dpi', '150',
+        env=spoof_tesseract_noop)
+
+    from ocrmypdf.ghostscript import rasterize_pdf
+    import logging
+    log = logging.getLogger()
+
+    output_png = _outfile('remove_bg.png')
+
+    rasterize_pdf(
+        output_pdf,
+        output_png,
+        xres=100,
+        yres=100,
+        raster_device='png16m',
+        log=log)
+
+
+    # The output image should contain pure white and black
+    im = Image.open(output_png)
+    assert im.getextrema() == ((0, 255), (0, 255), (0, 255))
+
+
 # This will run 5 * 2 * 2 = 20 test cases
@pytest.mark.parametrize(
    "pdf",
@@ -404,6 +439,7 @@ def test_maximum_options(spoof_tesseract_cache, renderer, output_type):
    check_ocrmypdf(
        'multipage.pdf', 'test_multipage%s.pdf' % renderer,
        '-d', '-c', '-i', '-g', '-f', '-k', '--oversample', '300',
+        '--remove-background',
        '--skip-big', '10', '--title', 'Too Many Weird Files',
        '--author', 'py.test', '--pdf-renderer', renderer,
        '--output-type', output_type,
@@ -615,19 +651,33 @@ def test_stdin(spoof_tesseract_noop):
    input_file = _infile('francais.pdf')
    output_file = _outfile('test_stdin.pdf')

-    # Runs: cat testfile.pdf | ocrmypdf - output.pdf
+    # Runs: ocrmypdf - output.pdf < testfile.pdf
+    with open(input_file, 'rb') as input_stream:
+        p_args = OCRMYPDF + ['-', output_file]
+        p = Popen(
+            p_args, close_fds=True, stdout=PIPE, stderr=PIPE,
+            stdin=input_stream, env=spoof_tesseract_noop)
+        out, err = p.communicate()

-    p1_args = ['cat', input_file]
-    p1 = Popen(p1_args, close_fds=True, stdin=DEVNULL, stdout=PIPE)
+        assert p.returncode == ExitCode.ok

-    p2_args = OCRMYPDF + ['-', output_file]
-    p2 = Popen(
-        p2_args, close_fds=True, stdout=PIPE, stderr=PIPE,
-        stdin=p1.stdout, env=spoof_tesseract_noop)
-    p1.stdout.close()
-    out, err = p2.communicate()

-    assert p2.returncode == ExitCode.ok
+def test_stdout(spoof_tesseract_noop):
+    input_file = _infile('francais.pdf')
+    output_file = _outfile('test_stdout.pdf')
+
+    # Runs: ocrmypdf francais.pdf - > test_stdout.pdf
+    with open(output_file, 'wb') as output_stream:
+        p_args = OCRMYPDF + [input_file, '-']
+        p = Popen(
+            p_args, close_fds=True, stdout=output_stream, stderr=PIPE,
+            stdin=DEVNULL, env=spoof_tesseract_noop)
+        out, err = p.communicate()
+
+        assert p.returncode == ExitCode.ok
+
+    from ocrmypdf import qpdf
+    assert qpdf.check(output_file, log=None)


 def test_masks(spoof_tesseract_noop):
@@ -635,4 +685,40 @@ def test_masks(spoof_tesseract_noop):


 def test_linearized_pdf_and_indirect_object(spoof_tesseract_noop):
-    check_ocrmypdf('milk.pdf', 'test_milk.pdf', env=spoof_tesseract_noop)
+    check_ocrmypdf(
+        'epson.pdf', 'test_epson.pdf',
+        env=spoof_tesseract_noop)
+
+
+def test_rotated_skew_timeout():
+    """This document contains an image that is rotated 90 into place with a
+    /Rotate tag and intentionally skewed by altering the transformation matrix.
+
+    This tests for a bug where the combinatino of preprocessing and a tesseract
+    timeout produced a page whose dimensions did not match the original's.
+    """
+
+    input_file = _infile('rotated_skew.pdf')
+    in_pageinfo = pdf_get_all_pageinfo(input_file)[0]
+
+    assert in_pageinfo['height_pixels'] < in_pageinfo['width_pixels'], \
+        "Expected the input page to be landscape"
+    assert in_pageinfo['rotate'] == 90, "Expected a rotated page"
+
+    out = check_ocrmypdf(
+        'rotated_skew.pdf', 'test_rotated_skew.pdf',
+        '--pdf-renderer', 'hocr',
+        '--deskew', '--tesseract-timeout', '0')
+
+    out_pageinfo = pdf_get_all_pageinfo(out)[0]
+
+    assert out_pageinfo['height_pixels'] > out_pageinfo['width_pixels'], \
+        "Expected the output page to be portrait"
+
+    assert out_pageinfo['rotate'] == 0, \
+        "Expected no page rotation for output"
+
+    assert in_pageinfo['width_pixels'] == out_pageinfo['height_pixels'] and \
+        in_pageinfo['height_pixels'] == out_pageinfo['width_pixels'], \
+        "Expected page rotation to be baked in"
+
@@ -97,11 +97,6 @@ def test_single_page_image():
    assert pdfimage['width'] == 8
    assert pdfimage['color'] == 'gray'

-    # While unexpected, this is correct
-    # PDF spec says /FlateDecode image must have /BitsPerComponent 8
-    # So mono images get upgraded to 8-bit
-    assert pdfimage['bpc'] == 8
-
    # DPI in a 1"x1" is the image width
    assert abs(pdfimage['dpi_w'] - 8) < 1e-5
    assert abs(pdfimage['dpi_h'] - 8) < 1e-5
Author	SHA1	Message	Date
James R. Barlow	d0c22ce01d	v4.3.2 release notes	2016-11-10 23:16:08 -08:00
James R. Barlow	23c95e9660	ghostscript: elide overprinting to fix PDF/A errors in GS 9.20 It looks like GS 9.19 can incorrectly set overprinting for the text layer even though this makes no sense in PDF/A, or at least someone produced PDFs that have this after a Tesseract PDF -> GS PDF/A conversion. GS 9.20 complains about this. Instead of aborting, elide the feature. See http://git.ghostscript.com/?p=ghostpdl.git;a=commitdiff;h=094d5a1880f1cb9ed320ca9353eb69436e09b594 and issue #107. It looks like it is better to elide features and warn about elision rather than abort with an error.	2016-11-10 14:48:02 -08:00
James R. Barlow	eecab9b95d	pdfa: fix KeyError on pdfa_dict if document has some xmp metadata but not exactly what we’re looking for	2016-11-09 05:41:12 -08:00
James R. Barlow	8abc2f113c	Merge branch 'develop'	2016-11-07 14:36:50 -08:00
James R. Barlow	949d2ff1c2	v4.3.1 release notes	2016-11-07 14:36:08 -08:00
James R. Barlow	1c8b763d53	test_pageinfo: Remove bits per component test The behavior of this test will ultimately depend on what version of img2pdf is installed, since after my patch it will be able to produce 1bpp images.	2016-11-07 14:35:54 -08:00
James R. Barlow	bb91393b85	Fix “deskew-rotate” bug. Turns out this occurred in any case where pdf-renderer hocr was used and a tesseract timeout or error occurred. We created a replacement page based on the unrotated page dimensions instead of the input image’s dimensions.	2016-11-07 14:17:31 -08:00
James R. Barlow	cc9c0d819e	Add test case for documents that get rotated incorrectly after deskew	2016-11-07 14:15:03 -08:00
James R. Barlow	a72b8caf47	Update documentation on other languages, multilingual documents	2016-11-07 14:14:06 -08:00
James R. Barlow	fdd9b8b8ce	Optimize some of the test resources to reduce file sizes Mostly by reducing RGB -> monochrome and applying JBIG2 compression	2016-11-07 14:01:23 -08:00
James R. Barlow	c096b4ca8c	Make debug dump of pageinfo at the end of processing readable	2016-11-04 02:23:02 -07:00
James R. Barlow	427add3008	Add @posttask debug hooks	2016-11-03 18:15:21 -07:00
James R. Barlow	c45871700d	Fix bug: LeptonicaErrorTrap() leaks file handles	2016-11-03 15:51:27 -07:00
Sean Whittonandjbarlow83	6821e8eeb2	disable mathjax sphinx extension (#103 ) Mathjax isn't actually needed for OCRmyPDF's docs, but enabling this extension causes the browser to download a copy of mathjax.js from cdn.mathjax.org anyway. I have to disable this for the offline docs bundled with Debian, but since you're not using mathjax, it would be nice to have the diff merged upstream.	2016-11-01 21:56:57 -07:00
James R. Barlow	a4f07756a5	tesseract caching: don't transcode tesseract's output, hash source file For sanity's sake, deal with tesseract streams in binary without transcoding (via universal_newlines, etc.). The only differences are printing messages regarding spoofing. Also hash the source file so that changes to the cache mechanism invalidate old cache automatically. That is probably too aggressive, but simple and safer than the previous approach.	2016-10-28 16:44:12 -07:00
James R. Barlow	f24fb0e0c5	Obligatory MANIFEST.in repair	2016-10-28 01:28:46 -07:00
James R. Barlow	73b88a0a6f	More work on documentation	2016-10-28 01:22:40 -07:00
James R. Barlow	c42f39e2d4	Update README to point to ReadTheDocs	2016-10-28 00:33:17 -07:00
James R. Barlow	5e5fe3175f	docs: OS X -> macOS branding change	2016-10-28 00:32:57 -07:00
James R. Barlow	cab65d1f11	pageinfo: add a python3.4 implementation of isclose()	2016-10-28 00:31:04 -07:00
James R. Barlow	245f05d5f4	docs: allow python setup.py install --force to bypass checks ReadTheDocs needs this.	2016-10-28 00:07:26 -07:00
James R. Barlow	dda751f9e3	Merge branch 'feature/docs' into develop # Conflicts: # ocrmypdf/__main__.py	2016-10-27 23:50:08 -07:00
James R. Barlow	3d37ae988a	Update release notes for 4.3	2016-10-27 23:48:12 -07:00
James R. Barlow	717acd9855	Prevent dumping binary PDFs to stdout	2016-10-27 16:20:53 -07:00
James R. Barlow	2e4431cc63	Allow piping output to stdout	2016-10-27 16:14:42 -07:00
James R. Barlow	f7387b0859	test_stdin: simplify this test No need to involve 'cat', just hook the file up to stdin.	2016-10-27 16:01:07 -07:00
James R. Barlow	a09f6b8977	Test cases: check that stdout is clear of output To ensure piping to stdout is possible.	2016-10-27 15:58:24 -07:00
James R. Barlow	d63449c214	main: don't print output file location to stdout, use stderr	2016-10-27 15:57:33 -07:00
James R. Barlow	a86805f0d9	Remove possibly non-free page from "multipage.pdf"	2016-10-27 15:56:43 -07:00
James R. Barlow	7d2009ccef	ghostscript: log errors from stdout	2016-10-27 15:36:20 -07:00
James R. Barlow	18ae5db06d	ghostscript: ensure raster resolution is specified in integer units	2016-10-27 15:35:33 -07:00
James R. Barlow	9a1838f102	pageinfo: accept "cm/Do" image drawing without the usual "q/Q" wrapper Some PDFs omit the traditional q/Q wrapper and alter ctm with a stack depth of zero, so make our test for stack depth specifically test for the case where the PDF calls for rendering to an uninitialized ctm. Probably related to #97.	2016-10-27 15:35:00 -07:00
James R. Barlow	e20346032d	leptonica: add color testing functions for future experiments	2016-10-27 14:49:49 -07:00
James R. Barlow	693a27d76c	leptonica: add iPython display hook and equality test	2016-10-26 14:44:41 -07:00
James R. Barlow	203966d86b	leptonica: fix Pillow conversion for 1-bit and 8-bit gray images	2016-10-26 13:10:13 -07:00
James R. Barlow	7eca8508fd	Implement new preprocessing feature, background removal	2016-10-14 17:23:34 -07:00
James R. Barlow	b85270df1c	Merge branch 'master' into develop	2016-10-14 15:56:58 -07:00
James R. Barlow	aff597cef4	v4.2.5: update release notes, fix silly typo in pageinfo.py	2016-10-13 13:26:39 -07:00
James R. Barlow	61b05b3dee	Fix issue: BitsPerComponent is an optional field, sometimes omitted	2016-10-13 13:15:27 -07:00
Julian Kahnertandjbarlow83	453c4ef602	Update README.rst (#98 ) `brew install tesseract` just installed the english language pack not French, German or Spanish	2016-10-12 11:20:58 -07:00
James R. Barlow	cf4b04f92d	The main 'quick' test should be a file that OCRs to recognizable text	2016-10-07 16:25:34 -07:00
James R. Barlow	06c6999987	Merge commit '07891d994aab92e7a14aebe1ac509aab2d4f170c'	2016-10-07 12:45:56 -07:00
James R. Barlow	013c5a369f	Replace redacted file with an OCR-able file	2016-10-07 12:45:22 -07:00
James R. Barlow	07891d994a	Replace redacted file with an OCR-able file	2016-10-07 12:44:49 -07:00
James R. Barlow	6baf8668a6	Replace with non-free file milk.pdf with free equivalent	2016-10-06 13:10:28 -07:00
James R. Barlow	4ba2962c56	Comment on non-free files	2016-10-05 16:48:16 -07:00
James R. Barlow	7ad92f5db4	Merge branch 'master' of https://github.com/jbarlow83/OCRmyPDF	2016-10-05 16:39:00 -07:00
James R. Barlow	4dad09cc91	resources/README: replace the other large table with a list table	2016-10-05 16:38:51 -07:00
Sean Whittonandjbarlow83	7b2e0c7a7a	also exclude .git in pytest.ini (#94 )	2016-09-15 08:56:14 -07:00
Sean Whittonandjbarlow83	7f08f15fc9	pytest skipif for milk.pdf test (#95 ) Skip the test if the fair use restricted milk.pdf is not present.	2016-09-15 08:55:31 -07:00
James R. Barlow	825c0f8b2a	Note that milk.pdf is non-free, start using list-tables	2016-09-10 14:44:00 -07:00
James R. Barlow	dbe880bc41	Update tesseract supported languages	2016-09-09 12:55:07 -07:00
James R. Barlow	2ec516b6ff	leptonica: learn a few new tricks Found some interesting options for background norm.	2016-09-09 12:54:36 -07:00
James R. Barlow	7942a01e50	leptonica: This is not a Py2 module anymore	2016-09-08 20:59:38 -07:00
James R. Barlow	df684f9344	Update tesseract supported languages	2016-09-08 15:52:25 -07:00
James R. Barlow	ae16e95e42	leptonica: scale should be a tuple for consistency	2016-09-08 11:38:10 -07:00
James R. Barlow	2ac8e8a0cc	More doc tweaks, mainly introduction	2016-09-08 11:29:53 -07:00
James R. Barlow	0a0ceda71f	Start the documentation	2016-09-06 13:52:40 -07:00
James R. Barlow	220f1ce161	tasks.py: stop tracking this file for now This helper script is still in development and needs to be changed each release, which breaks the release. It shouldn't be in MANIFEST.in at all because it's not part of a distribution.	2016-09-04 10:55:57 -07:00
James R. Barlow	c62a8a97c9	v4.2.4 release notes	2016-09-01 21:33:38 -07:00
James R. Barlow	f8a1136979	tasks: show logging info	2016-09-01 21:24:13 -07:00
James R. Barlow	9ca29c787b	Update description of masks.pdf to reflect what it actually tests	2016-09-01 21:21:14 -07:00
James R. Barlow	6af748a251	pageinfo: regression - didn't add inline images to list	2016-09-01 15:27:51 -07:00
James R. Barlow	9041867f86	pageinfo: exclude images from DPI calculation if drawn at stack depth 0 More thorough testing showed that Acrobat do not presume that images fill the page if the CTM is unspecified, as tests/resources/masks.pdf seems to want. Instead they treat it literally and draw the image as 1x1 PDF units or 1/72" square in the bottom left corner of the page. Seems like the best thing to do is ignore any such images for the purpose of DPI calculation. masks.pdf still works out okay because it has other images. For more robustness we could consider invalidating any DPI above some limit, or warning the user about these microdot thumbnails.	2016-09-01 14:23:31 -07:00
James R. Barlow	04099b087c	pageinfo: handle stencil masks when stack depth > 0	2016-09-01 14:03:30 -07:00
James R. Barlow	6d6234714c	tasks: fix logic error and make magic numbers disappear	2016-09-01 14:03:08 -07:00