v4.1.4 release notes

Issue #79.
User submitted PDF with ICC profile attached to the monochrome image
in the input file, which is not common but useful for PDFs that want to
define how light the paper is or how dark the black is. The code was
written to assume unusual images are full color unless it can prove
otherwise. Handle this simple case. Other ICC cases should be tested.

README.rst

@@ -54,6 +54,8 @@ Debian and Ubuntu
 Users of Debian 9 or later or Ubuntu 16.10 or later may simply
 ``apt-get install ocrmypdf``.
 
+.. _Docker:
+
 Installing the Docker image
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -63,11 +65,15 @@ If you have `Docker <https://docs.docker.com/>`__ installed on your system, you
 a Docker image of the latest release.
 
 Follow the Docker installation instructions for your platform.  If you can run this command
-successfully, your system is ready to download and execute the image::
+successfully, your system is ready to download and execute the image:
+
+.. code-block:: bash
 
    docker run hello-world
    
-OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker engine host, and then follow these optional steps to enable multiple CPUs::
+OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker engine host, and then follow these optional steps to enable multiple CPUs:
+
+.. code-block:: bash
 
    # Optional step for Mac OS X users
    docker-machine stop "yourVM"
@@ -76,29 +82,41 @@ OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine i
    eval $(docker-machine env "yourVM")
 
 Assuming you have a Docker engine running somewhere, you can run these commands to download
-the image::
+the image:
+
+.. code-block:: bash
 
    docker pull jbarlow83/ocrmypdf
 
-Then tag it to give a more convenient name, just ocrmypdf::
+Then tag it to give a more convenient name, just ocrmypdf:
+
+.. code-block:: bash
 
    docker tag jbarlow83/ocrmypdf ocrmypdf
 
-This image contains language packs for English, French, Spanish and German. The alternative "polyglot" image provides `all available language packs <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#languages>`__::
+This image contains language packs for English, French, Spanish and German. The alternative "polyglot" image provides `all available language packs <https://github.com/tesseract-ocr/tesseract/blob/master/doc/tesseract.1.asc#languages>`__:
+
+.. code-block:: bash
 
    # Alternative step: If you need all language packs
    docker pull jbarlow83/ocrmypdf-polyglot
    docker tag jbarlow83/ocrmypdf-polyglot ocrmypdf
 
-You can then run ocrmypdf using the command::
+You can then run ocrmypdf using the command:
+
+.. code-block:: bash
 
    docker run ocrmypdf --help
   
-To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`__, such as this in this template::
+To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`__, such as this in this template:
+
+.. code-block:: bash
 
    docker run -v "$(pwd):/home/docker" <other docker arguments>   ocrmypdf <your arguments to ocrmypdf>
 
-In this worked example, the current working directory contains an input file called ``test.pdf`` and the output will go to ``output.pdf``:: 
+In this worked example, the current working directory contains an input file called ``test.pdf`` and the output will go to ``output.pdf``: 
+
+.. code-block:: bash
 
    docker run -v "$(pwd):/home/docker"   ocrmypdf --skip-text test.pdf output.pdf
 
@@ -112,11 +130,15 @@ These instructions probably work on all Mac OS X versions later than 10.7 (Lion)
 
 If it's not already present, `install Homebrew <http://brew.sh/>`__.
 
-Update Homebrew::
+Update Homebrew:
+
+.. code-block:: bash
 
    brew update
    
-Install or upgrade the required Homebrew packages, if any are missing::
+Install or upgrade the required Homebrew packages, if any are missing:
+
+.. code-block:: bash
 
    brew install libpng openjpeg jbig2dec     # image libraries
    brew install qpdf
@@ -126,16 +148,22 @@ Install or upgrade the required Homebrew packages, if any are missing::
    brew install unpaper    # optional
    brew install tesseract
    
-Update the homebrew pip and install Pillow::
+Update the homebrew pip and install Pillow:
+
+.. code-block:: bash
 
    pip3 install --upgrade pip
    pip3 install --upgrade pillow
 
-You can then install OCRmyPDF from PyPI::
+You can then install OCRmyPDF from PyPI:
+
+.. code-block:: bash
 
    pip3 install ocrmypdf
 
-The command line program should now be available::
+The command line program should now be available:
+
+.. code-block:: bash
 
    ocrmypdf --help
 
@@ -144,12 +172,16 @@ Installing on Ubuntu 14.04 LTS
 
 Installing on Ubuntu 14.04 LTS (trusty) is more difficult than other options, because of certain bugs in Python package installation.
 
-Update apt-get::
+Update apt-get:
+
+.. code-block:: bash
 
    sudo apt-get update
    sudo apt-get upgrade
    
-Install system dependencies::
+Install system dependencies:
+
+.. code-block:: bash
 
    sudo apt-get install \
       zlib1g-dev \
@@ -165,13 +197,17 @@ Install system dependencies::
       python3-reportlab
 
 If you wish install OCRmyPDF to the system Python, then install as follows (note this installs new packages
-into your system Python, which could interfere with other programs)::
+into your system Python, which could interfere with other programs):
+
+.. code-block:: bash
 
    sudo pip3 install ocrmypdf
    
 If you wish to install OCRmyPDF to a virtual environment to isolate system Python from modified, you can
 follow these steps.  This includes a workaround `for a known, unresolved issue in Ubuntu 14.04's ensurepip
-package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`__::
+package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`__:
+
+.. code-block:: bash
 
    sudo apt-get install python3-venv
    python3 -m venv venv-ocrmypdf --without-pip
@@ -187,8 +223,18 @@ Ubuntu 14.04 only installs ``unpaper`` version 0.4.2, which is not supported by
 Installing on Windows
 ~~~~~~~~~~~~~~~~~~~~~
 
-Direct installation on Windows is not possible.  Install the Docker container as described above.
+Direct installation on Windows is not possible.  Install the _`Docker` container as described above.  Ensure that your command prompt can run the docker "hello world" container.
 
+Running on Windows
+~~~~~~~~~~~~~~~~~~
+
+The command line syntax to run ocrmypdf from a command prompt will resemble:
+
+.. code-block:: bat
+
+   docker run -v /c/Users/sampleuser:/home/docker ocrmypdf --skip-text test.pdf output.pdf
+
+where /c/Users/sampleuser is a Unix representation of the Windows path C:\\Users\\sampleuser, assuming a user named "sampleuser" is running ocrmypdf on a file in their home directory, and the files "test.pdf" and "output.pdf" are in the sampleuser folder. The Windows user must have read and write permissions.
       
 Installing HEAD revision from sources
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -196,21 +242,29 @@ Installing HEAD revision from sources
 If you have ``git`` and ``python3.4`` or ``python3.5`` installed, you can install from source. When the ``pip`` installer runs,
 it will alert you if dependencies are missing.
 
-To install the HEAD revision from sources in the current Python 3 environment::
+To install the HEAD revision from sources in the current Python 3 environment:
+
+.. code-block:: bash
 
    pip3 install git+https://github.com/jbarlow83/OCRmyPDF.git
 
-Or, to install in `development mode <https://pythonhosted.org/setuptools/setuptools.html#development-mode>`__,  allowing customization of OCRmyPDF, use the ``-e`` flag::
+Or, to install in `development mode <https://pythonhosted.org/setuptools/setuptools.html#development-mode>`__,  allowing customization of OCRmyPDF, use the ``-e`` flag:
+
+.. code-block:: bash
 
    pip3 install -e git+https://github.com/jbarlow83/OCRmyPDF.git
    
 On certain Linux distributions such as Ubuntu, you may need to use 
-run the install command as superuser::
+run the install command as superuser:
+
+.. code-block:: bash
 
    sudo pip3 install [-e] git+https://github.com/jbarlow83/OCRmyPDF.git
    
 Note that this will alter your system's Python distribution. If you prefer 
-to not install as superuser, you can install the package in a Python virtual environment::
+to not install as superuser, you can install the package in a Python virtual environment:
+
+.. code-block:: bash
 
    git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
    pyvenv venv
@@ -221,7 +275,9 @@ to not install as superuser, you can install the package in a Python virtual env
 However, ``ocrmypdf`` will only be accessible on the system PATH after
 you activate the virtual environment.
 
-To run the program::
+To run the program:
+
+.. code-block:: bash
    
    ocrmypdf --help
 
@@ -234,7 +290,9 @@ Languages
 ---------
 
 OCRmyPDF uses Tesseract for OCR, and relies on its language packs. For Linux users,
-you can often find packages that provide language packs::
+you can often find packages that provide language packs:
+
+.. code-block:: bash
 
    # Debian/Ubuntu users
    sudo apt-get install tesseract-ocr-chi-sim
@@ -245,9 +303,15 @@ languages can be requested.
 Support
 -------
 
-In case you detect an issue, please:
+Once ocrmypdf is installed, the built-in help which explains the command syntax and options can be accessed via:
 
--  Check if your issue is already known
+.. code-block:: bash
+
+   ocrmypdf --help
+
+If you detect an issue, please:
+
+-  Check whether your issue is already known
 -  If no problem report exists on github, please create one here:
    https://github.com/jbarlow83/OCRmyPDF/issues
 -  Describe your problem thoroughly

RELEASE_NOTES.rst

@@ -5,7 +5,23 @@ Please always read this file before installing the package
 
 Download software here: https://github.com/jbarlow83/OCRmyPDF/tags
 
-v4.1.1:
+
+v4.1.4:
+=======
+
+-  Bug fix: monochrome images with an ICC profile attached were incorrectly converted to full color images if lossless reconstruction was not possible due to other settings; consequence was increased file size for these images
+
+
+v4.1.3:
+=======
+
+-  More helpful error message for PDFs with version 4 security handler
+-  Update usage instructions for Windows/Docker users
+-  Fix order of operations for matrix multiplication (no effect on most users)
+-  Add a few leptonica wrapper functions (no effect on most users)
+
+
+v4.1.2:
 =======
 
 -  Replace IEC sRGB ICC profile with Debian's sRGB (from icc-profiles-free) which is more compatible with the MIT license

ocrmypdf/__init__.py

@@ -11,6 +11,7 @@ class ExitCode(IntEnum):
     file_access_error = 5
     already_done_ocr = 6
     child_process_error = 7
+    encrypted_pdf = 8
     other_error = 15
 
 

ocrmypdf/leptonica.py

@@ -17,6 +17,7 @@ from tempfile import TemporaryFile
 from ctypes.util import find_library
 from .lib._leptonica import ffi
 from functools import lru_cache
+from enum import Enum
 
 lept = ffi.dlopen(find_library('lept'))
 
@@ -83,6 +84,13 @@ class LeptonicaIOError(LeptonicaError):
     pass
 
 
+class RemoveColormap(Enum):
+    to_binary = 0
+    to_grayscale = 1
+    to_full_color = 2
+    based_on_src = 3
+
+
 class Pix:
     """Wrapper around leptonica's PIX object.
 
@@ -98,14 +106,14 @@ class Pix:
     in a threadsafe manner if a Python threading.Lock protects the data.
     """
 
-    def __init__(self, cpix):
-        self.cpix = ffi.gc(cpix, Pix._pix_destroy)
+    def __init__(self, pix):
+        self._pix = ffi.gc(pix, Pix._pix_destroy)
 
     def __repr__(self):
-        if self.cpix:
+        if self._pix:
             s = "<leptonica.Pix image size={0}x{1} depth={2} at 0x{3:x}>"
-            return s.format(self.cpix.w, self.cpix.h, self.cpix.d,
-                            int(ffi.cast("intptr_t", self.cpix)))
+            return s.format(self._pix.w, self._pix.h, self._pix.d,
+                            int(ffi.cast("intptr_t", self._pix)))
         else:
             return "<leptonica.Pix image NULL>"
 
@@ -113,12 +121,16 @@ class Pix:
         data = ffi.new('l_uint32 **')
         size = ffi.new('size_t *')
 
-        err = lept.pixSerializeToMemory(self.cpix, data, size)
+        err = lept.pixSerializeToMemory(self._pix, data, size)
         if err != 0:
             raise LeptonicaIOError("pixSerializeToMemory")
 
         char_data = ffi.cast('char *', data[0])
+
+        # Copy from C bytes to python bytes()
         data_bytes = ffi.buffer(char_data, size[0])[:]
+
+        # Can now free C bytes
         lept.lept_free(char_data)
         return dict(data=data_bytes)
 
@@ -126,16 +138,41 @@ class Pix:
         cdata_bytes = ffi.new('char[]', state['data'])
         cdata_uint32 = ffi.cast('l_uint32 *', cdata_bytes)
 
-        self.cpix = lept.pixDeserializeFromMemory(
+        pix = lept.pixDeserializeFromMemory(
             cdata_uint32, len(state['data']))
+        Pix.__init__(self, pix)
 
     @property
     def width(self):
-        return self.cpix.w
+        return self._pix.w
 
     @property
     def height(self):
-        return self.cpix.h
+        return self._pix.h
+
+    @property
+    def depth(self):
+        return self._pix.d
+
+    @property
+    def size(self):
+        return (self._pix.w, self._pix.h)
+
+    @property
+    def info(self):
+        return {'dpi': (self._pix.xres, self._pix.yres)}
+
+    @property
+    def mode(self):
+        "Return mode like PIL.Image"
+        if self.depth == 1:
+            return '1'
+        elif self.depth >= 16:
+            return 'RGB'
+        elif not self._pix.colormap:
+            return 'L'
+        else:
+            return 'P'
 
     @classmethod
     def read(cls, filename):
@@ -158,7 +195,29 @@ class Pix:
         with LeptonicaErrorTrap():
             lept.pixWriteImpliedFormat(
                 filename.encode(sys.getfilesystemencoding()),
-                self.cpix, jpeg_quality, jpeg_progressive)
+                self._pix, jpeg_quality, jpeg_progressive)
+
+    def topil(self):
+        "Returns a PIL.Image version of this Pix"
+        from PIL import Image
+
+        with LeptonicaErrorTrap():
+            pix_swapped = Pix(lept.pixEndianByteSwapNew(self._pix))
+
+        size = (pix_swapped._pix.wpl * 4, pix_swapped._pix.h)
+        buf = ffi.buffer(pix_swapped._pix.data, size[0] * size[1])
+
+        im_raw = Image.frombytes(self.mode, size, buf, 'raw')
+
+        # Leptonica stores images in 32-bit words
+        # Need to crop the any trailing amount
+        box = (0, 0, self.width, self.height)
+        im = im_raw.crop(box)
+
+        return im
+
+    def show(self):
+        return self.topil().show()
 
     def deskew(self, reduction_factor=0):
         """Returns the deskewed pix object.
@@ -170,16 +229,21 @@ class Pix:
             for skew angle
         """
         with LeptonicaErrorTrap():
-            return Pix(lept.pixDeskew(self.cpix, reduction_factor))
+            return Pix(lept.pixDeskew(self._pix, reduction_factor))
 
     def scale(self, scalex, scaley):
         "Returns the pix object rescaled according to the proportions given."
         with LeptonicaErrorTrap():
-            return Pix(lept.pixScale(self.cpix, scalex, scaley))
+            return Pix(lept.pixScale(self._pix, scalex, scaley))
 
     def rotate180(self):
         with LeptonicaErrorTrap():
-            return Pix(lept.pixRotate180(ffi.NULL, self.cpix))
+            return Pix(lept.pixRotate180(ffi.NULL, self._pix))
+
+    def rotate_orth(self, quads):
+        "Orthographic rotation, quads: 0-3, number of clockwise rotations"
+        with LeptonicaErrorTrap():
+            return Pix(lept.pixRotateOrth(self._pix, quads))
 
     def find_skew(self):
         """Returns a tuple (deskew angle in degrees, confidence value).
@@ -189,31 +253,105 @@ class Pix:
         with LeptonicaErrorTrap():
             angle = ffi.new('float *', 0.0)
             confidence = ffi.new('float *', 0.0)
-            result = lept.pixFindSkew(self.cpix, angle, confidence)
+            result = lept.pixFindSkew(self._pix, angle, confidence)
             if result == 0:
                 return (angle[0], confidence[0])
             else:
                 return (None, None)
 
+    def convert_rgb_to_luminance(self):
+        with LeptonicaErrorTrap():
+            gray_pix = lept.pixConvertRGBToLuminance(self._pix)
+            if gray_pix:
+                return Pix(gray_pix)
+            return None
+
+    def remove_colormap(self, removal_type):
+        """Remove a palette
+
+            removal_type - RemovalColormap()
+        """
+
+        with LeptonicaErrorTrap():
+            return Pix(lept.pixRemoveColormap(self._pix, removal_type))
+
     def otsu_adaptive_threshold(
             self, tile_size=(300, 300), kernel_size=(4, 4), scorefract=0.1):
         with LeptonicaErrorTrap():
             sx, sy = tile_size
             smoothx, smoothy = kernel_size
-            p_cpix = ffi.new('PIX **')
+            p_pix = ffi.new('PIX **')
 
             result = lept.pixOtsuAdaptiveThreshold(
-                self.cpix,
+                self._pix,
                 sx, sy,
                 smoothx, smoothy,
                 scorefract,
                 ffi.NULL,
-                p_cpix)
+                p_pix)
             if result == 0:
-                return Pix(p_cpix[0])
+                return Pix(p_pix[0])
             else:
                 return None
 
+    def otsu_threshold_on_background_norm(
+            self, mask=None, tile_size=(10, 15), thresh=100, mincount=50,
+            bgval=255, kernel_size=(2, 2), scorefract=0.1):
+        with LeptonicaErrorTrap():
+            sx, sy = tile_size
+            smoothx, smoothy = kernel_size
+            if mask is None:
+                mask = ffi.NULL
+            if isinstance(mask, Pix):
+                mask = mask._pix
+
+            thresh_pix = lept.pixOtsuThreshOnBackgroundNorm(
+                self._pix,
+                mask,
+                sx, sy,
+                thresh, mincount, bgval,
+                smoothx, smoothy,
+                scorefract,
+                ffi.NULL
+                )
+            if thresh_pix == ffi.NULL:
+                return None
+            return Pix(thresh_pix)
+
+    def crop_to_foreground(
+            self, threshold=128, mindist=70, erasedist=30, pagenum=0,
+            showmorph=0, display=0, pdfdir=ffi.NULL):
+        with LeptonicaErrorTrap():
+            cropbox = Box(lept.pixFindPageForeground(
+                self._pix,
+                threshold,
+                mindist,
+                erasedist,
+                pagenum,
+                showmorph,
+                display,
+                pdfdir))
+
+            print(repr(cropbox))
+
+            cropped_pix = lept.pixClipRectangle(
+                self._pix,
+                cropbox._box,
+                ffi.NULL)
+
+            return Pix(cropped_pix)
+
+    def clean_background_to_white(
+            self, mask=None, grayscale=None, gamma=1.0, black=0, white=255):
+        with LeptonicaErrorTrap():
+            return Pix(lept.pixCleanBackgroundToWhite(
+                self._pix,
+                mask or ffi.NULL,
+                grayscale or ffi.NULL,
+                gamma,
+                black,
+                white))
+
     @staticmethod
     @lru_cache(maxsize=1)
     def make_pixel_sum_tab8():
@@ -230,10 +368,10 @@ class Pix:
             pixn_count = ffi.new('l_int32 *')
             tab8 = Pix.make_pixel_sum_tab8()
 
-            lept.pixCountPixels(pix1.cpix, pix1_count, tab8)
-            lept.pixCountPixels(pix2.cpix, pix2_count, tab8)
-            pixn = Pix(lept.pixAnd(ffi.NULL, pix1.cpix, pix2.cpix))
-            lept.pixCountPixels(pixn.cpix, pixn_count, tab8)
+            lept.pixCountPixels(pix1._pix, pix1_count, tab8)
+            lept.pixCountPixels(pix2._pix, pix2_count, tab8)
+            pixn = Pix(lept.pixAnd(ffi.NULL, pix1._pix, pix2._pix))
+            lept.pixCountPixels(pixn._pix, pixn_count, tab8)
 
             # Python converts these int32s to larger units as needed
             # to avoid overflow. Overflow happens easily here.
@@ -244,7 +382,7 @@ class Pix:
             return correlation
         else:
             correlation = ffi.new('float *', 0.0)
-            result = lept.pixCorrelationBinary(pix1.cpix, pix2.cpix,
+            result = lept.pixCorrelationBinary(pix1._pix, pix2._pix,
                                                correlation)
             if result != 0:
                 raise LeptonicaError("Correlation failed")
@@ -252,11 +390,48 @@ class Pix:
 
     @staticmethod
     def _pix_destroy(pix):
-        ptr_to_pix = ffi.new('PIX **', pix)
-        lept.pixDestroy(ptr_to_pix)
+        p_pix = ffi.new('PIX **', pix)
+        lept.pixDestroy(p_pix)
         # print('pix destroy ' + repr(pix))
 
 
+class Box:
+    """Wrapper around Leptonica's BOX objects.
+
+    See class Pix for notes about reference counting.
+    """
+
+    def __init__(self, box):
+        self._box = ffi.gc(box, Box._box_destroy)
+
+    def __repr__(self):
+        if self._box:
+            return '<leptonica.Box x={0} y={1} w={2} h={3}>'.format(
+                self.x, self.y, self.w, self.h)
+        return '<leptonica.Box NULL>'
+
+    @property
+    def x(self):
+        return self._box.x
+
+    @property
+    def y(self):
+        return self._box.y
+
+    @property
+    def w(self):
+        return self._box.w
+
+    @property
+    def h(self):
+        return self._box.h
+
+    @staticmethod
+    def _box_destroy(box):
+        p_box = ffi.new('BOX **', box)
+        lept.boxDestroy(p_box)
+
+
 @lru_cache(maxsize=1)
 def get_leptonica_version():
     """Get Leptonica version string.

ocrmypdf/lib/compile_leptonica.py

@@ -42,6 +42,18 @@ struct PixColormap
     l_int32          n;         /* number of color entries used            */
 };
 typedef struct PixColormap  PIXCMAP;
+
+struct Box
+{
+    l_int32            x;
+    l_int32            y;
+    l_int32            w;
+    l_int32            h;
+    l_uint32           refcount;      /* reference count (1 if no clones)  */
+
+};
+typedef struct Box    BOX;
+
 """)
 
 ffi.cdef("""
@@ -50,10 +62,17 @@ PIX * pixScale ( PIX *pixs, l_float32 scalex, l_float32 scaley );
 l_int32 pixFindSkew ( PIX *pixs, l_float32 *pangle, l_float32 *pconf );
 l_int32 pixWriteImpliedFormat ( const char *filename, PIX *pix, l_int32 quality, l_int32 progressive );
 void pixDestroy ( PIX **ppix );
+
+PIX *
+pixEndianByteSwapNew(PIX  *pixs);
+
 PIX * pixDeskew ( PIX *pixs, l_int32 redsearch );
 char * getLeptonicaVersion (  );
 l_int32 pixCorrelationBinary(PIX *pix1, PIX *pix2, l_float32 *pval);
 PIX *pixRotate180(PIX *pixd, PIX *pixs);
+PIX *
+pixRotateOrth(PIX     *pixs,
+              l_int32  quads);
 
 l_int32 pixCountPixels ( PIX *pix, l_int32 *pcount, l_int32 *tab8 );
 PIX * pixAnd ( PIX *pixd, PIX *pixs1, PIX *pixs2 );
@@ -62,6 +81,10 @@ l_int32 * makePixelSumTab8 ( void );
 PIX * pixDeserializeFromMemory ( const l_uint32 *data, size_t nbytes );
 l_int32 pixSerializeToMemory ( PIX *pixs, l_uint32 **pdata, size_t *pnbytes );
 
+PIX * pixConvertRGBToLuminance(PIX *pixs);
+
+PIX * pixRemoveColormap(PIX *pixs, l_int32  type);
+
 l_int32
 pixOtsuAdaptiveThreshold(PIX       *pixs,
                          l_int32    sx,
@@ -72,6 +95,45 @@ pixOtsuAdaptiveThreshold(PIX       *pixs,
                          PIX      **ppixth,
                          PIX      **ppixd);
 
+PIX *
+pixOtsuThreshOnBackgroundNorm(PIX       *pixs,
+                              PIX       *pixim,
+                              l_int32    sx,
+                              l_int32    sy,
+                              l_int32    thresh,
+                              l_int32    mincount,
+                              l_int32    bgval,
+                              l_int32    smoothx,
+                              l_int32    smoothy,
+                              l_float32  scorefract,
+                              l_int32   *pthresh);
+
+PIX *
+pixCleanBackgroundToWhite(PIX       *pixs,
+                          PIX       *pixim,
+                          PIX       *pixg,
+                          l_float32  gamma,
+                          l_int32    blackval,
+                          l_int32    whiteval);
+
+BOX *
+pixFindPageForeground(PIX         *pixs,
+                      l_int32      threshold,
+                      l_int32      mindist,
+                      l_int32      erasedist,
+                      l_int32      pagenum,
+                      l_int32      showmorph,
+                      l_int32      display,
+                      const char  *pdfdir);
+
+PIX *
+pixClipRectangle(PIX   *pixs,
+                 BOX   *box,
+                 BOX  **pboxc);
+
+void
+boxDestroy(BOX  **pbox);
+
 void lept_free(void *ptr);
 """)
 

ocrmypdf/main.py

@@ -1078,6 +1078,19 @@ def do_ruffus_exception(ruffus_five_tuple):
                 ocrmypdf --pdf-renderer tesseract  [..other args..]
             """))
         return ExitCode.input_file
+    elif exc_name == 'PyPDF2.utils.PdfReadError' and \
+            'not been decrypted' in exc_value:
+        _log.error(textwrap.dedent("""\
+            Input PDF uses either an encryption algorithm or a PDF security
+            handler that is not supported by ocrmypdf.
+
+            For information about this PDF's security use
+                qpdf --show-encryption [...input PDF...]
+
+            (Only algorithms "R = 1" and "R = 2" are supported.)
+
+            """))
+        return ExitCode.encrypted_pdf
     elif not options.verbose:
         _log.error(exc_stack)
         return ExitCode.other_error

ocrmypdf/pageinfo.py

@@ -93,7 +93,7 @@ def _interpret_contents(contentstream):
             ctm = stack.pop()
         elif command == b'cm':
             ctm = matrix_mult(
-                ctm, _matrix_from_shorthand(operands))
+                _matrix_from_shorthand(operands), ctm)
         elif command == b'Do':
             image_name = operands[0]
             image_raster_settings.append(
@@ -230,6 +230,14 @@ def _find_page_regular_images(page, pageinfo, contentsinfo):
             image['color'] = 'jpx' if image['enc'] == 'jpx' else '?'
 
         image['comp'] = FRIENDLY_COMP.get(image['color'], '?')
+
+        # Bit of a hack... infer grayscale if component count is uncertain
+        # but encoding must be monochrome. This happens if a monochrome image
+        # has an ICC profile attached. Better solution would be to examine
+        # the ICC profile.
+        if image['comp'] == '?' and image['enc'] in ('ccitt', 'jbig2'):
+            image['comp'] = FRIENDLY_COMP['gray']
+
         image['dpi_w'] = image['dpi_h'] = 0
 
         for raster in contentsinfo.raster_settings:

ocrmypdf/pdfa.py

@@ -8,7 +8,7 @@ from string import Template
 import codecs
 import pkg_resources
 
-ICC_PROFILE_RELPATH = 'data/sRGB_IEC61966-2-1_black_scaled.icc'
+ICC_PROFILE_RELPATH = 'data/sRGB.icc'
 
 SRGB_ICC_PROFILE = pkg_resources.resource_filename(
     'ocrmypdf', ICC_PROFILE_RELPATH)

setup.py

@@ -230,6 +230,6 @@ setup(
             'ocrmypdf = ocrmypdf.main:run_pipeline'
         ],
     },
-    package_data={'ocrmypdf': ['data/sRGB_IEC61966-2-1_black_scaled.icc']},
+    package_data={'ocrmypdf': ['data/sRGB.icc']},
     include_package_data=True,
     zip_safe=False)

tests/resources/README.rst

@@ -47,12 +47,13 @@ These test resources are assemblies from other previously mentioned files, relea
 
 - cardinal.pdf (four cardinal directions, rotated copies of LinnSequencer.jpg)
 - ccitt.pdf (LinnSequencer.jpg, converted to CCITT encoding)
+- encrypted_algo4.pdf (congress.jpg, encrypted with algorithm 4 - not supported by PyPDF2)
 - graph_ocred.pdf (from graph.pdf)
 - jbig2.pdf (congress.jpg, converted to JBIG2 encoding)
 - multipage.pdf (from several other files)
 - palette.pdf (congress.jpg, converted to a 256-color palette)
 - skew.pdf (from c02-22.pdf)
-- skew-encrypted.pdf (skew.pdf with encrypted applied)
+- skew-encrypted.pdf (skew.pdf with encryption - access supported by PyPDF2)
 
 
 .. _`Wikimedia: LinnSequencer`: https://upload.wikimedia.org/wikipedia/en/b/b7/LinnSequencer_hardware_MIDI_sequencer_brochure_page_2_300dpi.jpg
@@ -63,4 +64,4 @@ These test resources are assemblies from other previously mentioned files, relea
 
 .. _`Wikimedia: Pandas text analysis.png`: https://en.wikipedia.org/wiki/File:Pandas_text_analysis.png
 
-.. _`Wikimedia: JPEG2000 Lichtenstein`: https://en.wikipedia.org/wiki/JPEG_2000#/media/File:Jpeg2000_2-level_wavelet_transform-lichtenstein.png
+.. _`Wikimedia: JPEG2000 Lichtenstein`: https://en.wikipedia.org/wiki/JPEG_2000#/media/File:Jpeg2000_2-level_wavelet_transform-lichtenstein.png

tests/test_main.py

@@ -547,3 +547,7 @@ def test_old_unpaper():
         'c02-22.pdf', 'wont_be_created.pdf', '--clean', env=env)
     assert sh.returncode == ExitCode.missing_dependency
 
+
+def test_algo4():
+    sh, _, _ = run_ocrmypdf_env('encrypted_algo4.pdf', 'wontwork.pdf')
+    assert sh.returncode == ExitCode.encrypted_pdf