Move to my repo: github.com/fritz-hh => jbarlow83

I made several efforts to contact fritz but he is no longer
communicating, and to set up Github integrations with Docker and Travis
CI I need admin access. Which I don't have. So I'm moving it to my own
and aiming the old one at me.

MANIFEST.in

@@ -1 +1,3 @@
 recursive-exclude tests/output *
+include requirements.txt
+include test_requirements.txt

README.rst

@@ -48,14 +48,14 @@ as an inspiration)
 Installation
 ------------
 
-Download OCRmyPDF here: https://github.com/fritz-hh/OCRmyPDF/releases
+Download OCRmyPDF here: https://github.com/jbarlow83/OCRmyPDF/releases
 
 You can install it to a Python virtual environment or system-wide. 
 
 Installing the Docker container
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-For many users, installing the Docker container will be easier than installing all of OCRmyPDF's dependencies.
+For many users, installing the Docker container will be easier than installing all of OCRmyPDF's dependencies. For Windows, it is the only option.
 
 If you have `Docker <https://docs.docker.com/>`__ installed on your system, you can install
 a Docker container of the latest release.
@@ -65,28 +65,36 @@ successfully, your system is ready to download and execute the image::
 
    docker run hello-world
    
+OCRmyPDF will use all available CPU cores.  By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker container host, and then follow these optional steps to enable multiple CPUs::
+
+   # Optional
+   docker-machine stop "yourVM"
+   VBoxManage modifyvm "yourVM" --cpus 2  # or whatever number of core is desired
+   docker-machine start "yourVM"
+   eval $(docker-machine env "yourVM")
+
 Assuming you have a Docker engine running somewhere, you can run these commands to download
 the image::
 
    docker pull jbarlow83/ocrmypdf
 
-Then give it a more friendly, local name::
+Then tag it to give a more convenient name, just ocrmypdf::
 
    docker tag jbarlow83/ocrmypdf ocrmypdf
   
-You can then run using the simple command::
+You can then run using the command::
 
    docker run ocrmypdf --help
   
 To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`__, such as this in this template::
 
-   docker run -v "$(pwd):/home/docker"   ocrmypdf <your arguments>
+   docker run -v "$(pwd):/home/docker" <other docker arguments>   ocrmypdf <your arguments to ocrmypdf>
 
 In this worked example, the current working directory contains an input file called `test.pdf` and the output will go to `output.pdf`:: 
 
    docker run -v "$(pwd):/home/docker"   ocrmypdf --skip-text test.pdf output.pdf
 
-Note that `ocrmypdf` has its own separate -v argument to control debug verbosity.
+Note that `ocrmypdf` has its own separate -v argument to control debug verbosity. All Docker arguments should before the `ocrmypdf` container name and all arguments to `ocrmypdf` should be listed after.
 
 Installing on Mac OS X Yosemite
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -129,7 +137,7 @@ The command line program should now be available::
 Installing on Ubuntu 14.04 LTS
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Installing on Ubuntu 14.04 LTS (trusty) is more difficult than other options.
+Installing on Ubuntu 14.04 LTS (trusty) is more difficult than other options, because of certain bugs in package installation.
 
 Update apt-get::
 
@@ -176,7 +184,7 @@ Installing HEAD revision from sources
 
 To install the HEAD revision from sources in development mode::
 
-   git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
+   git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
    cd OCRmyPDF
    pip3 install -e .
    
@@ -188,7 +196,7 @@ run the install command as superuser::
 Note that this will alter your system's Python distribution. If you prefer 
 to not install as superuser, you can install the package in a Python virtual environment::
 
-   git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
+   git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
    pyvenv venv
    source venv/bin/activate
    cd OCRmyPDF

RELEASE_NOTES.rst

@@ -3,7 +3,7 @@ RELEASE NOTES
 
 Please always read this file before installing the package
 
-Download software here: https://github.com/fritz-hh/OCRmyPDF/tags
+Download software here: https://github.com/jbarlow83/OCRmyPDF/tags
 
 v3.0:
 =====
@@ -21,12 +21,15 @@ New features
 -  PDF metadata (title, author, keywords) are now transferred to the 
    output PDF
 -  PDF metadata can also be set from the command line (``--title``, etc.)
+-  Automatic repairs malformed input PDFs if possible
 -  Added test cases to confirm everything is working
 -  Added option to skip extremely large pages that take too long to OCR and are 
    often not OCRable (e.g. large scanned maps or diagrams); other pages are still
    processed (``--skip-big``)
 -  Added option to kill Tesseract OCR process if it seems to be taking too long on
    a page, while still processing other pages (``--tesseract-timeout``)
+-  Less common colorspaces (CMYK, palette) are now supported by conversion to RGB
+-  Multiple images on the same PDF page are now supported
 
 Changes
 -------
@@ -67,6 +70,21 @@ Changes
 Release candidates
 ------------------
 
+-  rc9:
+
+   - fix issue #118: report error if ghostscript iccprofiles are missing
+   - fixed another issue related to #111: PDF rasterized to palette file
+   - add support image files with a palette
+   - don't try to validate PDF file after an exception occurs
+
+-  rc8:
+
+   - fix issue #111: exception thrown if PDF is missing DocumentInfo dictionary
+
+-  rc7:
+
+   - fix error when installing direct from pip, "no such file 'requirements.txt'"
+
 -  rc6:
 
    - dropped libxml2 (Python lxml) since Python 3's internal XML parser is sufficient
@@ -122,12 +140,18 @@ Fixes
 
 -  Handling of filenames containing spaces: fixed
 
-Notes
+Notes and known issues
------
+----------------------
 
 -  Some dependencies may work with lower versions than tested, so try
    overriding dependencies if they are "in the way" to see if they work.
 
+-  ``--pdf-renderer tesseract`` will output files with an incorrect page size in Tesseract 3.03,
+   due to a bug in Tesseract.
+
+-  PDF files containing "inline images" are not supported and won't be for the 3.0 release. Scanned
+   images almost never contain inline images.
+
 
 v2.2-stable (2014-09-29):
 =========================

ocrmypdf/ghostscript.py

@@ -10,7 +10,9 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log):
     with NamedTemporaryFile(delete=True) as tmp:
         args_gs = [
             'gs',
-            '-dBATCH', '-dNOPAUSE',
+            '-dQUIET',
+            '-dBATCH',
+            '-dNOPAUSE',
             '-sDEVICE=%s' % raster_device,
             '-o', tmp.name,
             '-r{0}x{1}'.format(str(xres), str(yres)),

ocrmypdf/main.py

@@ -103,7 +103,7 @@ check_pil_encoder('zlib', 'PNG')
 parser = cmdline.get_argparse(
     prog="ocrmypdf",
     description="Generate searchable PDF file from an image-only PDF file.",
-    version='3.0rc6',
+    version='3.0',
     fromfile_prefix_chars='@',
     ignored_args=[
         'touch_files_only', 'recreate_database', 'checksum_file_name',
@@ -475,10 +475,15 @@ def rasterize_with_ghostscript(
     if all(image['comp'] == 1 for image in pageinfo['images']):
         if all(image['bpc'] == 1 for image in pageinfo['images']):
             device = 'pngmono'
-        elif not any(image['color'] == 'color'
+        elif all(image['bpc'] > 1 and image['color'] == 'index'
-                     for image in pageinfo['images']):
+                 for image in pageinfo['images']):
+            device = 'png256'
+        elif all(image['bpc'] > 1 and image['color'] == 'gray'
+                 for image in pageinfo['images']):
             device = 'pnggray'
 
+    log.debug("Rendering {0} with {1}".format(
+            os.path.basename(input_file), device))
     xres = max(pageinfo['xres'], options.oversample or 0)
     yres = max(pageinfo['yres'], options.oversample or 0)
 
@@ -729,11 +734,13 @@ def generate_postscript_stub(
     pdf = pypdf.PdfFileReader(input_file)
 
     def from_document_info(key):
-        # pdf.documentInfo.get() DOES NOT work as expected
+        # pdf.documentInfo.get() DOES NOT behave as expected for a dict-like
+        # object, so call with precautions.  TypeError may occur if the PDF
+        # is missing the optional document info section.
         try:
             s = pdf.documentInfo[key]
             return str(s)
-        except KeyError:
+        except (KeyError, TypeError):
             return ''
 
     pdfmark = {
@@ -892,6 +899,7 @@ def run_pipeline():
                 return eval(
                     exc_value,
                     {'ExitCode': ExitCode}, {'exc_value': exc_value})
+        return ExitCode.other_error
 
     if not validate_pdfa(options.output_file, _log):
         _log.warning('Output file: The generated PDF/A file is INVALID')

ocrmypdf/pageinfo.py

@@ -34,6 +34,7 @@ FRIENDLY_COMP = {
     'rgb': 3,
     'cmyk': 4,
     'lab': 3,
+    'index': 1
 }
 
 

ocrmypdf/pdfa.py

@@ -115,6 +115,8 @@ def _get_postscript_icc_path():
         if os.path.exists(path):
             return path
 
+    raise FileNotFoundError("Could not find Ghostscript's iccprofiles")
+
 
 def generate_pdfa_def(target_filename, pdfmark, icc='sRGB'):
     if icc == 'sRGB':

ocrmypdf/unpaper.py

@@ -8,6 +8,7 @@ from tempfile import NamedTemporaryFile
 import sys
 import os
 from functools import lru_cache
+from . import ExitCode
 
 
 @lru_cache(maxsize=1)
@@ -38,10 +39,27 @@ def run(input_file, output_file, dpi, log, mode_args):
     ] + mode_args
 
     SUFFIXES = {'1': '.pbm', 'L': '.pgm', 'RGB': '.ppm'}
-    suffix = ''
 
     im = Image.open(input_file)
-    suffix = SUFFIXES[im.mode]
+    if im.mode not in SUFFIXES.keys():
+        log.info("Converting image to other colorspace")
+        try:
+            if im.mode == 'P' and len(im.getcolors()) == 2:
+                im = im.convert(mode='1')
+            else:
+                im = im.convert(mode='RGB')
+        except IOError:
+            log.error(
+                    "Could not convert image with type " + im.mode)
+            sys.exit(ExitCode.missing_dependency)
+
+    try:
+        suffix = SUFFIXES[im.mode]
+    except KeyError:
+        log.error(
+                "Failed to convert image to a supported format.")
+        sys.exit(ExitCode.missing_dependency)
+
     with NamedTemporaryFile(suffix=suffix) as input_pnm, \
             NamedTemporaryFile(suffix=suffix, mode="r+b") as output_pnm:
         im.save(input_pnm, format='PPM')

setup.py

@@ -179,17 +179,17 @@ tests_require = open('test_requirements.txt').read().splitlines()
 
 setup(
     name='ocrmypdf',
-    version='3.0rc6',  # also update: release notes, main.py
+    version='3.0',  # also update: release notes, main.py
     description='OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched',
-    url='https://github.com/fritz-hh/OCRmyPDF',
+    url='https://github.com/jbarlow83/OCRmyPDF',
-    author='James. R. Barlow',
+    author='James R. Barlow',
     author_email='jim@purplerock.ca',
     license='Public Domain',
     packages=['ocrmypdf'],
     keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
     classifiers=[
         "Programming Language :: Python :: 3",
-        "Development Status :: 4 - Beta",
+        "Development Status :: 5 - Production/Stable",
         "Environment :: Console",
         "Intended Audience :: End Users/Desktop",
         "Intended Audience :: Science/Research",

tests/resources/NOTE.rst

@@ -4,29 +4,31 @@ copyright reasons.
 Test files do not necessarily produce perfect (or even good) OCR
 results.
 
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| File              | Source                                                                         |
+| File                | Source                                                                         |
-+===================+================================================================================+
++=====================+================================================================================+
-| graph.pdf         | Wikimedia                                                                      |
+| graph.pdf           | Wikimedia                                                                      |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| c02-22.pdf        | Project Gutenberg: https://www.gutenberg.org/files/76/76-h/images/c02-22.jpg   |
+| c02-22.pdf          | Project Gutenberg: https://www.gutenberg.org/files/76/76-h/images/c02-22.jpg   |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| LinnSequencer.jpg | Wikimedia_                                                                     |
+| LinnSequencer.jpg   | Wikimedia_                                                                     |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| congress.jpg      | http://www.baxleystamps.com/litho/meiji/courts_1871.jpg                        |
+| congress.jpg        | http://www.baxleystamps.com/litho/meiji/courts_1871.jpg                        |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| blank.pdf         | Blank page from Adobe Illustrator CC 2015                                      |
+| blank.pdf           | Blank page from Adobe Illustrator CC 2015                                      |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| enormous.pdf      | PNG file saved to PDF using img2pdf                                            |
+| enormous.pdf        | PNG file saved to PDF using img2pdf                                            |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| invalid.pdf       | PDF file header followed by EOF marker; not valid                              |
+| invalid.pdf         | PDF file header followed by EOF marker; not valid                              |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| multipage.pdf     | several other files concatenated                                               |
+| multipage.pdf       | several other files concatenated                                               |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| skew.pdf          | skewed version of c02-22.PDF                                                   |
+| skew.pdf            | skewed version of c02-22.PDF                                                   |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
-| Test_Issue_28.pdf | file with some syntax errors                                                   |
+| Test_Issue_28.pdf   | file with some syntax errors                                                   |
-+-------------------+--------------------------------------------------------------------------------+
++---------------------+--------------------------------------------------------------------------------+
+| missing_docinfo.pdf | file missing its DocumentInfo dictionary                                       |
++---------------------+--------------------------------------------------------------------------------+ 
 
 
 .. _Wikimedia: https://upload.wikimedia.org/wikipedia/en/b/b7/LinnSequencer_hardware_MIDI_sequencer_brochure_page_2_300dpi.jpg

tests/test_main.py

@@ -114,6 +114,21 @@ def test_clean():
     check_ocrmypdf('skew.pdf', 'test_clean.pdf', '-c')
 
 
+def check_exotic_image(pdf, renderer):
+    check_ocrmypdf(
+        pdf,
+        'test_{0}_{1}.pdf'.format(pdf, renderer),
+        '-dc',
+        '--pdf-renderer', renderer)
+
+
+def test_exotic_image():
+    yield check_exotic_image, 'palette.pdf', 'hocr'
+    yield check_exotic_image, 'palette.pdf', 'tesseract'
+    yield check_exotic_image, 'cmyk.pdf', 'hocr'
+    yield check_exotic_image, 'cmyk.pdf', 'tesseract'
+
+
 def test_preserve_metadata():
     pdf_before = pypdf.PdfFileReader(_make_input('graph.pdf'))
 
@@ -139,9 +154,7 @@ def test_override_metadata():
         '--author', chinese,
         '--subject', high_unicode)
 
-    if p.returncode == ExitCode.invalid_output_pdfa:
+    assert p.returncode == ExitCode.ok
-        print("Got invalid PDF return code, as expected - JHOVE bug")
-    assert p.returncode in (ExitCode.ok, ExitCode.invalid_output_pdfa)
 
     pdf = output_file
 
@@ -308,3 +321,10 @@ def test_klingon():
     p, out, err = run_ocrmypdf_env(
         'francais.pdf', 'francais.pdf', '-l', 'klz')
     assert p.returncode == ExitCode.bad_args
+
+
+def test_missing_docinfo():
+    p, out, err = run_ocrmypdf_env(
+        'missing_docinfo.pdf', 'missing_docinfo.pdf', '-l', 'eng', '-c')
+    assert p.returncode == ExitCode.ok, err
+