Merge branch 'master' into develop

This commit is contained in:
James R. Barlow
2015-09-14 00:34:21 -07:00
17 changed files with 3356 additions and 68 deletions
+41
View File
@@ -0,0 +1,41 @@
language: python
cache:
directories:
- $HOME/.cache/pip
- $HOME/.ccache
- tarballs
python:
- 3.4
before_install:
# Ubuntu packages
- sudo add-apt-repository ppa:evl.ms/precise -y # for Ghostscript 9.15
- sudo add-apt-repository ppa:lyrasis/precise-backports -y # for Tesseract 3.03
- sudo add-apt-repository ppa:b-eltzner/qpdfview-exp -y # for QPDF 5
- sudo add-apt-repository ppa:itachi-san/ffmpeg -y # for libav 11.2 (for unpaper)
- sudo apt-get update -qq # must go after all add-apt-repo
- sudo apt-get install -y ghostscript tesseract-ocr tesseract-ocr-deu tesseract-ocr-eng tesseract-ocr-fra qpdf poppler-utils gcc libavformat-dev libavcodec-dev libavutil-dev automake make pkg-config xsltproc
# pip
- pip install --upgrade pip
# Download, make and install unpaper (using ccache)
- mkdir -p tarballs
- "[ -f tarballs/unpaper-6.1.tar.xz ] || wget -q https://www.flameeyes.eu/files/unpaper-6.1.tar.xz -O tarballs/unpaper-6.1.tar.xz"
- tar -xvf tarballs/unpaper-6.1.tar.xz
- export PATH="/usr/lib/ccache:$PATH"
- pushd unpaper-6.1 && ./configure --prefix=/usr && make -j && sudo make install && popd
install:
- pip install -r requirements.txt
- pip install -r test_requirements.txt
script:
- python setup.py clean
- python setup.py install
- py.test
os:
- linux
+3 -2
View File
@@ -44,12 +44,13 @@ RUN apt-get install -y \
xsltproc
WORKDIR /root
RUN wget https://github.com/Flameeyes/unpaper/archive/unpaper-6.1.tar.gz
RUN wget -q https://github.com/Flameeyes/unpaper/archive/unpaper-6.1.tar.gz
RUN tar xf unpaper-6.1.tar.gz
WORKDIR /root/unpaper-unpaper-6.1
RUN autoreconf -i
RUN ./configure CFLAGS="-O2 -march=native -pipe -flto"
RUN make -j install
WORKDIR /
RUN apt-get remove -y \
gcc \
@@ -59,7 +60,7 @@ RUN apt-get remove -y \
xsltproc \
make
RUN apt-get autoremove -y && apt-get clean -y
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /root/*
# Set up a Python virtualenv and take all of the system packages, so we can
# rely on the platform packages rather than importing GCC and compiling them
+2
View File
@@ -1 +1,3 @@
recursive-exclude tests/output *
include requirements.txt
include test_requirements.txt
+115 -25
View File
@@ -10,23 +10,25 @@ Main features
- Generates a searchable
`PDF/A <https://en.wikipedia.org/?title=PDF/A>`__ file from a regular PDF
only containing images
- Places OCRed text accurately below the image to ease copy / paste
- Places OCR text accurately below the image to ease copy / paste
- Keeps the exact resolution of the original embedded images
- or if requested oversamples the images before OCRing so as to get
better results
- When possible, copies input images directly to output without transcoding them,
- When possible, copies input images directly to output without transcoding,
to preserve image quality
- Keeps file size about the same
- If requested deskews and/or cleans the image before performing OCR
- Validates input and output files
- Provides debug mode to enable easy verification of the OCR results
- Processes several pages in parallel when more than one CPU core is
- Processes pages in parallel when more than one CPU core is
available
- Uses Tesseract OCR engine
- Uses `Tesseract OCR <https://github.com/tesseract-ocr/tesseract>`__ engine
- Supports the `39 languages <https://code.google.com/p/tesseract-ocr/downloads/list>`__ recognized by Tesseract
- Battle-tested on thousands of PDFs, a test suite and continuous integration
For details: please consult the `release notes <RELEASE_NOTES.rst>`__
For details: please consult the `release notes <RELEASE_NOTES.rst>`__.
Motivation
----------
@@ -48,22 +50,67 @@ as an inspiration)
Installation
------------
Download OCRmyPDF here: https://github.com/fritz-hh/OCRmyPDF/releases
Download OCRmyPDF here: https://github.com/jbarlow83/OCRmyPDF/releases
You can install it to a Python virtual environment or system-wide.
Installing the Docker container
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Installing dependencies on Mac OS X Yosemite
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For many users, installing the Docker container will be easier than installing all of OCRmyPDF's dependencies. For Windows, it is the only option.
If it's not already present, `install Homebrew <http://brew.sh/>`__
If you have `Docker <https://docs.docker.com/>`__ installed on your system, you can install
a Docker container of the latest release.
Follow the Docker installation instructions for your platform. If you can run this command
successfully, your system is ready to download and execute the image::
docker run hello-world
OCRmyPDF will use all available CPU cores. By default, the VirtualBox machine instance on Windows and OS X has only a single CPU core enabled. Use the VirtualBox Manager to determine the name of your Docker container host, and then follow these optional steps to enable multiple CPUs::
# Optional
docker-machine stop "yourVM"
VBoxManage modifyvm "yourVM" --cpus 2 # or whatever number of core is desired
docker-machine start "yourVM"
eval $(docker-machine env "yourVM")
Assuming you have a Docker engine running somewhere, you can run these commands to download
the image::
docker pull jbarlow83/ocrmypdf
Then tag it to give a more convenient name, just ocrmypdf::
docker tag jbarlow83/ocrmypdf ocrmypdf
You can then run using the command::
docker run ocrmypdf --help
To execute the OCRmyPDF on a local file, you must `provide a writable volume to the Docker image <https://docs.docker.com/userguide/dockervolumes/>`__, such as this in this template::
docker run -v "$(pwd):/home/docker" <other docker arguments> ocrmypdf <your arguments to ocrmypdf>
In this worked example, the current working directory contains an input file called ``test.pdf`` and the output will go to ``output.pdf``::
docker run -v "$(pwd):/home/docker" ocrmypdf --skip-text test.pdf output.pdf
Note that ``ocrmypdf`` has its own separate ``-v VERBOSITYLEVEL`` argument to control debug verbosity. All Docker arguments should before the ``ocrmypdf`` container name and all arguments to ``ocrmypdf`` should be listed after.
Installing on Mac OS X Yosemite
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These instructions probably work on all Mac OS X versions later than 10.7 (Lion), but were only tested
on Yosemite.
If it's not already present, `install Homebrew <http://brew.sh/>`__.
Update Homebrew::
brew update
brew upgrade
Install the required Homebrew packages, if any are missing::
Install or upgrade the required Homebrew packages, if any are missing::
brew install libpng openjpeg jbig2dec # image libraries
brew install qpdf
@@ -78,20 +125,31 @@ It is also recommended that install Pillow and confirm it can read and write JPE
pip3 install --upgrade pip
pip3 install --upgrade pillow
To test that your dependencies are working, try this command::
Sometimes, the Python imaging library (Pillow) can end up being compiled and installed without support for JPEG and PNG files. (Arguably, this is an unfixed bug in Pillow's installer.) To confirm that Pillow is compiled correctly and can access JPEG and PNG files, try this command::
python3 -c "from PIL import Image; im = Image.new('1', (1, 1)); im.save('test.png'); im.save('test.jpg')"
If you have trouble getting Pillow to access JPEG and PNG files, `review the installation instructions <https://pillow.readthedocs.org/installation.html>`__.
Installing dependencies on Ubuntu 14.04 LTS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can then install OCRmyPDF from PyPI::
pip3 install ocrmypdf
The command line program should now be available::
ocrmypdf --help
Installing on Ubuntu 14.04 LTS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Installing on Ubuntu 14.04 LTS (trusty) is more difficult than other options, because of certain bugs in Python package installation.
Update apt-get::
sudo apt-get update
sudo apt-get upgrade
Install dependencies::
Install system dependencies::
sudo apt-get install \
zlib1g-dev \
@@ -104,32 +162,64 @@ Install dependencies::
python3-pil \
python3-pytest \
python3-reportlab
If you wish install OCRmyPDF to the system Python, then install as follows (note this installs new packages
into your system Python, which could interfere with other programs)::
sudo pip3 install ocrmypdf
If you wish to install OCRmyPDF to a virtual environment to isolate system Python from modified, you can
follow these steps. This includes a workaround `for a known, unresolved issue in Ubuntu 14.04's ensurepip
package <http://www.thefourtheye.in/2014/12/Python-venv-problem-with-ensurepip-in-Ubuntu.html>`__::
sudo apt-get install python3-venv
python3 -m venv venv-ocrmypdf --without-pip
source venv-ocrmypdf/bin/activate
wget -O - -o /dev/null https://bootstrap.pypa.io/get-pip.py | python
deactivate
pyvenv --system-site-packages venv-ocrmypdf
source venv-ocrmypdf/bin/activate
pip install ocrmypdf
Ubuntu 14.04 only installs ``unpaper`` version 0.4.2, which is not supported by OCRmyPDF because it is produces invalid output. This program is an optional dependency, and provides page deskewing and cleaning. See `Dockerfile <Dockerfile>`__ for an example of how to building unpaper 6.1 from source. If you choose to install unpaper later, OCRmyPDF will use the foremost version on the system PATH.
Installing HEAD revision from sources
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
To install the HEAD revision from sources in development mode::
If you have ``git`` and ``python3.4`` installed, you can install from source. When the ``pip`` installer runs,
it will alert you if dependencies are missing.
git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
First, clone the HEAD revision::
git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
cd OCRmyPDF
To install the HEAD revision from sources::
pip3 install .
Or, to install in `development mode <https://pythonhosted.org/setuptools/setuptools.html#development-mode>`__,
allowing customization of OCRmyPDF, use the ``-e`` flag::
pip3 install -e .
On certain Linux/UNIX platforms such as Ubuntu, you may need to use
On certain Linux distributions such as Ubuntu, you may need to use
run the install command as superuser::
sudo pip3 install -e .
sudo pip3 install [-e] .
Note that this will alter your system's Python distribution. If you prefer
to not install as superuser, you can install the package in a Python virtual environment::
git clone -b master https://github.com/fritz-hh/OCRmyPDF.git
git clone -b master https://github.com/jbarlow83/OCRmyPDF.git
pyvenv venv
source venv/bin/activate
cd OCRmyPDF
pip3 install -e .
pip3 install .
If your platform does not have ``pip3``, make sure that Python 3.4+ and the `pip`
package are installed.
However, ``ocrmypdf`` will only be accessible on the system PATH after
you activate the virtual environment.
To run the program::
@@ -147,10 +237,10 @@ In case you detect an issue, please:
- Check if your issue is already known
- If no problem report exists on github, please create one here:
https://github.com/fritz-hh/OCRmyPDF/issues
https://github.com/jbarlow83/OCRmyPDF/issues
- Describe your problem thoroughly
- Append the console output of the script when running the debug mode
(-v 1 option)
(``-v 1`` option)
- If possible provide your input PDF file as well as the content of the
temporary folder (using a file sharing service like
www.file-upload.net)
+29 -3
View File
@@ -3,7 +3,7 @@ RELEASE NOTES
Please always read this file before installing the package
Download software here: https://github.com/fritz-hh/OCRmyPDF/tags
Download software here: https://github.com/jbarlow83/OCRmyPDF/tags
v3.0:
=====
@@ -21,18 +21,23 @@ New features
- PDF metadata (title, author, keywords) are now transferred to the
output PDF
- PDF metadata can also be set from the command line (``--title``, etc.)
- Automatic repairs malformed input PDFs if possible
- Added test cases to confirm everything is working
- Added option to skip extremely large pages that take too long to OCR and are
often not OCRable (e.g. large scanned maps or diagrams); other pages are still
processed (``--skip-big``)
- Added option to kill Tesseract OCR process if it seems to be taking too long on
a page, while still processing other pages (``--tesseract-timeout``)
- Less common colorspaces (CMYK, palette) are now supported by conversion to RGB
- Multiple images on the same PDF page are now supported
Changes
-------
- New, robust rewrite in Python 3.4+ with ruffus_ pipelines
- Now uses Ghostscript 9.14's improved color conversion model to preserve PDF colors
- OCR text is now rendered in the PDF as invisible text. Previous versions of OCRmyPDF
incorrectly rendered visible text with an image on top.
- All "tasks" in the pipeline can be executed in parallel on any
available CPUs, increasing performance
- The ``-o DPI`` argument has been phased out, in favor of ``--oversample DPI``, in
@@ -67,6 +72,21 @@ Changes
Release candidates
------------------
- rc9:
- fix issue #118: report error if ghostscript iccprofiles are missing
- fixed another issue related to #111: PDF rasterized to palette file
- add support image files with a palette
- don't try to validate PDF file after an exception occurs
- rc8:
- fix issue #111: exception thrown if PDF is missing DocumentInfo dictionary
- rc7:
- fix error when installing direct from pip, "no such file 'requirements.txt'"
- rc6:
- dropped libxml2 (Python lxml) since Python 3's internal XML parser is sufficient
@@ -122,12 +142,18 @@ Fixes
- Handling of filenames containing spaces: fixed
Notes
-----
Notes and known issues
----------------------
- Some dependencies may work with lower versions than tested, so try
overriding dependencies if they are "in the way" to see if they work.
- ``--pdf-renderer tesseract`` will output files with an incorrect page size in Tesseract 3.03,
due to a bug in Tesseract.
- PDF files containing "inline images" are not supported and won't be for the 3.0 release. Scanned
images almost never contain inline images.
v2.2-stable (2014-09-29):
=========================
+3 -1
View File
@@ -10,7 +10,9 @@ def rasterize_pdf(input_file, output_file, xres, yres, raster_device, log):
with NamedTemporaryFile(delete=True) as tmp:
args_gs = [
'gs',
'-dBATCH', '-dNOPAUSE',
'-dQUIET',
'-dBATCH',
'-dNOPAUSE',
'-sDEVICE=%s' % raster_device,
'-o', tmp.name,
'-r{0}x{1}'.format(str(xres), str(yres)),
+13 -5
View File
@@ -103,7 +103,7 @@ check_pil_encoder('zlib', 'PNG')
parser = cmdline.get_argparse(
prog="ocrmypdf",
description="Generate searchable PDF file from an image-only PDF file.",
version='3.0rc6',
version='3.0',
fromfile_prefix_chars='@',
ignored_args=[
'touch_files_only', 'recreate_database', 'checksum_file_name',
@@ -475,10 +475,15 @@ def rasterize_with_ghostscript(
if all(image['comp'] == 1 for image in pageinfo['images']):
if all(image['bpc'] == 1 for image in pageinfo['images']):
device = 'pngmono'
elif not any(image['color'] == 'color'
for image in pageinfo['images']):
elif all(image['bpc'] > 1 and image['color'] == 'index'
for image in pageinfo['images']):
device = 'png256'
elif all(image['bpc'] > 1 and image['color'] == 'gray'
for image in pageinfo['images']):
device = 'pnggray'
log.debug("Rendering {0} with {1}".format(
os.path.basename(input_file), device))
xres = max(pageinfo['xres'], options.oversample or 0)
yres = max(pageinfo['yres'], options.oversample or 0)
@@ -729,11 +734,13 @@ def generate_postscript_stub(
pdf = pypdf.PdfFileReader(input_file)
def from_document_info(key):
# pdf.documentInfo.get() DOES NOT work as expected
# pdf.documentInfo.get() DOES NOT behave as expected for a dict-like
# object, so call with precautions. TypeError may occur if the PDF
# is missing the optional document info section.
try:
s = pdf.documentInfo[key]
return str(s)
except KeyError:
except (KeyError, TypeError):
return ''
pdfmark = {
@@ -892,6 +899,7 @@ def run_pipeline():
return eval(
exc_value,
{'ExitCode': ExitCode}, {'exc_value': exc_value})
return ExitCode.other_error
if not validate_pdfa(options.output_file, _log):
_log.warning('Output file: The generated PDF/A file is INVALID')
+1
View File
@@ -34,6 +34,7 @@ FRIENDLY_COMP = {
'rgb': 3,
'cmyk': 4,
'lab': 3,
'index': 1
}
+2
View File
@@ -115,6 +115,8 @@ def _get_postscript_icc_path():
if os.path.exists(path):
return path
raise FileNotFoundError("Could not find Ghostscript's iccprofiles")
def generate_pdfa_def(target_filename, pdfmark, icc='sRGB'):
if icc == 'sRGB':
+2
View File
@@ -11,6 +11,7 @@ import shutil
import pytest
import img2pdf
from pkg_resources import Requirement, resource_filename
import pytest
req = Requirement.parse('ocrmypdf')
@@ -99,6 +100,7 @@ def test_single_page_inline_image():
pageinfo.pdf_get_all_pageinfo(filename)
@pytest.mark.skipif(True, reason="need to fix packaging error")
def test_jpeg():
filename = resource_filename(req, 'tests/resources/c02-22.pdf')
+20 -2
View File
@@ -8,6 +8,7 @@ from tempfile import NamedTemporaryFile
import sys
import os
from functools import lru_cache
from . import ExitCode
@lru_cache(maxsize=1)
@@ -38,10 +39,27 @@ def run(input_file, output_file, dpi, log, mode_args):
] + mode_args
SUFFIXES = {'1': '.pbm', 'L': '.pgm', 'RGB': '.ppm'}
suffix = ''
im = Image.open(input_file)
suffix = SUFFIXES[im.mode]
if im.mode not in SUFFIXES.keys():
log.info("Converting image to other colorspace")
try:
if im.mode == 'P' and len(im.getcolors()) == 2:
im = im.convert(mode='1')
else:
im = im.convert(mode='RGB')
except IOError:
log.error(
"Could not convert image with type " + im.mode)
sys.exit(ExitCode.missing_dependency)
try:
suffix = SUFFIXES[im.mode]
except KeyError:
log.error(
"Failed to convert image to a supported format.")
sys.exit(ExitCode.missing_dependency)
with NamedTemporaryFile(suffix=suffix) as input_pnm, \
NamedTemporaryFile(suffix=suffix, mode="r+b") as output_pnm:
im.save(input_pnm, format='PPM')
+4 -4
View File
@@ -179,17 +179,17 @@ tests_require = open('test_requirements.txt').read().splitlines()
setup(
name='ocrmypdf',
version='3.0rc6', # also update: release notes, main.py
version='3.0', # also update: release notes, main.py
description='OCRmyPDF adds an OCR text layer to scanned PDF files, allowing them to be searched',
url='https://github.com/fritz-hh/OCRmyPDF',
author='James. R. Barlow',
url='https://github.com/jbarlow83/OCRmyPDF',
author='James R. Barlow',
author_email='jim@purplerock.ca',
license='Public Domain',
packages=['ocrmypdf'],
keywords=['PDF', 'OCR', 'optical character recognition', 'PDF/A', 'scanning'],
classifiers=[
"Programming Language :: Python :: 3",
"Development Status :: 4 - Beta",
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"Intended Audience :: End Users/Desktop",
"Intended Audience :: Science/Research",
+25 -23
View File
@@ -4,29 +4,31 @@ copyright reasons.
Test files do not necessarily produce perfect (or even good) OCR
results.
+-------------------+--------------------------------------------------------------------------------+
| File | Source |
+===================+================================================================================+
| graph.pdf | Wikimedia |
+-------------------+--------------------------------------------------------------------------------+
| c02-22.pdf | Project Gutenberg: https://www.gutenberg.org/files/76/76-h/images/c02-22.jpg |
+-------------------+--------------------------------------------------------------------------------+
| LinnSequencer.jpg | Wikimedia_ |
+-------------------+--------------------------------------------------------------------------------+
| congress.jpg | http://www.baxleystamps.com/litho/meiji/courts_1871.jpg |
+-------------------+--------------------------------------------------------------------------------+
| blank.pdf | Blank page from Adobe Illustrator CC 2015 |
+-------------------+--------------------------------------------------------------------------------+
| enormous.pdf | PNG file saved to PDF using img2pdf |
+-------------------+--------------------------------------------------------------------------------+
| invalid.pdf | PDF file header followed by EOF marker; not valid |
+-------------------+--------------------------------------------------------------------------------+
| multipage.pdf | several other files concatenated |
+-------------------+--------------------------------------------------------------------------------+
| skew.pdf | skewed version of c02-22.PDF |
+-------------------+--------------------------------------------------------------------------------+
| Test_Issue_28.pdf | file with some syntax errors |
+-------------------+--------------------------------------------------------------------------------+
+---------------------+--------------------------------------------------------------------------------+
| File | Source |
+=====================+================================================================================+
| graph.pdf | Wikimedia |
+---------------------+--------------------------------------------------------------------------------+
| c02-22.pdf | Project Gutenberg: https://www.gutenberg.org/files/76/76-h/images/c02-22.jpg |
+---------------------+--------------------------------------------------------------------------------+
| LinnSequencer.jpg | Wikimedia_ |
+---------------------+--------------------------------------------------------------------------------+
| congress.jpg | http://www.baxleystamps.com/litho/meiji/courts_1871.jpg |
+---------------------+--------------------------------------------------------------------------------+
| blank.pdf | Blank page from Adobe Illustrator CC 2015 |
+---------------------+--------------------------------------------------------------------------------+
| enormous.pdf | PNG file saved to PDF using img2pdf |
+---------------------+--------------------------------------------------------------------------------+
| invalid.pdf | PDF file header followed by EOF marker; not valid |
+---------------------+--------------------------------------------------------------------------------+
| multipage.pdf | several other files concatenated |
+---------------------+--------------------------------------------------------------------------------+
| skew.pdf | skewed version of c02-22.PDF |
+---------------------+--------------------------------------------------------------------------------+
| Test_Issue_28.pdf | file with some syntax errors |
+---------------------+--------------------------------------------------------------------------------+
| missing_docinfo.pdf | file missing its DocumentInfo dictionary |
+---------------------+--------------------------------------------------------------------------------+
.. _Wikimedia: https://upload.wikimedia.org/wikipedia/en/b/b7/LinnSequencer_hardware_MIDI_sequencer_brochure_page_2_300dpi.jpg
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
+23 -3
View File
@@ -114,6 +114,21 @@ def test_clean():
check_ocrmypdf('skew.pdf', 'test_clean.pdf', '-c')
def check_exotic_image(pdf, renderer):
check_ocrmypdf(
pdf,
'test_{0}_{1}.pdf'.format(pdf, renderer),
'-dc',
'--pdf-renderer', renderer)
def test_exotic_image():
yield check_exotic_image, 'palette.pdf', 'hocr'
yield check_exotic_image, 'palette.pdf', 'tesseract'
yield check_exotic_image, 'cmyk.pdf', 'hocr'
yield check_exotic_image, 'cmyk.pdf', 'tesseract'
def test_preserve_metadata():
pdf_before = pypdf.PdfFileReader(_make_input('graph.pdf'))
@@ -139,9 +154,7 @@ def test_override_metadata():
'--author', chinese,
'--subject', high_unicode)
if p.returncode == ExitCode.invalid_output_pdfa:
print("Got invalid PDF return code, as expected - JHOVE bug")
assert p.returncode in (ExitCode.ok, ExitCode.invalid_output_pdfa)
assert p.returncode == ExitCode.ok
pdf = output_file
@@ -308,3 +321,10 @@ def test_klingon():
p, out, err = run_ocrmypdf_env(
'francais.pdf', 'francais.pdf', '-l', 'klz')
assert p.returncode == ExitCode.bad_args
def test_missing_docinfo():
p, out, err = run_ocrmypdf_env(
'missing_docinfo.pdf', 'missing_docinfo.pdf', '-l', 'eng', '-c')
assert p.returncode == ExitCode.ok, err