Use any installed Noto font when named families lack glyphs (closes #1722)
SystemFontProvider.NOTO_FONT_PATTERNS enumerates about two dozen Noto families by name, and MultiFontManager only ever asked for those. Any script outside that list fell through to the glyphless Occulta fallback even when the correct font was installed, which is the common case on macOS: it ships around a hundred script-specific Noto faces in /System/Library/Fonts/Supplemental, almost none of which we knew how to ask for. The reporter had the fonts and still got told to install them. Add an optional GlyphSearchingFontProvider protocol (find_font_with_glyphs) implemented by SystemFontProvider, BuiltinFontProvider and ChainedFontProvider, and a new selection phase that uses it once the named families fail. The system scan enumerates every Noto face, reusing the existing variable-font/-Regular/-VF filename classification, and keeps only the font it selects so a full scan does not retain every font file on the system. Fonts found this way are remembered and retried by name for later words. Capability detection is isinstance-based so third-party providers keep working unchanged. The warning itself was also unactionable: it named neither the characters nor the script, which is why the reporter had to ask which package to install. It now identifies what it could not render, e.g. 'Ꮳ' U+13E3 CHEROKEE LETTER TSA. A word mixing scripts that no single font covers now gets a distinct message saying that installing fonts will not help, rather than sending the user after fonts they already have. Finally, the documented macOS install command was wrong: `brew install font-noto` does not exist, since Homebrew has no single Noto package, only a cask per family. The Fedora package name was also corrected, as google-noto-fonts-common ships no actual fonts.
This commit is contained in:
James R. Barlow
+19
-3
@@ -663,9 +663,25 @@ provides text shaping for proper multilingual support. These replace the
|
||||
legacy hOCR-based renderer. Install with: `pip install fpdf2 uharfbuzz`
|
||||
|
||||
**fonts-noto** (or an equivalent comprehensive font package) is recommended
|
||||
for proper text rendering, especially for non-Latin scripts. On Debian/Ubuntu:
|
||||
`apt install fonts-noto`. On Fedora: `dnf install google-noto-fonts-common`.
|
||||
On macOS with Homebrew: `brew install font-noto`.
|
||||
for proper text rendering, especially for non-Latin scripts. OCRmyPDF bundles
|
||||
a Latin font only, and discovers the rest from the fonts installed on your
|
||||
system.
|
||||
|
||||
- Debian/Ubuntu: `apt install fonts-noto`
|
||||
- Fedora: `dnf install google-noto-fonts-all`
|
||||
- macOS with Homebrew: Homebrew has no single Noto package; each family is a
|
||||
separate cask. Install at least
|
||||
`brew install --cask font-noto-sans font-noto-serif`, plus a cask per
|
||||
additional script you OCR, for example
|
||||
`brew install --cask font-noto-sans-arabic font-noto-sans-cjk`. Run
|
||||
`brew search font-noto` to list them all.
|
||||
|
||||
If OCRmyPDF warns that no installed font has glyphs for some of the text, the
|
||||
message names the characters it could not render, for example
|
||||
`'Ꮳ' U+13E3 CHEROKEE LETTER TSA`. Install the Noto font for that script — here,
|
||||
`fonts-noto-core` on Debian or `font-noto-sans-cherokee` on Homebrew. The text
|
||||
layer remains searchable and copyable either way; only its appearance when
|
||||
highlighted in a PDF viewer is affected.
|
||||
|
||||
**pypdfium2**, if present, provides fast PDF page rasterization using
|
||||
the pdfium library (the same library used by Google Chrome). It is
|
||||
|
||||
+3
-1
@@ -47,7 +47,9 @@ OCRmyPDF has the following runtime dependencies:
|
||||
**For text rendering** (expressing OCR results in PDF):
|
||||
- `fpdf2` (Python package) - Required for text layer rendering
|
||||
- `uharfbuzz` (Python package) - Required for text layer rendering
|
||||
- `font-noto` (system package) - Recommended for text layer rendering
|
||||
- Noto fonts (system package) - Recommended for text layer rendering.
|
||||
`fonts-noto` on Debian/Ubuntu, `google-noto-fonts-all` on Fedora; Homebrew
|
||||
has no single Noto package, only per-family casks such as `font-noto-sans`.
|
||||
|
||||
**Other dependencies**:
|
||||
- `unpaper` (system binary) - Optional, enables `--clean` and `--clean-final`
|
||||
|
||||
@@ -3,6 +3,27 @@
|
||||
|
||||
# v17
|
||||
|
||||
## v17.9.0
|
||||
|
||||
- OCRmyPDF now uses any Noto font installed on the system, not just the two
|
||||
dozen script families it knows by name ({issue}`1722`). Previously a document
|
||||
in, say, Cherokee or Vai was rendered with the glyphless fallback font even
|
||||
though the matching font was installed — a common situation on macOS, which
|
||||
ships around a hundred script-specific Noto faces. When the named fonts
|
||||
cannot cover a word, OCRmyPDF now searches the installed fonts for one that
|
||||
can.
|
||||
- The "no installed font has glyphs" warning now names the characters it could
|
||||
not render, with their codepoints and Unicode names, so it is clear which
|
||||
font to install. Text that mixes scripts no single font covers is now
|
||||
reported as such, instead of advising the user to install fonts they may
|
||||
already have.
|
||||
- Fixed the macOS font installation instructions, which recommended a Homebrew
|
||||
package (`font-noto`) that does not exist ({issue}`1722`). Homebrew has no
|
||||
single Noto package; each family is a separate cask. The Fedora package name
|
||||
was also corrected to `google-noto-fonts-all`.
|
||||
- Font providers may now implement the optional `GlyphSearchingFontProvider`
|
||||
protocol to participate in coverage-based font search.
|
||||
|
||||
## v17.8.1
|
||||
|
||||
- Improved the `--tesseract-pagesegmode` help text to point to
|
||||
|
||||
@@ -18,6 +18,7 @@ from ocrmypdf.font.font_provider import (
|
||||
BuiltinFontProvider,
|
||||
ChainedFontProvider,
|
||||
FontProvider,
|
||||
GlyphSearchingFontProvider,
|
||||
)
|
||||
from ocrmypdf.font.multi_font_manager import MultiFontManager
|
||||
from ocrmypdf.font.system_font_provider import SystemFontProvider
|
||||
@@ -25,6 +26,7 @@ from ocrmypdf.font.system_font_provider import SystemFontProvider
|
||||
__all__ = [
|
||||
"FontManager",
|
||||
"FontProvider",
|
||||
"GlyphSearchingFontProvider",
|
||||
"BuiltinFontProvider",
|
||||
"ChainedFontProvider",
|
||||
"MultiFontManager",
|
||||
|
||||
@@ -7,7 +7,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
from typing import Protocol, runtime_checkable
|
||||
|
||||
from ocrmypdf.font.font_manager import FontManager
|
||||
|
||||
@@ -52,6 +52,34 @@ class FontProvider(Protocol):
|
||||
...
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class GlyphSearchingFontProvider(Protocol):
|
||||
"""Optional capability: find a font by glyph coverage rather than by name.
|
||||
|
||||
A provider only knows a limited set of logical font names, but it may have
|
||||
access to many more fonts than it can name (e.g. the ~100 script-specific
|
||||
Noto faces macOS installs). Implementing this lets MultiFontManager use
|
||||
them as a last resort instead of falling back to glyphless rendering.
|
||||
|
||||
Providers that do not implement this are used as-is; the capability is
|
||||
detected at runtime with ``isinstance``.
|
||||
"""
|
||||
|
||||
def find_font_with_glyphs(self, text: str) -> tuple[str, FontManager] | None:
|
||||
"""Find a font that has glyphs for every character in text.
|
||||
|
||||
The returned name must subsequently resolve through ``get_font()``, so
|
||||
that callers can cache the selection by name.
|
||||
|
||||
Args:
|
||||
text: Text the font must fully cover
|
||||
|
||||
Returns:
|
||||
(logical font name, FontManager), or None if no font covers text
|
||||
"""
|
||||
...
|
||||
|
||||
|
||||
class BuiltinFontProvider:
|
||||
"""Font provider using builtin fonts from ocrmypdf/data directory."""
|
||||
|
||||
@@ -119,6 +147,18 @@ class BuiltinFontProvider:
|
||||
"""Get the glyphless fallback font."""
|
||||
return self._fonts['Occulta']
|
||||
|
||||
def find_font_with_glyphs(self, text: str) -> tuple[str, FontManager] | None:
|
||||
"""Find a bundled font that covers text, ignoring glyphless Occulta."""
|
||||
if not text:
|
||||
return None
|
||||
codepoints = {ord(c) for c in text}
|
||||
for name, font in self._fonts.items():
|
||||
if name == 'Occulta':
|
||||
continue
|
||||
if all(font.has_glyph(cp) for cp in codepoints):
|
||||
return name, font
|
||||
return None
|
||||
|
||||
|
||||
class ChainedFontProvider:
|
||||
"""Font provider that tries multiple providers in order.
|
||||
@@ -170,6 +210,25 @@ class ChainedFontProvider:
|
||||
result.append(name)
|
||||
return result
|
||||
|
||||
def find_font_with_glyphs(self, text: str) -> tuple[str, FontManager] | None:
|
||||
"""Ask each capable provider in turn for a font that covers text.
|
||||
|
||||
Providers that don't implement the search are skipped.
|
||||
|
||||
Args:
|
||||
text: Text the font must fully cover
|
||||
|
||||
Returns:
|
||||
(logical font name, FontManager) from the first provider with a
|
||||
match, or None if no provider found one
|
||||
"""
|
||||
for provider in self.providers:
|
||||
if not isinstance(provider, GlyphSearchingFontProvider):
|
||||
continue
|
||||
if found := provider.find_font_with_glyphs(text):
|
||||
return found
|
||||
return None
|
||||
|
||||
def get_fallback_font(self) -> FontManager:
|
||||
"""Get the glyphless fallback font.
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ language hints and glyph coverage analysis.
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
from ocrmypdf.font.font_manager import FontManager
|
||||
@@ -17,6 +18,7 @@ from ocrmypdf.font.font_provider import (
|
||||
BuiltinFontProvider,
|
||||
ChainedFontProvider,
|
||||
FontProvider,
|
||||
GlyphSearchingFontProvider,
|
||||
)
|
||||
from ocrmypdf.font.system_font_provider import SystemFontProvider
|
||||
|
||||
@@ -33,9 +35,17 @@ class MultiFontManager:
|
||||
Font selection strategy:
|
||||
1. Try language-preferred font (if language hint available)
|
||||
2. Try fallback fonts in order by glyph coverage
|
||||
3. Fall back to Occulta.ttf (glyphless fallback)
|
||||
3. Ask the provider for any installed font that covers the text
|
||||
4. Fall back to Occulta.ttf (glyphless fallback)
|
||||
"""
|
||||
|
||||
# How many uncoverable characters to name in the missing-font warning
|
||||
MAX_REPORTED_CHARS = 3
|
||||
|
||||
# How many characters of a word to look up individually when composing that
|
||||
# warning; each lookup may scan every font installed on the system
|
||||
MAX_EXAMINED_CHARS = 8
|
||||
|
||||
# Language to font mapping
|
||||
# Keys are ISO 639-2/3 codes or Tesseract language codes
|
||||
LANGUAGE_FONT_MAP = {
|
||||
@@ -173,6 +183,9 @@ class MultiFontManager:
|
||||
self._selection_cache: dict[tuple[str, str | None], str] = {}
|
||||
# Track whether we've warned about missing fonts (warn once per script)
|
||||
self._warned_scripts: set[str] = set()
|
||||
# Fonts found by glyph coverage rather than by name, tried before
|
||||
# repeating the (expensive) provider search
|
||||
self._discovered_fonts: list[str] = []
|
||||
|
||||
@property
|
||||
def fonts(self) -> dict[str, FontManager]:
|
||||
@@ -208,7 +221,8 @@ class MultiFontManager:
|
||||
Uses a hybrid approach:
|
||||
1. Language-based selection (if language hint available)
|
||||
2. Ordered fallback through available fonts by glyph coverage
|
||||
3. Final fallback to Occulta.ttf (glyphless)
|
||||
3. Provider search over every installed font, by glyph coverage
|
||||
4. Final fallback to Occulta.ttf (glyphless)
|
||||
|
||||
Args:
|
||||
word_text: The text content of the word
|
||||
@@ -233,19 +247,50 @@ class MultiFontManager:
|
||||
if result := self._try_font(preferred, word_text, cache_key):
|
||||
return result
|
||||
|
||||
# Phase 2: Try fallback fonts in order
|
||||
for font_name in self.FALLBACK_FONTS:
|
||||
# Phase 2: Try fallback fonts in order, then anything a previous
|
||||
# coverage search turned up
|
||||
for font_name in [*self.FALLBACK_FONTS, *self._discovered_fonts]:
|
||||
if font_name in tried_fonts:
|
||||
continue
|
||||
tried_fonts.add(font_name)
|
||||
if result := self._try_font(font_name, word_text, cache_key):
|
||||
return result
|
||||
|
||||
# Phase 3: Glyphless fallback (always succeeds)
|
||||
# Phase 3: Ask the provider to search every installed font. The named
|
||||
# families cover common scripts only, but systems ship many more (macOS
|
||||
# installs ~100 Noto faces), and those should be used before giving up
|
||||
# on rendering the text at all. See issue #1722.
|
||||
if found := self._search_font_by_coverage(word_text):
|
||||
font_name, font = found
|
||||
self._selection_cache[cache_key] = font_name
|
||||
return font
|
||||
|
||||
# Phase 4: Glyphless fallback (always succeeds)
|
||||
# Warn if we're falling back for non-ASCII text (likely missing font)
|
||||
self._warn_missing_font(word_text, line_language)
|
||||
self._selection_cache[cache_key] = 'Occulta'
|
||||
return self.font_provider.get_fallback_font()
|
||||
|
||||
def _search_font_by_coverage(self, text: str) -> tuple[str, FontManager] | None:
|
||||
"""Search the provider for any font covering text, if it supports it.
|
||||
|
||||
Args:
|
||||
text: Text the font must fully cover
|
||||
|
||||
Returns:
|
||||
(font name, FontManager), or None if unsupported or nothing matched
|
||||
"""
|
||||
provider = self.font_provider
|
||||
if not isinstance(provider, GlyphSearchingFontProvider):
|
||||
return None
|
||||
found = provider.find_font_with_glyphs(text)
|
||||
if found is None:
|
||||
return None
|
||||
font_name, _font = found
|
||||
if font_name not in self._discovered_fonts:
|
||||
self._discovered_fonts.append(font_name)
|
||||
return found
|
||||
|
||||
def _warn_missing_font(self, word_text: str, line_language: str | None) -> None:
|
||||
"""Warn user about missing font for non-Latin text.
|
||||
|
||||
@@ -264,26 +309,97 @@ class MultiFontManager:
|
||||
|
||||
self._warned_scripts.add(warn_key)
|
||||
|
||||
uncoverable = self._uncoverable_characters(word_text)
|
||||
if not uncoverable:
|
||||
# Every character has a font, but no single font has them all.
|
||||
# Telling the user to install fonts would be wrong advice here.
|
||||
log.warning(
|
||||
"Text mixing scripts that no single installed font covers (%r) "
|
||||
"was added as an invisible text layer: it stays searchable and "
|
||||
"copyable, but appears blank when highlighted in a PDF viewer. "
|
||||
"Installing more fonts will not help; OCRmyPDF uses one font "
|
||||
"per word.",
|
||||
word_text,
|
||||
)
|
||||
return
|
||||
|
||||
missing = self._describe_characters(uncoverable)
|
||||
if line_language and line_language in self.LANGUAGE_FONT_MAP:
|
||||
font_family = self.LANGUAGE_FONT_MAP[line_language].removesuffix('-Regular')
|
||||
log.warning(
|
||||
"No installed font has glyphs for the detected '%s' text, so "
|
||||
"it was added as an invisible text layer: it stays searchable "
|
||||
"No installed font has glyphs for the detected '%s' text (%s), "
|
||||
"so it was added as an invisible text layer: it stays searchable "
|
||||
"and copyable, but appears blank when highlighted in a PDF "
|
||||
"viewer. Install the %s font family (via your OS package "
|
||||
"manager or https://fonts.google.com/noto) for full rendering.",
|
||||
line_language,
|
||||
missing,
|
||||
font_family,
|
||||
)
|
||||
else:
|
||||
log.warning(
|
||||
"No installed font has glyphs for some of the detected text, "
|
||||
"so it was added as an invisible text layer: it stays "
|
||||
"No installed font has glyphs for some of the detected text "
|
||||
"(%s), so it was added as an invisible text layer: it stays "
|
||||
"searchable and copyable, but appears blank when highlighted "
|
||||
"in a PDF viewer. Install the matching Noto fonts "
|
||||
"(https://fonts.google.com/noto) for full rendering."
|
||||
"in a PDF viewer. Install a Noto font covering that script "
|
||||
"(https://fonts.google.com/noto) for full rendering.",
|
||||
missing,
|
||||
)
|
||||
|
||||
def _uncoverable_characters(self, word_text: str) -> list[str]:
|
||||
"""Find the characters of word_text that no installed font can render.
|
||||
|
||||
Args:
|
||||
word_text: The word that fell back to glyphless rendering
|
||||
|
||||
Returns:
|
||||
The distinct uncoverable characters, in order of first appearance,
|
||||
considering at most MAX_EXAMINED_CHARS of them
|
||||
"""
|
||||
candidates = [
|
||||
char
|
||||
for char in dict.fromkeys(word_text) # de-duplicate, keep order
|
||||
if not char.isspace() and not self._is_char_renderable(char)
|
||||
]
|
||||
# The named fonts missed these, but the provider may still have a font
|
||||
# for them, so confirm before telling the user to install anything. The
|
||||
# search walks every installed font, hence the cap on how many
|
||||
# characters we are willing to look up for one warning.
|
||||
return [
|
||||
char
|
||||
for char in candidates[: self.MAX_EXAMINED_CHARS]
|
||||
if self._search_font_by_coverage(char) is None
|
||||
]
|
||||
|
||||
def _describe_characters(self, chars: list[str]) -> str:
|
||||
"""Describe characters by codepoint and Unicode name.
|
||||
|
||||
Naming the codepoints tells the user which font to install even for
|
||||
scripts OCRmyPDF has no language mapping for, which the generic
|
||||
"install the matching Noto fonts" advice did not. See issue #1722.
|
||||
|
||||
Args:
|
||||
chars: Characters to describe
|
||||
|
||||
Returns:
|
||||
Human-readable description, truncated to MAX_REPORTED_CHARS
|
||||
"""
|
||||
described = ", ".join(
|
||||
f"{char!r} U+{ord(char):04X} {unicodedata.name(char, 'unnamed character')}"
|
||||
for char in chars[: self.MAX_REPORTED_CHARS]
|
||||
)
|
||||
if len(chars) > self.MAX_REPORTED_CHARS:
|
||||
described += f", and {len(chars) - self.MAX_REPORTED_CHARS} more"
|
||||
return described
|
||||
|
||||
def _is_char_renderable(self, char: str) -> bool:
|
||||
"""Check whether any font already known to us has a glyph for char."""
|
||||
for font_name in [*self.FALLBACK_FONTS, *self._discovered_fonts]:
|
||||
font = self.font_provider.get_font(font_name)
|
||||
if font is not None and self._has_all_glyphs(font, char):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _has_all_glyphs(self, font: FontManager, text: str) -> bool:
|
||||
"""Check if a font has glyphs for all characters in text.
|
||||
|
||||
|
||||
@@ -209,6 +209,13 @@ class SystemFontProvider:
|
||||
self._not_found: set[str] = set()
|
||||
# Cached font directories (computed lazily)
|
||||
self._font_dirs: list[Path] | None = None
|
||||
# Cached (logical name, path) of every Noto face on the system, in the
|
||||
# order the coverage search should try them (computed lazily)
|
||||
self._noto_candidates: list[tuple[str, Path]] | None = None
|
||||
# Memoized results of find_font_with_glyphs(), keyed by codepoint set
|
||||
self._coverage_cache: dict[frozenset[int], str | None] = {}
|
||||
# Font files that failed to load, so we only complain about them once
|
||||
self._unloadable: set[Path] = set()
|
||||
|
||||
def _get_platform(self) -> str:
|
||||
"""Get the current platform identifier.
|
||||
@@ -352,6 +359,145 @@ class SystemFontProvider:
|
||||
return best[1]
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _family_base(stem: str) -> str | None:
|
||||
"""Get the Noto family base a filename stem is the Regular face of.
|
||||
|
||||
Args:
|
||||
stem: Filename without extension, e.g. 'NotoSansCherokee-Regular'
|
||||
|
||||
Returns:
|
||||
The family base ('NotoSansCherokee') or None if the stem is not a
|
||||
Noto font, or is a weight/slope variant such as '-Bold' or
|
||||
'-Italic' that should not stand in for the family.
|
||||
"""
|
||||
head = stem.split('[', 1)[0] # drop variable-font axes, e.g. '[wght]'
|
||||
if head.endswith('-Regular'):
|
||||
head = head[: -len('-Regular')]
|
||||
elif head.endswith('-VF'):
|
||||
head = head[: -len('-VF')]
|
||||
elif '-' in head:
|
||||
return None
|
||||
return head if head.startswith('Noto') else None
|
||||
|
||||
@classmethod
|
||||
def _candidate_sort_key(cls, base: str) -> tuple[int, int, str]:
|
||||
"""Rank a family base for the coverage search.
|
||||
|
||||
Sans comes before serif before everything else, and plain families come
|
||||
ahead of their narrower UI and Mono cousins.
|
||||
"""
|
||||
if base.startswith('NotoSans'):
|
||||
family_rank = 0
|
||||
elif base.startswith('NotoSerif'):
|
||||
family_rank = 1
|
||||
else:
|
||||
family_rank = 2
|
||||
narrow_use = base.endswith('UI') or base.startswith('NotoSansMono')
|
||||
return (family_rank, int(narrow_use), base)
|
||||
|
||||
def _get_noto_candidates(self) -> list[tuple[str, Path]]:
|
||||
"""Enumerate every Noto family installed on the system.
|
||||
|
||||
Scans each font directory once and keeps the best-ranked file per
|
||||
family, so a family present in several directories or in several
|
||||
variants contributes a single candidate.
|
||||
|
||||
Returns:
|
||||
List of (logical font name, path) in the order to try them.
|
||||
"""
|
||||
if self._noto_candidates is not None:
|
||||
return self._noto_candidates
|
||||
|
||||
best: dict[str, tuple[int, Path]] = {}
|
||||
for font_dir in self._get_font_dirs():
|
||||
if not font_dir.exists():
|
||||
continue
|
||||
try:
|
||||
paths = sorted(font_dir.rglob('Noto*'))
|
||||
except OSError:
|
||||
# Skip directories we can't read
|
||||
continue
|
||||
for path in paths:
|
||||
if path.suffix.lower() not in self._FONT_EXTENSIONS:
|
||||
continue
|
||||
base = self._family_base(path.stem)
|
||||
if base is None:
|
||||
continue
|
||||
kind = self._classify_variant(path.stem, base)
|
||||
if kind is None:
|
||||
continue
|
||||
rank = self._VARIANT_RANK[kind]
|
||||
if base not in best or rank < best[base][0]:
|
||||
best[base] = (rank, path)
|
||||
|
||||
self._noto_candidates = [
|
||||
(f'{base}-Regular', path)
|
||||
for base, (_rank, path) in sorted(
|
||||
best.items(), key=lambda item: self._candidate_sort_key(item[0])
|
||||
)
|
||||
]
|
||||
return self._noto_candidates
|
||||
|
||||
def find_font_with_glyphs(self, text: str) -> tuple[str, FontManager] | None:
|
||||
"""Find any installed Noto font that covers every character in text.
|
||||
|
||||
``NOTO_FONT_PATTERNS`` enumerates the couple dozen scripts OCRmyPDF
|
||||
knows by name, but systems ship far more: macOS alone installs around a
|
||||
hundred script-specific Noto faces in
|
||||
``/System/Library/Fonts/Supplemental``. This is the last resort that
|
||||
makes those usable, so a document is only rendered glyphless when no
|
||||
installed font can actually cover it. See issue #1722.
|
||||
|
||||
This walks every Noto face on the system and is therefore expensive;
|
||||
results are memoized, and callers should only reach it after the named
|
||||
fonts have failed.
|
||||
|
||||
Args:
|
||||
text: Text that the returned font must fully cover
|
||||
|
||||
Returns:
|
||||
(logical font name, FontManager) of the first covering font, or
|
||||
None if nothing installed covers the text.
|
||||
"""
|
||||
if not text:
|
||||
return None
|
||||
needed = frozenset(ord(c) for c in text)
|
||||
|
||||
if needed in self._coverage_cache:
|
||||
cached_name = self._coverage_cache[needed]
|
||||
if cached_name is None:
|
||||
return None
|
||||
if cached := self._font_cache.get(cached_name):
|
||||
return cached_name, cached
|
||||
|
||||
for font_name, path in self._get_noto_candidates():
|
||||
font = self._font_cache.get(font_name)
|
||||
if font is None:
|
||||
if path in self._unloadable:
|
||||
continue
|
||||
try:
|
||||
font = FontManager(path)
|
||||
except Exception as e:
|
||||
log.debug("Skipping unreadable font %s: %s", path, e)
|
||||
self._unloadable.add(path)
|
||||
continue
|
||||
if all(font.has_glyph(cp) for cp in needed):
|
||||
# Keep only fonts we actually use; the rest are released so a
|
||||
# full scan doesn't retain every font file on the system.
|
||||
self._font_cache[font_name] = font
|
||||
self._not_found.discard(font_name)
|
||||
self._coverage_cache[needed] = font_name
|
||||
log.debug(
|
||||
"Found system font %s at %s (glyph coverage match)",
|
||||
font_name,
|
||||
path,
|
||||
)
|
||||
return font_name, font
|
||||
|
||||
self._coverage_cache[needed] = None
|
||||
return None
|
||||
|
||||
def get_font(self, font_name: str) -> FontManager | None:
|
||||
"""Get a FontManager for the named font (lazy loading).
|
||||
|
||||
|
||||
@@ -569,3 +569,117 @@ def test_missing_font_warning_explains_consequences(font_dir, caplog):
|
||||
# the text stays searchable but renders blank when highlighted.
|
||||
assert 'searchable' in msg.lower()
|
||||
assert 'highlight' in msg.lower() or 'select' in msg.lower()
|
||||
|
||||
|
||||
# --- Coverage-driven last-resort font search (#1722) ---
|
||||
|
||||
|
||||
class _FakeProviderWithSearch(_FakeFontProvider):
|
||||
"""FontProvider that can also search unlisted fonts by glyph coverage."""
|
||||
|
||||
def __init__(self, fonts, searchable):
|
||||
super().__init__(fonts)
|
||||
self._searchable = searchable
|
||||
self.search_calls: list[str] = []
|
||||
|
||||
def find_font_with_glyphs(self, text):
|
||||
self.search_calls.append(text)
|
||||
for name, font in self._searchable.items():
|
||||
hb = font.get_hb_font()
|
||||
if all(hb.get_nominal_glyph(ord(c)) for c in text):
|
||||
self._fonts[name] = font # discovered fonts become resolvable
|
||||
return name, font
|
||||
return None
|
||||
|
||||
|
||||
def test_unlisted_font_found_by_coverage_search():
|
||||
"""A script outside FALLBACK_FONTS is rendered if the font is installed."""
|
||||
searchable = {'NotoSansCherokee-Regular': _FakeFontManager('Cherokee.ttf', 'ᏣᎳᎩ')}
|
||||
provider = _FakeProviderWithSearch({}, searchable)
|
||||
manager = MultiFontManager(font_provider=provider)
|
||||
|
||||
font = manager.select_font_for_word('ᏣᎳᎩ', None)
|
||||
|
||||
assert font.font_path.name == 'Cherokee.ttf'
|
||||
assert provider.search_calls == ['ᏣᎳᎩ']
|
||||
|
||||
|
||||
def test_coverage_search_result_is_cached():
|
||||
"""The expensive coverage search runs once per distinct word."""
|
||||
searchable = {'NotoSansCherokee-Regular': _FakeFontManager('Cherokee.ttf', 'ᏣᎳᎩ')}
|
||||
provider = _FakeProviderWithSearch({}, searchable)
|
||||
manager = MultiFontManager(font_provider=provider)
|
||||
|
||||
manager.select_font_for_word('ᏣᎳᎩ', None)
|
||||
manager.select_font_for_word('ᏣᎳᎩ', None)
|
||||
|
||||
assert len(provider.search_calls) == 1
|
||||
|
||||
|
||||
def test_named_fonts_take_precedence_over_coverage_search():
|
||||
"""The coverage search is a last resort, not a substitute for named fonts."""
|
||||
fonts = {'NotoSans-Regular': _FakeFontManager('NotoSans.ttf', 'abc')}
|
||||
searchable = {'NotoSansMono-Regular': _FakeFontManager('NotoSansMono.ttf', 'abc')}
|
||||
provider = _FakeProviderWithSearch(fonts, searchable)
|
||||
manager = MultiFontManager(font_provider=provider)
|
||||
|
||||
font = manager.select_font_for_word('abc', None)
|
||||
|
||||
assert font.font_path.name == 'NotoSans.ttf'
|
||||
assert provider.search_calls == []
|
||||
|
||||
|
||||
def test_provider_without_coverage_search_still_falls_back():
|
||||
"""Providers predating find_font_with_glyphs() keep working (duck-typed)."""
|
||||
manager = MultiFontManager(font_provider=_FakeFontProvider({}))
|
||||
font = manager.select_font_for_word('ᏣᎳᎩ', None)
|
||||
assert font.font_path.name == 'Occulta.ttf'
|
||||
|
||||
|
||||
def test_missing_font_warning_names_the_missing_characters(font_dir, caplog):
|
||||
"""The warning must identify what could not be rendered (#1722).
|
||||
|
||||
The user's real question is "which font package do I install?" — naming the
|
||||
offending codepoints and their Unicode names answers it even for scripts
|
||||
OCRmyPDF has no language mapping for.
|
||||
"""
|
||||
manager = MultiFontManager(font_provider=BuiltinFontProvider(font_dir))
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
manager.select_font_for_word('ᏣᎳᎩ', None)
|
||||
|
||||
msg = caplog.text
|
||||
assert 'U+13E3' in msg # CHEROKEE LETTER TSA
|
||||
assert 'CHEROKEE' in msg.upper()
|
||||
|
||||
|
||||
def test_missing_character_warning_ignores_covered_characters(font_dir, caplog):
|
||||
"""Only the uncoverable characters are reported, not the whole word."""
|
||||
manager = MultiFontManager(font_provider=BuiltinFontProvider(font_dir))
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
manager.select_font_for_word('aᏣb', None)
|
||||
|
||||
msg = caplog.text
|
||||
assert 'U+13E3' in msg
|
||||
assert 'U+0061' not in msg # 'a' is covered by the builtin Latin font
|
||||
|
||||
|
||||
def test_mixed_script_word_warning_does_not_advise_installing_fonts(caplog):
|
||||
"""A word no single font covers is reported as such, not as a missing font.
|
||||
|
||||
Every character here has an installed font; telling the user to install
|
||||
more would be wrong advice and is exactly the confusion behind #1722.
|
||||
"""
|
||||
fonts = {'NotoSans-Regular': _FakeFontManager('NotoSans.ttf', 'ab')}
|
||||
searchable = {'NotoSansCherokee-Regular': _FakeFontManager('Cherokee.ttf', 'Ꮳ')}
|
||||
manager = MultiFontManager(font_provider=_FakeProviderWithSearch(fonts, searchable))
|
||||
|
||||
with caplog.at_level(logging.WARNING):
|
||||
font = manager.select_font_for_word('aᏣb', None)
|
||||
|
||||
assert font.font_path.name == 'Occulta.ttf'
|
||||
msg = caplog.text
|
||||
assert 'mixing scripts' in msg
|
||||
assert 'will not help' in msg
|
||||
assert 'U+' not in msg # no codepoints to blame; nothing to install
|
||||
|
||||
@@ -363,6 +363,140 @@ class TestSystemFontProviderVariableFonts:
|
||||
assert provider.get_font('NotoSansSC-Regular') is None
|
||||
|
||||
|
||||
class TestSystemFontProviderUnlistedFamilies:
|
||||
"""Test the coverage-driven search over Noto families we don't enumerate.
|
||||
|
||||
``NOTO_FONT_PATTERNS`` names only the couple dozen most common scripts, but
|
||||
macOS ships ~100 script-specific Noto fonts and Homebrew/Linux distros offer
|
||||
even more. Those fonts must still be usable when the enumerated families
|
||||
cannot cover the text. See issue #1722.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def real_font_bytes(self):
|
||||
"""Bytes of a real, loadable font covering ASCII."""
|
||||
font_path = (
|
||||
Path(__file__).parent.parent
|
||||
/ "src"
|
||||
/ "ocrmypdf"
|
||||
/ "data"
|
||||
/ "NotoSans-Regular.ttf"
|
||||
)
|
||||
if not font_path.exists():
|
||||
pytest.skip("Builtin font not available")
|
||||
return font_path.read_bytes()
|
||||
|
||||
def _provider_for(self, tmp_path, filenames, real_font_bytes):
|
||||
"""Build a provider whose only font dir is tmp_path with given files."""
|
||||
for name in filenames:
|
||||
(tmp_path / name).write_bytes(real_font_bytes)
|
||||
provider = SystemFontProvider()
|
||||
provider._font_dirs = [tmp_path]
|
||||
return provider
|
||||
|
||||
def test_finds_unlisted_family_by_coverage(self, tmp_path, real_font_bytes):
|
||||
"""A Noto family absent from NOTO_FONT_PATTERNS is still usable."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
assert 'NotoSansCherokee-Regular' not in provider.NOTO_FONT_PATTERNS
|
||||
found = provider.find_font_with_glyphs('A')
|
||||
assert found is not None
|
||||
name, font = found
|
||||
assert name == 'NotoSansCherokee-Regular'
|
||||
assert font.font_path.name == 'NotoSansCherokee-Regular.ttf'
|
||||
|
||||
def test_finds_unlisted_variable_family(self, tmp_path, real_font_bytes):
|
||||
"""Bracketed variable filenames are eligible for the coverage search."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansVithkuqi[wght].ttf'], real_font_bytes
|
||||
)
|
||||
found = provider.find_font_with_glyphs('A')
|
||||
assert found is not None
|
||||
assert found[0] == 'NotoSansVithkuqi-Regular'
|
||||
|
||||
def test_discovered_font_resolves_by_logical_name(self, tmp_path, real_font_bytes):
|
||||
"""A font found by coverage is afterwards reachable via get_font()."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
name, font = provider.find_font_with_glyphs('A')
|
||||
assert provider.get_font(name) is font
|
||||
|
||||
def test_negative_cache_does_not_block_discovery(self, tmp_path, real_font_bytes):
|
||||
"""A prior failed get_font() must not hide a later coverage match."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
assert provider.get_font('NotoSansCherokee-Regular') is None # not listed
|
||||
name, font = provider.find_font_with_glyphs('A')
|
||||
assert provider.get_font(name) is font
|
||||
|
||||
def test_skips_bold_and_italic_styles(self, tmp_path, real_font_bytes):
|
||||
"""Only Regular/variable faces are candidates, never Bold or Italic."""
|
||||
provider = self._provider_for(
|
||||
tmp_path,
|
||||
[
|
||||
'NotoSansCherokee-Bold.ttf',
|
||||
'NotoSansCherokee-Italic.ttf',
|
||||
'NotoSans-Italic[wdth,wght].ttf',
|
||||
],
|
||||
real_font_bytes,
|
||||
)
|
||||
assert provider.find_font_with_glyphs('A') is None
|
||||
|
||||
def test_ignores_non_noto_fonts(self, tmp_path, real_font_bytes):
|
||||
"""Non-Noto system fonts are not enlisted by the coverage search."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['DejaVuSans.ttf', 'Arial.ttf'], real_font_bytes
|
||||
)
|
||||
assert provider.find_font_with_glyphs('A') is None
|
||||
|
||||
def test_returns_none_when_no_font_covers_text(self, tmp_path, real_font_bytes):
|
||||
"""Text no installed font covers yields no match rather than a wrong one."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
# U+13A3 CHEROKEE LETTER O is absent from the Latin font's cmap.
|
||||
assert provider.find_font_with_glyphs('Ꭳ') is None
|
||||
|
||||
def test_empty_text_does_not_match(self, tmp_path, real_font_bytes):
|
||||
"""Empty text has nothing to cover, so no font is claimed for it."""
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
assert provider.find_font_with_glyphs('') is None
|
||||
|
||||
def test_unloadable_font_file_is_skipped(self, tmp_path, real_font_bytes):
|
||||
"""A corrupt font file does not abort the search for a usable one."""
|
||||
(tmp_path / 'NotoSansBroken-Regular.ttf').write_bytes(b'not a font')
|
||||
provider = self._provider_for(
|
||||
tmp_path, ['NotoSansCherokee-Regular.ttf'], real_font_bytes
|
||||
)
|
||||
found = provider.find_font_with_glyphs('A')
|
||||
assert found is not None
|
||||
assert found[0] == 'NotoSansCherokee-Regular'
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
'stem,expected',
|
||||
[
|
||||
('NotoSansCherokee-Regular', 'NotoSansCherokee'),
|
||||
('NotoSansCherokee[wght]', 'NotoSansCherokee'),
|
||||
('NotoSansArabic[wdth,wght]', 'NotoSansArabic'),
|
||||
('NotoSansCJKsc-VF', 'NotoSansCJKsc'),
|
||||
('NotoMusic', 'NotoMusic'),
|
||||
('NotoSansCJK-Regular', 'NotoSansCJK'),
|
||||
('NotoSans-Bold', None),
|
||||
('NotoSans-Italic[wdth,wght]', None),
|
||||
('NotoSans-SemiCondensedBlackItalic', None),
|
||||
('DejaVuSans-Regular', None),
|
||||
],
|
||||
)
|
||||
def test_family_base_parsing(self, stem, expected):
|
||||
"""Filename stems map to family bases, rejecting non-Regular styles."""
|
||||
assert SystemFontProvider._family_base(stem) == expected
|
||||
|
||||
|
||||
# --- ChainedFontProvider Tests ---
|
||||
|
||||
|
||||
@@ -507,3 +641,49 @@ class TestChainedFontProviderIntegration:
|
||||
|
||||
# Chain should have at least as many fonts as builtin
|
||||
assert chain_fonts >= builtin_fonts
|
||||
|
||||
|
||||
class TestChainedFontProviderCoverageSearch:
|
||||
"""Test that the chain delegates the coverage search to its members."""
|
||||
|
||||
class _Searchable:
|
||||
"""Provider stub that reports one findable font."""
|
||||
|
||||
def __init__(self, result):
|
||||
self.result = result
|
||||
self.calls = 0
|
||||
|
||||
def get_font(self, name):
|
||||
return None
|
||||
|
||||
def get_available_fonts(self):
|
||||
return []
|
||||
|
||||
def get_fallback_font(self):
|
||||
raise NotImplementedError
|
||||
|
||||
def find_font_with_glyphs(self, text):
|
||||
self.calls += 1
|
||||
return self.result
|
||||
|
||||
def test_delegates_to_first_provider_that_finds_a_font(self):
|
||||
"""The first provider with a match wins; later ones are not consulted."""
|
||||
first = self._Searchable(('NotoSansX-Regular', MagicMock()))
|
||||
second = self._Searchable(('NotoSansY-Regular', MagicMock()))
|
||||
chain = ChainedFontProvider([first, second])
|
||||
|
||||
assert chain.find_font_with_glyphs('x')[0] == 'NotoSansX-Regular'
|
||||
assert second.calls == 0
|
||||
|
||||
def test_skips_providers_without_the_capability(self):
|
||||
"""Providers lacking find_font_with_glyphs() are skipped, not fatal."""
|
||||
legacy = MagicMock(spec=['get_font', 'get_available_fonts'])
|
||||
searchable = self._Searchable(('NotoSansX-Regular', MagicMock()))
|
||||
chain = ChainedFontProvider([legacy, searchable])
|
||||
|
||||
assert chain.find_font_with_glyphs('x')[0] == 'NotoSansX-Regular'
|
||||
|
||||
def test_returns_none_when_nothing_matches(self):
|
||||
"""No provider matching yields None so the caller can use Occulta."""
|
||||
chain = ChainedFontProvider([self._Searchable(None)])
|
||||
assert chain.find_font_with_glyphs('x') is None
|
||||
|
||||
Reference in New Issue
Block a user