Robust PDF Text Extraction

Problem

Standard file reading tools (e.g.,

read_file

) often fail to extract text from PDF documents. Instead of returning parsed text, they may return:

• Raw binary data
• Base64 encoded images
• Garbled characters or null bytes

This occurs because PDFs are complex binary formats, not plain text files. Attempts to parse them using general-purpose Python libraries (like PyMuPDF) in sandboxed environments may also fail due to missing dependencies or environment restrictions.

Solution

Use the

pdftotext

command-line utility (part of

poppler-utils

) via

run_shell

. This tool is commonly pre-installed in Linux environments and reliably extracts text content from PDFs.

Procedure

1. Detect Extraction Failure

When attempting to read a PDF:

• Check the content returned by

  read_file

  .
• If the content contains null bytes (

  \x00

  ), appears as base64, or is clearly binary/garbled, assume standard reading has failed.

2. Execute pdftotext

Run the following shell command using

run_shell

:

pdftotext -layout -nopgbrk <file_path> -

• -layout

  : Maintains the physical layout of the text (optional but recommended).

• -nopgbrk

  : Prevents inserting form feed characters between pages.

• -

  : Outputs content to stdout instead of creating a new file.

3. Parse Output

Capture the stdout from the shell command. This string is the extracted text.

Example Usage

Scenario: You need to read

document.pdf

.

Step 1: Attempt standard read

content = read_file("document.pdf")
if "\x00" in content or not content.strip():
    # Fallback needed
    pass

Step 2: Fallback to shell

result = run_shell("pdftotext -layout -nopgbrk document.pdf -")
text = result.stdout

Prerequisites

• The environment must have

  pdftotext

  installed (usually via

  poppler-utils

  ).
• If

  pdftotext

  is not found, attempt to install it (

  apt-get install poppler-utils

  ) if permissions allow, or notify the user.

Benefits

• Reliability: Bypasses Python library dependency issues in sandboxes.
• Speed: Command-line tools are often faster than loading heavy Python libraries.
• Compatibility: Works consistently across most Linux-based agent environments.