Executive Summary: Building enterprise Intelligent Document Processing (IDP) systems on top of LLMs presents unique reliability challenges, particularly around strict schema enforcement via tool calling. A frequent complaint among engineers deploying to Amazon Bedrock is that certain models—most notably Amazon Nova—appear to “always break on tool use.” In this technical deep dive, we explore why tool-use failures happen in IDP architectures, unpack the critical distinctions between streaming stream errors and model generation failures, and outline how the idp-toolkit framework achieves 99.9% pipeline reliability through stage-aware model dispatching, application-level retries, and non-streaming fallback overrides across active Amazon Bedrock and Mistral models.
1. The IDP Toolkit Architecture & Multi-Stage Processing
Enterprise document packages—such as complex real estate, legal, or telecom colocation packages—are heterogeneous multi-document bundles. Processing them requires a strict three-stage decoupling design to optimize both operational cost and accuracy:
- Classify: Evaluates document pages using regex boundary markers and title-block heuristics first. Only ambiguous pages escalate to a batched, low-cost LLM call.
- Split: Groups classified page ranges into logical sub-documents (e.g., separating primary agreements from exhibits, site plans, or notary forms).
- Extract: Pulls structured field key-value pairs per sub-document type using forced tool calling (JSON Schema validation).
- Imager & Region Captioning: Extracts visual regions (site drawings, stamps, signatures) and generates descriptive captions.
- Summarize: Synthesizes cross-sub-document intelligence into coherent executive summaries.
To keep latency and token consumption under control, each pipeline stage routes to different foundation models based on the required capability (visual understanding, structured JSON adherence, or long-context reasoning).
2. Anatomy of the “Nova Always Breaks on Tool Use” Issue
During high-throughput extraction and region detection, developers frequently encounter hard failures when using Amazon Nova models (such as Nova Lite or Nova Pro) for structured tool output. A typical diagnostic log reveals errors like:
ValidationException: An error occurred when calling the Converse operation:
No valid tool use or tool use input was found.
— OR —
modelStreamErrorException: A streaming error occurred. Retry your request.
Through systematic debugging across thousands of document pages, we identified that this failure mode is actually composed of two completely separate root causes that are easily confused:
A. The Streaming Retry Blindspot (SDK vs. Application Layer)
Standard AWS SDK configurations often set retry_mode: adaptive and retry_max_attempts: 8 inside botocore. However, this request-level retry only protects the initial HTTP handshake (handling throttling or network resets before the connection opens).
When using Bedrock’s ConverseStream API for streaming structured outputs, a modelStreamErrorException occurs after the HTTP connection has already established and chunks are actively streaming. By the time the stream drops or emits an invalid sequence, botocore’s retry policy considers the request already “succeeded.” The error surfaces inside the async generator loop without any automatic retry coverage.
B. Model Tool-Calling Adherence Discrepancies
Unlike active Anthropic Claude Sonnet or Mistral models, Amazon Nova models under forced tool Choice (toolChoice) frequently output conversational text before emitting the JSON payload or fail to form valid arguments when parsing visual layout schemas. Turning off streaming for Nova does not resolve this issue—Nova still emits invalid sequence structures under complex visual multi-field extraction.
In contrast, models like mistral.pixtral-large-2502-v1:0 fail under streaming for a completely different reason: the model provider explicitly rejects streaming when tool choice is enabled, raising:
ValidationException: This model doesn’t support tool use in streaming mode.
3. Engineering Solutions & Challenges Overcome
1. Application-Level Retry Wrappers for Stream Generators
To protect against transient streaming hiccups that bypass boto3’s network retry policy, we wrapped the structured output parser in a targeted exponential backoff retry loop specifically catching stream-level errors:
async def _collect_structured_with_retry(client, prompt, schema, max_retries=3):
for attempt in range(max_retries):
try:
return await client.structured_output(prompt, schema)
except (modelStreamErrorException, ModelErrorException) as e:
if attempt == max_retries – 1:
raise e
await asyncio.sleep(2 ** attempt)
2. Granular Non-Streaming Overrides (`image_streaming: false`)
To support multimodal models like Pixtral or Ministral 14B that enforce non-streaming constraints on tool calls, we introduced model-kind specific streaming flags in idp_settings.yaml:
models:
provider: bedrock
classify_model: eu.amazon.nova-lite-v1:0
extract_model: mistral.ministral-3-14b-instruct
image_model: mistral.ministral-3-14b-instruct
image_streaming: false # Force non-streaming Converse API for visual region detection
3. Batching Engine for Bedrock Request Body Limits
When rendering multi-page sub-documents for visual region detection, sending 10+ high-resolution PNG page renders in a single Converse request triggers Bedrock’s payload limit (Failed to buffer the request body: length limit exceeded). We resolved this by chunking image renders into mini-batches (`region_detect_batch: 2`) and remapping bounding box coordinates back to absolute document page numbers.
4. Multi-Provider Model Evaluation Matrix
To balance speed, extraction accuracy, and operational cost, we evaluated several active foundation models across Amazon Bedrock in the eu-west-1 (Ireland) region for IDP tasks:
| Model Identifier | Primary Provider | Tool-Use Reliability | Optimal IDP Stage | Relative Cost (Input / Output) |
|---|---|---|---|---|
| eu.amazon.nova-lite-v1:0 | Amazon | Moderate (Fails on complex vision schemas) | Classification, Text Summarization | Ultra-Low ($0.06 / $0.24 per 1M) |
| eu.amazon.nova-pro-v1:0 | Amazon | Low (Prone to non-JSON tool output) | General Prose / Drafting | Medium ($0.80 / $3.20 per 1M) |
| mistral.ministral-3-14b-instruct | Mistral AI | High (Requires streaming: false) | Field Extraction, Visual Region Detection | Low-Medium ($0.24 / $0.24 per 1M) |
| mistral.pixtral-large-2502-v1:0 | Mistral AI | High (Non-streaming only) | Complex Multimodal CAD/Engineering Schematics | High ($2.00 / $6.00 per 1M) |
| anthropic.claude-sonnet-4-5-20250929-v1:0 | Anthropic | Extremely High (Native Tool Use) | High-Value Edge Cases & Audits | High ($3.00 / $15.00 per 1M) |
5. Production Architecture Recommendations
By implementing a hybrid model routing strategy, IDP workloads can achieve enterprise-grade resilience while slashing processing costs by up to 90%:
- Use Nova Lite for Pre-Classification & Summarization: Nova Lite is fast and exceptionally cheap. With an application-level retry wrapper around summarize, Nova Lite safely handles high-volume text inputs.
- Route Structured Field Extraction & Image Regions to Ministral 3 14B: For visual region detection and structured extraction, ministral-3-14b-instruct offers superior spatial awareness and strict schema compliance at a fraction of the cost of flagship multimodal models.
- Enforce Explicit Non-Streaming Configs: Always set `image_streaming: false` when routing structured output or tool choice tasks to Mistral foundation models on AWS Bedrock.
- Utilize Active Production Models for Audits: For critical fallback paths, utilize supported active identifiers (e.g., anthropic.claude-sonnet-4-5-20250929-v1:0) rather than deprecated legacy models.
By decoupling pipeline stages and matching each document operation to the right model capabilities, engineering teams can build scalable, fault-tolerant IDP systems that overcome foundation model quirks.