BroadOakData.UK

RAG Pipeline Edge Cases: Lessons From German Court Decisions

Written by

broadoakdata

in

German federal court decisions look like a clean, ideal Retrieval-Augmented Generation (RAG) use case on paper: a bounded corpus, structured legal metadata, and questions that usually have a single correct answer sitting in a specific paragraph of a specific decision.

In practice, this domain punishes every shortcut that generic RAG tutorials teach you to take.

A single German judicial citation — such as BVerwG, Beschluss vom 08.01.2010 – 9 B 3/09 — compresses court identity, document type, ruling date, and docket number (Aktenzeichen) into one string. When you build a pipeline over this corpus, users ask questions in four totally different ways:

  1. Fact patterns — “The fax arrived a day late due to transmission error.” (Needs dense vector search.)
  2. Exact case numbers — “Worum ging es in 9 B 3/09?” (Needs exact-match lexical or regex filtering.)
  3. Statutory rules — “What are the deadlines under § 58 VwGO?” (Needs strict grounding without pre-training hallucination.)
  4. Case comparisons — “Compare BVerwG 9 B 3/09 and BVerwG 10 C 6/09.” (Needs dual, independent retrieval budgets.)

Here’s a breakdown of what broke when building a real 30-record BGH/BVerwG RAG pipeline, how each failure showed up in the logs, and how to tell a genuine pipeline bug apart from a local model just being sloppy.

1. The ingestion trap: silent metadata degradation

The problem

Source database exports rarely hand you a flat schema. In our raw corpus (federal_docs_enriched.jsonl), top-level fields included court, date, tenor, and text. The actual docket number (Aktenzeichen), document type (Doktyp), and chamber, however, lived inside a nested object:

{
  "id": "bmj_0",
  "court": "BVerwG",
  "date": "20100108",
  "tenor": "...",
  "metadata": {
    "aktenzeichen": "9 B 3/09",
    "doktyp": "Beschluss",
    "spruchkoerper": "9. Senat"
  }
}

Most ingestion tools run basic key lookups like record.get(field_name) without dotted-path resolution. Pointing --filter-field aktenzeichen or --source-field aktenzeichen at this record evaluates record.get("aktenzeichen"), which silently returns None every time.

How it fails in production

It doesn’t crash — it degrades. With no citation string built, the ingester falls back to the raw record ID (bmj_0). The LLM then answers confidently, producing output like:

Aktenzeichen: bmj_0, Gericht: Bundesverwaltungsgericht, Datum: nicht explizit in der gegebenen Information…

The summary is right, but the citation is garbage. Automated evaluations fail — not because retrieval found the wrong text, but because metadata was never promoted to a top-level schema key.

The fix

Promote nested fields inline during a single streaming pass, using a transform callback:

def prepare_record(record: dict) -> dict:
    meta = record.get("metadata") or {}
    record["citation"] = build_citation(
        record.get("court", ""),
        meta.get("doktyp", ""),
        format_date(record.get("date", "")),
        meta.get("aktenzeichen", ""),
    )
    record["aktenzeichen"] = meta.get("aktenzeichen", "")
    record["doktyp"] = meta.get("doktyp", "")
    return record

Developer tip: Don’t write a second preprocessed file (_prepared.jsonl) to disk — that doesn’t scale to multi-gigabyte exports. Run the transform inline. Also make sure your execution path includes your local working directory (PYTHONPATH=.) when calling console CLI scripts, or you’ll get hit with ModuleNotFoundError.

2. Hybrid search and reciprocal rank fusion (RRF) bias

To catch both natural-language fact patterns and literal legal terms, you need hybrid search: dense vector search (bi-encoders) plus BM25 (a sparse keyword index).

  • BM25 is an inverted index counting exact word frequencies. It easily finds exact docket tokens like 9 B 3/09, but fails completely on cross-lingual queries (e.g., an English query against German source text).
  • Vector search embeds text as floating-point numbers. It handles cross-lingual concept matching easily, but treats 9 B 3/09 as an arbitrary character string — confusing it with 9 B 20/06.

Why RRF breaks

When combining BM25 and vector search results, most frameworks use reciprocal rank fusion (RRF):

RRF_score(document) = Σ  1 / (k + rank_m(document))   across each retrieval method m, with k = 60

RRF ignores raw relevance scores and looks only at rank position. The failure mode: it heavily rewards documents that appear on both candidate lists, regardless of actual relevance.

If your knowledge base has a small auxiliary table — say, a 15-chunk citation style guide or glossary — sitting alongside a 300,000-chunk court decision table, every item in that small table gets returned on both candidate lists for nearly every query, simply because the table is small enough to be exhaustively searched.

Because those auxiliary chunks show up on both lists, they accumulate double RRF points and structurally outrank genuinely relevant court decisions.

The fix: cross-encoder reranking

Add a cross-encoder reranker after RRF fusion. Where bi-encoders process the query and each document separately, a cross-encoder evaluates the query and a candidate chunk jointly, through full self-attention:

Hybrid search candidates (top 20) → [ cross-encoder reranker ] → final top 5

Watch for silent soft failures. Frameworks often fail soft if a reranker library (like fastembed) is missing — they log a subtle warning and silently fall back to un-reranked RRF. Always verify your dependencies loaded in the actual runtime interpreter, not just in requirements.txt.

3. Subword tokenization and regex pre-routing

Embedding models break text into subwords (byte-pair encoding). An Aktenzeichen like 9 B 3/09 gets sliced into generic tokens: ["9", "B", "3", "/", "09"].

In vector space, these generic tokens map to diffuse clusters that overlap with thousands of unrelated procedural rulings — the model has no way to represent “this specific docket number” as a coherent concept.

The deterministic fix

Before passing exact identifier queries to vector models or BM25, run a deterministic regex interceptor:

import re

AKTENZEICHEN_PATTERN = r'\b(?:\d+\s+[A-Z]+\s+\d+/\d+|[V|VIB|VIII]+\s+[A-Z]+\s+\d+/\d+)\b'

def route_query(query_text: str, table_client: Any, top_k: int = 5):
    matches = re.findall(AKTENZEICHEN_PATTERN, query_text, re.IGNORECASE)
    if matches:
        # Route directly to an indexed database lookup (WHERE aktenzeichen = match)
        return table_client.exact_filter("aktenzeichen", matches[0])

    return execute_hybrid_search(query_text, table_client, top_k)

Crucial detail for case comparisons: if a query mentions two case numbers (“Compare 9 B 3/09 and 10 C 6/09”), allocate an independent retrieval budget per matched identifier. A single global top_k limit over an OR filter lets chunks from the first case crowd out the second.

4. Local testing: disambiguating pipeline bugs from model ceilings

Iterating locally with small open-weight models (like Ollama running qwen2.5:7b) gives you a fast, offline dev loop. But smaller models introduce generation quirks that mimic pipeline bugs — and it’s easy to misdiagnose one as the other.

Failure symptomWhat it looks likeWhat’s actually happeningDiagnostic check
Wrong case returned in comparisonSearch or indexing bugThe local LLM rewrote or paraphrased the tool query inside search_knowledge(), stripping the exact docket strings before the backend regex could catch themCheck audit.jsonl / agent.log for the exact string emitted in Tool Call #1
Missing legal outcome termsRetrieval missRetrieval was 100% correct — the LLM simply paraphrased formal terms (“verworfen”) into generic language (“deadlines missed”)Compare the raw retrieved context payload against the final answer text
Symmetric language mirroringPrompt template bugThe 7B model flip-flops — answering German queries in English and English queries in German — due to limited instruction-following capacityPass the exact retrieved context payload to a larger model (e.g. Claude Sonnet or GPT-4o) and compare

The production checklist

  1. Promote nested metadata inline. Lift nested fields (metadata.aktenzeichen) to top-level schema columns during streaming ingestion reads.
  2. Set PYTHONPATH=. Explicitly include the current directory when invoking CLI console scripts.
  3. Audit reranker logs. Verify cross-encoder models actually load into memory — don’t trust a clean exit code alone.
  4. Deploy regex pre-routing. Bypass vector search and embeddings entirely for exact legal identifiers, using deterministic pattern matching.
  5. Set per-entity retrieval budgets. Give multi-case comparison queries independent candidate limits, not one shared top_k.
  6. Separate retrieval from generation. Inspect raw tool-call traces (audit.jsonl) before assuming an evaluation failure means bad search indexing — often it’s the model paraphrasing, not the retriever missing.

More posts