Aggregate Accuracy Trap
Overall accuracy = 97%
This may still hide poor accuracy for a particular document type or field.
Measure Accuracy By
- Document type.
- Individual field.
- Relevant segment.
Stratified Random Sampling
Do not review only low-confidence results.
Sample some high-confidence outputs too.
Why? Detect confidently wrong predictions and new failure patterns.
Confidence Calibration
Field confidence
+
Labeled validation set
↓
Calibrated threshold
↓
Human review
Memory: Confidence must be validated against actual correctness.
Claim-Source Mapping
Every research finding should preserve:
Claim
Evidence
Source URL / document
Publication date
Conflicting Sources
Do NOT arbitrarily choose one value.
Source A → 42%
Source B → 51%
Report both with attribution.
Temporal Context
Preserve:
- Publication date.
- Data-collection date.
Statistics from different periods may not actually conflict.
Coverage Gaps
If sources are unavailable:
- Explicitly report the gap.
- Identify affected topic.
- Do not pretend research is complete.
Report Structure
Well-supported findings
Contested findings
Coverage gaps
Content-Specific Formatting
- Financial data → tables.
- News → prose.
- Technical findings → structured lists.
Fast Revision
Overall accuracy can hide weak segments.
Sample high-confidence outputs.
Confidence must be calibrated.
Claim must keep source.
Conflicting sources → preserve both.
Missing coverage → report it.
