puodziukas.dev · remote · available Q3 2026
LLMOps / FMOps Engineer
Deterministic replay over a 47,312-record deterministic replay corpus. 0 divergence, hash-verified. LLM gate: public corpus n=220 (140 adversarial + 80 benign), 140/140 caught, 0/80 over-blocked.
a 47,312-record deterministic replay corpus, re-run on every model push. Zero false negatives across the public adversarial corpus (n=220): 140/140 caught, 0/80 over-blocked.
reproducible · verified · available Q3 2026
The problem
A sampling-based eval suite passes, and a hallucination still surfaces six weeks later in a client report. The gap is not in the eval suite itself: it is in what the eval suite was never measuring.
What exists, reproducible
a 47,312-record deterministic replay corpus catches the failure class your sampling-based evals miss. It runs in CI. Every model push gets the same adversarial probe. Zero false negatives across the public adversarial corpus (140/140 caught, 0/80 over-blocked, n=220), not a claim.
Relevant projects
- →COBOL PICTURE-clause truncation surfaced via adversarial probe. Missed by standard monitoring.
- →Deterministic replay pipeline over a 47,312-record deterministic replay corpus: zero false negatives across the public adversarial corpus (140/140 caught, 0/80 over-blocked, n=220)
- →OWASP LLM Top 10 adversarial test suite: injection, jailbreak, hallucination, consistency
- →Inference API cost: $26,400/mo → $0 via local inference mesh (self-hosted), quality parity
- →Automated retrain trigger on drift detection, gated by the deterministic replay suite
- →Prompt versioning architecture with A/B routing and statistical significance gate
- →KV-Cache aggressive strategy: 40-60% token cost reduction documented
What you receive in 30 days
- Adversarial eval suite deployed in your CI pipeline (Week 1)
- Hallucination rate baseline via deterministic oracle, not sampled estimate
- Prompt versioning + A/B routing with statistical significance gate
Companies at this intersection
Weights & Biases · LangChain · Arize AI · Scale AI · Comet ML · Fiddler · Helicone · Langfuse
Working terms
W2 remote · async-first · deliverable-gated output · first artifact in 48h · available Q3 2026
All results under NDA. Representative figures. SHA-256 hash-chained audit trail available on request. · puodziukas.dev