Adversarial evaluation suites for agent products
You are shipping an agent; I build the test set that tries to break it. Contradictory sources, long dependency chains, tool-use traps: the cases your demo never hits and your customers will.
AI agent evaluation · adversarial task design
I break AI agents before your customers do.
I design adversarial evaluations and simulated enterprise environments that catch agents skipping verification, losing state, and trusting the wrong source.
01
What I do
Agentic task and environment design
Training and benchmarking agents takes realistic work, not toy prompts. I design the tasks and build the simulated business stacks where an agent completes multi-step jobs that a machine can grade.
Bilingual English and Spanish data work
If your product serves Latin America, English-only evals leave most of your users untested. I design and review evaluation and training data in both languages, in natural Mexican Spanish, not translationese.
02
How I work
Good evaluation starts from a failure hypothesis, a specific way agents break: skipped verification, stale state, blind trust in a single source. I build data where that break is the only route to a wrong answer, then write a rubric small enough for a machine to grade. If a task cannot make a strong model fail for the right reason, it gets rewritten or cut.
-
T-01
Cross-document contradiction design
Input files that each hold together on their own but contradict each other on a key claim. Passing requires verifying across sources instead of trusting any single document.
-
T-02
Long-horizon state tracking
Tasks where step 200 has to honor a decision made back at step 30, with drifting state and delayed dependencies in between.
-
T-03
Rubric and verifier design
Few, tight, machine-gradable criteria. The data does the work; the rubric just measures.
-
T-04
Environment building
Simulated business stacks where agents perform realistic multi-step work: email, chat, project trackers, ERP, HR, cloud file systems.
-
T-05
Calibration against frontier models
Every task runs against multiple frontier models before it ships, to confirm failures are genuine capability gaps and not tooling noise.
03
The contradiction trap
Two documents from a fictional company. Each one holds up on its own. Together they disagree on one number. This is the kind of trap I build, reduced to run in your browser.
NORWOOD SUPPLY CO. · Finance memo
Re: Q3 hardware budget
Board approved the Q3 hardware budget at $48,000 on May 12.
Rack upgrades are confirmed at $9,500, matching the vendor quote.
Finance releases the rest for the laptop refresh and spares next week.
Laptop refresh$31,000
Rack upgrades$9,500
Spares and peripherals$12,000
Q3 hardware total$52,500
FAIL
agent answer
“Norwood's Q3 hardware budget is $52,500, straight from the spreadsheet summary.”
trusted each source independently, never cross-checked
Norwood Supply Co. is fictional. The failure pattern is not.
04
Background
Since 2025 I have designed, built, and calibrated adversarial evaluation tasks and simulated enterprise environments used to stress-test frontier AI models, working through AI training-data platforms for frontier AI labs, under NDA. At Mercor, a nine-month contract across 2025 and 2026, I was QA and project manager on multiple evaluation projects: I reviewed and quality-controlled task submissions, led a domain team of task writers, and coordinated deliverables and calibration standards with senior project leads. I currently work with Alignerr. Across this work I have held all three seats: task designer, QC reviewer, and team lead.
Before AI evaluation: a B.S. in Computer Science from San Diego State University and research work in bioinformatics. I am a co-author on a 2023 methods paper on accurate reconstruction of genome-scale metabolic models (Canto-Encalada et al., 2023, Preprints, DOI 10.20944/preprints202311.0461.v1).
Python · JavaScript / TypeScript · Docker · Linux · LLM evaluation · RLHF and preference data · prompt engineering · adversarial robustness testing · agentic tool-use environments
05
Contact
Currently taking on evaluation and reliability projects.
- Emailjnxx74@gmail.com
- LinkedInlinkedin.com/in/julio-n-719b44201
- LanguagesEnglish and Spanish
- LocationSan Diego / Tijuana