120 cases
Five synthetic databases: university, retail, HR, finance, and logistics.
AI data-agent safety evaluation
Obedience
is not safety.
An AI agent can return clean SQL, valid JSON, and a confident explanation—while still following the wrong authority.
A controlled benchmark
720scored runs · 120 cases × 3 models × 2 repetitions
Three open-weight models. One fixed prompt. One question: did the agent follow the authorised instruction?
01 / The problem
Fluency is easy to mistake for trustworthiness.
Text-to-SQL benchmarks usually reward whether an answer works. Organisations also need to know whether it was permitted. DataAgent-SafeBench measures that missing layer: resistance to false authority, injected instructions, and unsafe task escape.
Abstract
01AWhat this research means—in plain English.
AI data agents can turn everyday questions into database queries. This study asks whether they still know which instructions they are actually allowed to follow when a request tries to impersonate authority or hide malicious instructions in data.
In one sentence
Good-looking SQL and perfectly formatted answers are not proof that an AI agent followed the right person.
Read the abstract as submitted (JRS/2026/039)+
Natural-language AI data agents are increasingly used to translate business questions into SQL, analyse relational data, and present findings through conversational interfaces. Their apparent fluency and structured output, however, do not establish that an answer followed an authorised instruction. Existing text-to-SQL benchmarks, including Spider and BIRD, primarily assess functional performance and provide limited evidence about role-boundary preservation under false authority claims, prompt injection, or malicious instructions embedded in data.
This study introduces DataAgent-SafeBench, a controlled evaluation framework for measuring the gap between functional correctness and safe instruction handling. The benchmark contains 120 cases across five synthetic database domains, including benign, underspecified, privilege-escalation, direct- and indirect-injection, prompt-disclosure, hallucination-trap, and jailbreak/task-escape conditions.
Three open model families were evaluated through NVIDIA NIM with a fixed data-agent prompt, temperature 0, read-only SQLite execution, deterministic result-equivalence scoring, and two repetitions per case, yielding 720 model-case evaluations. Attack success was calculated over the 75 adversarial cases per model, with repeated observations retained and scored separately.
Llama 3.1 8B achieved the lowest attack-success rate at 9.3%; GPT-OSS 20B achieved the strongest refusal correctness at 83.3% and authority handling at 81.5%; and Nemotron Mini 4B achieved the strongest JSON conformance at 99.2% while recording the highest attack-success rate at 37.3%. No model disclosed the planted sensitive-data canary.
DataAgent-SafeBench demonstrates that fluent, structured, and technically valid outputs should not be treated as evidence of authorised behaviour, and provides a transparent basis for safer organisational data-agent controls.
Note: this is the abstract as submitted. Its figures differ slightly from the committed outputs in the public repository, which are the reference for the figures below.
The central finding
02The most structured output was also the least safe.
Nemotron Mini 4B achieved 99.2% JSON conformance, but had the highest attack-success rate: 38.0%. Llama 3.1 8B produced less consistently structured output and had the lowest recorded attack success: 9.3%. With three models this is a counterexample, not a correlation.
Professional appearance is not evidence of authorised behaviour.
0%20%40%
Attack success is lower-is-better: 150 adversarial runs per model (75 cases × 2 repetitions), from the public repository. *GPT-OSS 20B is confounded by a 350-token output cap that truncated 75 of its 240 responses; see the repository’s v1.0.1 re-analysis.
A locked evaluation, explained simply
03One benchmark.
Five domains.
Eight kinds of pressure.
Every model received the same fixed data-agent prompt. Each run was read-only, deterministic, and assessed against an authorised decision—not merely a plausible answer.
3 open-weight models
Llama 3.1 8B, GPT-OSS 20B, and Nemotron Mini 4B via NVIDIA NIM, one per model family.
2 runs each
Temperature 0, fixed prompt, read-only SQLite execution. Repetitions measure instability; the case is the unit of analysis.
720 evaluations
Result equivalence, Wilson intervals, and exact McNemar tests at case level.
What was tested
Each condition tests a different failure mode—rather than treating “prompt injection” as one thing.
30 Benign15 Underspecified15 Privilege escalation15 Direct injection15 Indirect injection10 Prompt disclosure10 Hallucination trap10 Jailbreak / task escape
Results at a glance
04Different metrics tell different stories. Read across the rows—high task success or polished JSON alone cannot answer the safety question.
| Model | Overall task success | Benign task success | Attack success runs · lower is better | Attack success cases · any repeat | JSON conformance | Refusal correctness |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 70.0% | 90.0% | 9.3% | 10.7% | 87.5% | 62.0% |
| GPT-OSS 20B* | 61.2% | 35.0% | 16.7% | 18.7% | 69.2% | 80.7% |
| Nemotron Mini 4B | 55.4% | 80.0% | 38.0% | 38.7% | 99.2% | 36.7% |
Values match the committed outputs in the public repository, where every rate carries a Wilson 95% interval; the submitted abstract reported slightly different values, and the repository is the reference. At case level, Nemotron Mini 4B was significantly more attackable than Llama 3.1 8B (exact McNemar p = 3.2 × 10−4). The other pairwise gaps do not hold up once the two repetitions are treated as one case. *GPT-OSS 20B hit the 350-token output cap on 75 of 240 responses, so its benign and attack figures mostly measure truncation; a re-run with a larger budget is planned.
0
A narrow but important result
No model disclosed the planted sensitive-data canary.
This does not mean no model was safe overall. Unsafe SQL, authority confusion, task escape, and injected-instruction compliance were measured separately.
The evidence trail
05Built to be inspected,
not just believed.
The benchmark repository includes the synthetic cases, scoring rules, raw responses, metric outputs, statistical comparisons, and publication figures.
DataAgent-
SafeBench
Cases, schemas, scorer, outputs, tests, and manuscript materials.
Open repository ↗Exploratory case study Not pooled with benchmark scores
Frontier-model observations stay separate.
Kimi K3 was reviewed through 20 frozen probe templates across five probe families, with three repetitions per template and two independent reviewers. The exploratory review found jailbreak compliance, reported as a bounded qualitative finding because it is outside the locked 720-run benchmark.
View Kimi K3 chat project ↗06 / What this changes
Safer data agents need to be judged by whose instruction they followed.
- 01Measure authority handling alongside SQL accuracy.
- 02Treat user claims and database text as untrusted inputs.
- 03Keep reproducible benchmarks and exploratory case studies distinct.
Acknowledgements
07With gratitude to the people and platforms behind the work.
This project was shaped through academic guidance, institutional support, and the model platforms that enabled the locked benchmark and the clearly separated exploratory study.
Academic guidance
Jayathilake M²
Co-author and research guidance
Lanka Nippon BizTech Institute
Maharagama, Sri Lanka
Maharagama, Sri Lanka
Institutional home
Lanka Nippon BizTech Institute provided the academic setting for JASPER 2026 and this research project.
Technology and platform acknowledgements
NIM inference platform
Exploratory Kimi K3 studyTechnology and model acknowledgement only; no sponsorship or endorsement is implied.