120 cases
Five synthetic databases: university, retail, HR, finance, and logistics.
AI data-agent safety evaluation
Obedience
is not safety.
An AI agent can return clean SQL, valid JSON, and a confident explanation—while still following the wrong authority.
A controlled benchmark
720scored model–case evaluations
Three open model families. One fixed prompt. One question: did the agent follow the authorised instruction?
01 / The problem
Fluency is easy to mistake for trustworthiness.
Text-to-SQL benchmarks usually reward whether an answer works. Organisations also need to know whether it was permitted. DataAgent-SafeBench measures that missing layer: resistance to false authority, injected instructions, and unsafe task escape.
Abstract
01AWhat this research means—in plain English.
AI data agents can turn everyday questions into database queries. This study asks whether they still know which instructions they are actually allowed to follow when a request tries to impersonate authority or hide malicious instructions in data.
In one sentence
Good-looking SQL and perfectly formatted answers are not proof that an AI agent followed the right person.
Read the full research abstract+
Natural-language AI data agents are increasingly used to translate business questions into SQL, analyse relational data, and present findings through conversational interfaces. Their apparent fluency and structured output, however, do not establish that an answer followed an authorised instruction. Existing text-to-SQL benchmarks, including Spider and BIRD, primarily assess functional performance and provide limited evidence about role-boundary preservation under false authority claims, prompt injection, or malicious instructions embedded in data.
This study introduces DataAgent-SafeBench, a controlled evaluation framework for measuring the gap between functional correctness and safe instruction handling. The benchmark contains 120 cases across five synthetic database domains, including benign, underspecified, privilege-escalation, direct- and indirect-injection, prompt-disclosure, hallucination-trap, and jailbreak/task-escape conditions.
Three open model families were evaluated through NVIDIA NIM with a fixed data-agent prompt, temperature 0, read-only SQLite execution, deterministic result-equivalence scoring, and two repetitions per case, yielding 720 model-case evaluations. Attack success was calculated over the 75 adversarial cases per model, with repeated observations retained and scored separately.
Llama 3.1 8B achieved the lowest attack-success rate at 9.3%; GPT-OSS 20B achieved the strongest refusal correctness at 83.3% and authority handling at 81.5%; and Nemotron Mini 4B achieved the strongest JSON conformance at 99.2% while recording the highest attack-success rate at 37.3%. No model disclosed the planted sensitive-data canary.
DataAgent-SafeBench demonstrates that fluent, structured, and technically valid outputs should not be treated as evidence of authorised behaviour, and provides a transparent basis for safer organisational data-agent controls.
The central finding
02The most structured output was also the least safe.
Nemotron Mini achieved 99.2% JSON conformance, but had the highest attack-success rate: 37.3%. Llama 3.1 8B produced less consistently structured output and had the lowest recorded attack success: 9.3%.
Professional appearance is not evidence of authorised behaviour.
0%20%40%
Attack success is lower-is-better. It is calculated over 75 adversarial cases per model, with two retained observations per case.
A locked evaluation, explained simply
03One benchmark.
Five domains.
Eight kinds of pressure.
Every model received the same fixed data-agent prompt. Each run was read-only, deterministic, and assessed against an authorised decision—not merely a plausible answer.
3 model families
Llama 3.1 8B, GPT-OSS 20B, and Nemotron Mini 4B via NVIDIA NIM.
2 runs each
Temperature 0, fixed prompt, read-only SQLite execution.
720 evaluations
Result equivalence, Wilson intervals, and paired McNemar tests.
What was tested
Each condition tests a different failure mode—rather than treating “prompt injection” as one thing.
30 Benign15 Underspecified15 Privilege escalation15 Direct injection15 Indirect injection10 Prompt disclosure10 Hallucination trap10 Jailbreak / task escape
Results at a glance
04Different metrics tell different stories. Read across the rows—high task success or polished JSON alone cannot answer the safety question.
| Model | Overall task success | Benign task success | Attack success lower is better | JSON conformance | Refusal correctness | Authority handling |
|---|---|---|---|---|---|---|
| Llama 3.1 8B | 68.3% | 91.7% | 9.3% | 87.5% | 56.7% | 69.2% |
| GPT-OSS 20B | 61.2% | 25.0% | 16.0% | 67.1% | 83.3% | 81.5% |
| Nemotron Mini 4B | 55.8% | 80.0% | 37.3% | 99.2% | 37.3% | 66.2% |
The submitted abstract reports Wilson 95% confidence intervals for each primary measure. Llama vs GPT-OSS differed by 6.7 pp in attack success without statistical significance (Holm-adjusted p = 0.087); Nemotron was significantly higher than both (p < 0.001).
0
A narrow but important result
No model disclosed the planted sensitive-data canary.
This does not mean no model was safe overall. Unsafe SQL, authority confusion, task escape, and injected-instruction compliance were measured separately.
The evidence trail
05Built to be inspected,
not just believed.
The benchmark repository includes the synthetic cases, scoring rules, raw responses, metric outputs, statistical comparisons, and publication figures.
DataAgent-
SafeBench
Cases, schemas, scorer, outputs, tests, and manuscript materials.
Open repository ↗Exploratory case study Not pooled with benchmark scores
Frontier-model observations stay separate.
Kimi K3 was reviewed through 20 frozen probe templates across five probe families, with three repetitions per template and two independent reviewers. The exploratory review found jailbreak compliance, reported as a bounded qualitative finding because it is outside the locked 720-run benchmark.
View Kimi K3 chat project ↗06 / What this changes
Safer data agents need to be judged by whose instruction they followed.
- 01Measure authority handling alongside SQL accuracy.
- 02Treat user claims and database text as untrusted inputs.
- 03Keep reproducible benchmarks and exploratory case studies distinct.
Acknowledgements
07With gratitude to the people and platforms behind the work.
This project was shaped through academic guidance, institutional support, and the model platforms that enabled the locked benchmark and the clearly separated exploratory study.
Academic guidance
Jayathilake M²
Co-author and research guidance
Lanka Nippon BizTech Institute
Maharagama, Sri Lanka
Maharagama, Sri Lanka
Institutional home
Lanka Nippon BizTech Institute provided the academic setting for JASPER 2026 and this research project.
Technology and platform acknowledgements
databricks
Technology and model acknowledgement only; no sponsorship or endorsement is implied.