EEGAgentBench: Benchmarking LLM Agents on EEG Analysis

Abstract

A Unified Benchmark for Agentic EEG Analysis

Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence accumulation, multi-step reasoning, and coordinated use of specialized signal-processing tools. Although large language models (LLMs) have recently shown promise as autonomous agents for EEG analysis, existing EEG agentic evaluations remain fragmented, covering limited tasks over narrow temporal horizons with inconsistent protocols. To address this gap, we propose EEGAgentBench, a unified benchmark for systematically evaluating LLM agents on both short- and long-horizon EEG analysis. It spans six representative EEG applications ranging from knowledge question answering to sleep staging, with signal durations from 2 seconds to nearly 23 hours and prediction targets ranging from class labels to event intervals and epoch-level sequences. The benchmark provides 10 deterministic EEG analysis tools that expose only task-relevant measurements, so agents must select tools autonomously, accumulate evidence iteratively, and construct multi-step workflows. Benchmarking 29 frontier LLMs from 15 model families, we find that EEGAgentBench effectively distinguishes agent capabilities beyond model scale and inference cost, while revealing substantial limitations of current agents in long-horizon EEG analysis, particularly in sustained evidence accumulation and multi-step reasoning.

Benchmark

Six Tasks Across Three Capability Dimensions

KnowledgeT1

EEG Knowledge QA

Four-choice medical questions on EEG & clinical neurophysiology.

Source: MedMCQA-EEG · N: 38
Signal: No signal
Metrics: Accuracy

Short horizonT2

Artifact Identification

Decide clean vs. ocular (EOG) vs. muscle (EMG) contamination.

Source: EEGdenoiseNet · N: 300
Signal: 2 s, 1 ch
Metrics: Accuracy, Macro-F1

Short horizonT3

Normal-vs-Epileptic

Screen a single-channel epoch as normal or epileptic.

Source: Bonn · N: 188
Signal: 23.6 s, 1 ch
Metrics: Accuracy, Macro-F1

Short horizonT4

Dementia Cohort

Classify resting-state EEG as AD / FTD / HC cohort.

Source: ds004504 · N: 69
Signal: 5.1–21.5 min, 19 ch
Metrics: Accuracy, Macro-F1

Long horizonT5

Seizure Detection

Localize a variable number of seizure onset/offset intervals.

Source: CHB-MIT · N: 280
Signal: 12.4 min–4.0 h, 18–32 ch
Metrics: Dice-S, Event-F1

Long horizonT6

Sleep Staging

Produce a full-night 30-s epoch AASM 5-class hypnogram.

Source: Sleep-EDFx · N: 197
Signal: 5.7–23.0 h, 5–7 ch
Metrics: Macro-F1, Cohen's κ

Environment

Interactive, Tool-Based EEG Analysis

EEGAgentBench formulates EEG analysis as an interactive process between an agent and an EEG environment, following a ReAct-style multi-turn protocol. Instead of exposing raw waveforms, recordings are accessed exclusively through a unified tool interface. At each step, the agent may invoke an EEG analysis tool with user-defined arguments or terminate by submitting a final prediction. All tools perform deterministic analysis without task-specific classifiers or label information, so performance depends entirely on the agent's reasoning, evidence integration, and workflow construction.

The environment exposes 10 EEG analysis tools spanning recording metadata, signal-quality assessment, frequency- and time-domain analysis, windowed feature extraction, transient candidate detection, and cross-channel analysis. The same tool suite is shared across all signal tasks (T2–T6), letting agents build task-specific workflows without predefined pipelines.

Results

Main Leaderboard

29 LLM agents evaluated on all 1,072 instances. Blue bold = best, italic = second best.

Rank	Model	Overall ↑	T1	T2 Artifact		T3 Screening		T4 Cohort		T5 Seizure		T6 Sleep
Rank	Model	Overall ↑	Acc	F1	Acc	F1	Acc	F1	Acc	Event-F1	Dice-S	F1	κ
Proprietary models
1	GPT-5.6-sol	0.631	0.816	0.599	0.643	0.898	0.899	0.429	0.493	0.549	0.404	0.492	0.488
4	Grok-4.5	0.518	0.789	0.563	0.637	0.620	0.654	0.239	0.348	0.522	0.367	0.372	0.277
13	Claude-Opus-4.6	0.463	0.842	0.498	0.567	0.543	0.612	0.294	0.362	0.289	0.366	0.312	0.204
7	GPT-5.5	0.500	0.737	0.651	0.690	0.543	0.612	0.194	0.333	0.530	0.449	0.347	0.263
21	GPT-5.4	0.438	0.684	0.730	0.733	0.790	0.798	0.205	0.319	0.069	0.063	0.150	-0.004
19	Gemini-2.5-Pro	0.442	0.684	0.576	0.610	0.485	0.569	0.279	0.290	0.291	0.299	0.336	0.252
3	Qwen3.7-Max	0.529	0.763	0.747	0.770	0.909	0.910	0.208	0.319	0.325	0.305	0.219	0.094
8	Gemini-3.5-Flash	0.495	0.684	0.536	0.590	0.709	0.723	0.376	0.464	0.377	0.378	0.291	0.193
Open-weight models: above 1T parameters
18	LongCat-2	0.445	0.737	0.532	0.557	0.724	0.739	0.255	0.348	0.242	0.250	0.182	0.054
20	DeepSeek-V4-Pro	0.439	0.684	0.560	0.640	0.726	0.734	0.194	0.333	0.275	0.269	0.194	0.081
10	MiMo-V2.5-Pro	0.491	0.763	0.745	0.747	0.624	0.660	0.281	0.391	0.313	0.261	0.218	0.065
5	Kimi-K2.6	0.513	0.842	0.642	0.670	0.767	0.777	0.272	0.319	0.347	0.341	0.208	0.071
Open-weight models: 100B–1T parameters
17	GLM-5.2	0.455	0.816	0.629	0.700	0.490	0.580	0.163	0.319	0.329	0.357	0.304	0.205
12	GLM-5.1	0.467	0.763	0.636	0.670	0.704	0.723	0.198	0.348	0.265	0.320	0.236	0.114
14	DeepSeek-V3.2	0.460	0.711	0.631	0.693	0.776	0.777	0.163	0.304	0.271	0.258	0.207	0.075
2	MiniMax-M3	0.530	0.816	0.666	0.683	0.754	0.755	0.360	0.420	0.354	0.270	0.229	0.089
25	ERNIE-4.5-VL-424B	0.402	0.737	0.451	0.457	0.654	0.654	0.391	0.449	0.034	0.019	0.146	0.037
6	Qwen3.5-397B-A17B	0.501	0.763	0.786	0.783	0.711	0.718	0.193	0.333	0.370	0.321	0.185	0.037
9	Hy3	0.492	0.763	0.774	0.777	0.608	0.649	0.307	0.391	0.236	0.243	0.264	0.173
16	DeepSeek-V4-Flash	0.459	0.763	0.495	0.587	0.805	0.803	0.222	0.348	0.284	0.320	0.185	0.047
24	Mistral-Medium-3.5	0.419	0.737	0.505	0.580	0.568	0.628	0.220	0.348	0.264	0.274	0.217	0.079
11	Qwen3.5-122B-A10B	0.485	0.711	0.701	0.710	0.656	0.676	0.338	0.420	0.298	0.250	0.208	0.067
Open-weight models: 10–100B parameters
23	Llama3.3-70B	0.429	0.868	0.452	0.470	0.653	0.654	0.324	0.406	0.104	0.016	0.173	0.023
15	Qwen3.5-35B-A3B	0.459	0.711	0.622	0.640	0.702	0.702	0.313	0.391	0.248	0.248	0.161	0.035
22	Qwen3.6-27B	0.433	0.632	0.542	0.653	0.713	0.718	0.232	0.348	0.294	0.304	0.184	0.059
29	ERNIE-4.5-21B-A3B	0.228	0.526	0.167	0.333	0.301	0.394	0.220	0.217	0.000	0.000	0.151	0.010
Open-weight models: below 10B parameters
26	Qwen3-8B	0.332	0.632	0.221	0.347	0.488	0.521	0.287	0.362	0.143	0.036	0.224	0.081
27	Qwen3-4B	0.299	0.395	0.405	0.417	0.509	0.548	0.248	0.304	0.047	0.014	0.188	0.065
28	Qwen3-1.7B	0.233	0.421	0.261	0.347	0.345	0.505	0.241	0.319	0.000	0.000	0.131	-0.040

Overall is the unweighted mean of T1 accuracy, Macro-F1 for T2–T4 and T6, and Event-F1 for T5.

Case Studies

Agent Trajectories in Action

Real GPT-5.6 trajectories from EEGAgentBench. For every task we show a successful and a failed run; each turn is collapsed by default — click Expand to reveal the agent's full reasoning, tool call, and raw tool output.

Citation

BibTeX

@article{wu2025eegagentbench,
  title   = {EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis},
  author  = {Wu, Huyu and Weng, Weining and Liu, Yuchen and Gu, Yang},
  journal = {arXiv preprint},
  year    = {2025}
}