SPARTA: Scalable and Principled Benchmark of Tree-Structured Multi-hop QA over Text and Tables

Paper arXiv Code Dataset Leaderboard

About SPARTA

SPARTA introduces a groundbreaking benchmark for tree-structured multi-hop question answering (QA) across text and tables, addressing the critical shortcomings of existing datasets like HybridQA and OTT-QA, which suffer from shallow reasoning, annotation errors, and limited scale. By constructing a unified reference fact database that merges source tables with grounding tables derived from unstructured passages, our end-to-end framework automates the generation of thousands of high-fidelity QA pairs—requiring only a quarter of the annotation effort—while incorporating advanced operations like aggregation, grouping, and deep nested predicates. Innovative techniques such as provenance-based refinement and realistic-structure enforcement ensure executable, semantically sound queries that mimic real-world complexity, spanning domains like NBA, movies, and medicine. On SPARTA, state-of-the-art models plummet by over 30 F1 points, exposing gaps in cross-modal reasoning and paving the way for more robust QA systems.

Representative examples of SPARTA benchmark

Figure 1: Representative examples from the SPARTA benchmark showing multi-hop reasoning across tables and text with varying query structures.

Why SPARTA?

In recent years, benchmarks like HybridQA and OTT-QA have pioneered Table-Text QA. However, their reliance on manual curation has led to fundamental limitations: shallow reasoning, high annotation noise, and toy-scale data environments.

As LLMs evolve to handle increasingly complex analytical tasks, we present SPARTA, a scalable and principled framework designed to advance cross-modal reasoning through automated, high-fidelity benchmark construction.

SPARTA redefines the standards of Table-Text QA with three core advancements:

  • Real-World Scale: We move beyond tiny web tables (averaging 15 rows) to realistic settings, featuring relational data with up to thousands of rows.
  • Tree-Structured Multi-Hop Reasoning: Unlike existing benchmarks limited to linear, shallow chains, SPARTA synthesizes complex queries requiring tree-structured reasoning, including advanced operations like aggregation and grouping.
  • High Fidelity & Reliability: By replacing error-prone manual labeling (which has up to a 21% error rate) with provenance-based refinement, SPARTA ensures executable, natural-sounding, and noise-free benchmarks at 4x the construction efficiency.

The challenge is significant: even state-of-the-art LLMs experience a performance drop of over 30 F1 points on SPARTA compared to previous benchmarks, highlighting the critical need for more robust and scalable evaluation.

Benchmark Comparison

SPARTA addresses critical limitations of existing Table-Text QA benchmarks

Benchmark Avg. Rows Question Gen. Grouping/Having Deep Multi-hop (>3) Star-shape Query Annotation Error
(over 100 sampled queries)
TAT-QA 9.4 Manual 30%
FinQA 6.4 Manual 27%
MultiHierTT 10.8 Manual 26%
HybridQA 15.7 Manual 21%
OTT-QA 15.7 Manual 21%
SPARTA (NBA) 3,280.5 Auto (LLM) 0%
SPARTA (Movie) 10,054.0 Auto (LLM) 0%
SPARTA (Medical) 200.0 Auto (LLM) 0%

  Examples from SPARTA Benchmark

Natural Language Question NBA Height: 1 Breadth: 1
"Which Point Guards, drafted between 2000 and 2005, had more than 4 three-pointers, more than 8 field goals and more than 1 steal in a game?"
Generated SQL Query
SELECT player_name
FROM nba_player_game_stats
WHERE player_name IN (
    SELECT player_name
    FROM nba_player_information
    WHERE position = 'Point Guard'
      AND draft_year BETWEEN 2000 AND 2005
  )
  AND number_of_three_point_field_goals_made > 4
  AND number_of_field_goals_made > 8
  AND number_of_steal > 1
Required Reasoning
  1. Find Point Guards drafted between 2000 and 2005 (Table: nba_player_information)
  2. Filter games with >4 three-pointers, >8 field goals, >1 steal (Text: nba_player_game_stats)
  3. Join results via nested subquery to find matching players
Answer
Chris Paul
Natural Language Question NBA Height: 1 Breadth: 1
"Which NBA players who were drafted in the first round and play the center position have a salary of over 1 million in the 2016-17 season?"
Generated SQL Query
SELECT player_name
FROM nba_player_information
WHERE player_name IN (
    SELECT player_name
    FROM nba_player_affiliation
    WHERE salary > 1000000
      AND season = '2016-17'
  )
  AND draft_round = '1st round'
  AND position = 'Center'
Required Reasoning
  1. Find players with salary >1M in 2016-17 season (Table: nba_player_affiliation)
  2. Filter for 1st round draft picks who play Center (Table: nba_player_information)
  3. Join results via nested subquery to find matching players
Answer
Alex Len, Al Horford, Andre Drummond, ...
Natural Language Question Medical Height: 1 Breadth: 1
"What are the names of patients who have an appointment with a doctor who works at the central hospital and has more than 20 years of experience?"
Generated SQL Query
SELECT patient_name
FROM appointments
WHERE doctor_name IN (
    SELECT name
    FROM doctors
    WHERE hospital_branch = 'Central Hospital'
      AND years_experience > 20
  )
Required Reasoning
  1. Find doctors at Central Hospital with >20 years of experience (Text: doctors)
  2. Find patients who have appointments with those doctors (Table: appointments)
Answer
Alex Smith, Alex Aiden Moore, Emily Miller, ...
Natural Language Question NBA Height: 1 Breadth: 2
"What are the years of birth of the players who have a lane agility time of more than 11.5 seconds, a three quarter sprint of less than 3.35 seconds, more than 10 field goals made and more than 8 rebounds in a game?"
Generated SQL Query
SELECT birthyear
FROM nba_player_information
WHERE player_name IN (
    SELECT player_name
    FROM nba_draft_combine_stats
    WHERE lane_agility_time > 11.5
      AND three_quarter_sprint < 3.35
  )
  AND player_name IN (
    SELECT player_name
    FROM nba_player_game_stats
    WHERE number_of_field_goals_made > 10
      AND number_of_rebound > 8
  )
GROUP BY birthyear
Required Reasoning
  1. Find players with lane agility >11.5s and sprint <3.35s (Table: nba_draft_combine_stats)
  2. Find players with >10 field goals and >8 rebounds (Text: nba_player_game_stats)
  3. Intersect both conditions and retrieve birth years (Breadth-2 join via nba_player_information)
Answer
1984, 1985, 1989, ...
Natural Language Question Movie Height: 1 Breadth: 2
"Which movies, starring Vincent D Onofrio as an actor, have an average rating greater than 5 and a median rating of 6, excluding 'Kolonya Cumhuriyeti'?"
Generated SQL Query
SELECT title
FROM movie
WHERE title IN (
    SELECT movie_title
    FROM role_mapping
    WHERE category = 'actor'
      AND name = 'Vincent D Onofrio'
      AND movie_title = title
  )
  AND title IN (
    SELECT movie_title
    FROM ratings
    WHERE avg_rating > 5
      AND median_rating = 6
      AND movie_title <> 'Kolonya Cumhuriyeti'
  )
Required Reasoning
  1. Find movies where Vincent D Onofrio is an actor (Text: role_mapping)
  2. Find movies with avg_rating > 5 and median_rating = 6 (Table: ratings)
  3. Intersect both conditions and exclude specific movie (Breadth-2 join)
Answer
CHIPS, In Dubious Battle
Natural Language Question NBA Height: 1 Breadth: 2
"Who are the top 5 centers drafted in the 1st round, who have won the dpoy award after 2000, and who have earned more than 2 million dollars in the 2004-05 season, sorted by their draft year in descending order and birth year in ascending order?"
Generated SQL Query
SELECT player_name
FROM nba_player_information
WHERE player_name IN (
    SELECT player_name
    FROM nba_player_affiliation
    WHERE salary > 2000000
      AND season = '2004-05'
  )
  AND player_name IN (
    SELECT player_name
    FROM nba_player_award
    WHERE season > 2000
      AND award = 'dpoy'
  )
  AND position = 'Center'
  AND draft_round = '1st round'
ORDER BY draft_year DESC, birthyear ASC
LIMIT 5
Required Reasoning
  1. Find players with salary >2M in 2004-05 season (Table: nba_player_affiliation)
  2. Find players who won DPOY award after 2000 (Table: nba_player_award)
  3. Intersect both, filter for 1st round Centers, sort and limit (Breadth-2 join)
Answer
Dwight Howard, Ben Wallace, ...
Natural Language Question Medical Height: 2 Breadth: 1
"Find the addresses of male patients born after January 1, 1980, who have MedCare Plus insurance and have made payments that exceed the average failed payments greater than 2500."
Generated SQL Query
SELECT address
FROM patients
WHERE name IN (
    SELECT patient_name
    FROM billing
    WHERE amount > (
        SELECT AVG(amount)
        FROM billing
        WHERE payment_status = 'Failed'
          AND amount > 2500
      )
  )
  AND date_of_birth > '1980-01-01'
  AND gender = 'M'
  AND insurance_provider = 'MedCare Plus'
Required Reasoning
  1. Compute average of failed payments exceeding 2500 (Nested aggregation: billing)
  2. Find patients with payments above that average (Table: billing)
  3. Filter male patients born after 1980 with MedCare Plus (Text: patients)
Answer
123 Elm St, 789 Pine Rd, ...
Natural Language Question NBA Height: 2 Breadth: 1
"Which NBA players, who are centers and taller than the average height of point guards drafted after 1990, have more than 8 rebounds in a game?"
Generated SQL Query
SELECT player_name
FROM nba_player_game_stats
WHERE player_name IN (
    SELECT player_name
    FROM nba_player_information
    WHERE height > (
        SELECT AVG(height)
        FROM nba_player_information
        WHERE position = 'Point Guard'
          AND draft_year > 1990
      )
      AND position = 'Center'
  )
  AND number_of_rebound > 8
Required Reasoning
  1. Compute average height of Point Guards drafted after 1990 (Nested aggregation)
  2. Find Centers taller than that average (Table: nba_player_information)
  3. Filter games with >8 rebounds (Text: nba_player_game_stats)
Answer
Alex Len, Al Horford, Andre Drummond, ...
Natural Language Question Medical Height: 2 Breadth: 1
"What are the names of female patients who registered after 2021-09-02 and have billed amounts greater than the average amount of failed payments over 2500?"
Generated SQL Query
SELECT name
FROM patients
WHERE name IN (
    SELECT patient_name
    FROM billing
    WHERE amount > (
        SELECT AVG(amount)
        FROM billing
        WHERE payment_status = 'Failed'
          AND amount > 2500
      )
  )
  AND gender = 'F'
  AND registration_date > '2021-09-02'
Required Reasoning
  1. Compute average of failed payments exceeding 2500 (Nested aggregation: billing)
  2. Find patients with billed amounts above that average (Table: billing)
  3. Filter for female patients registered after 2021-09-02 (Text: patients)
Answer
Emily Jones, Laura Aiden Davis, ...
Natural Language Question Movie Height: 1 Breadth: 2
"How many movies starring John Abraham have a median rating above 5 and average rating above 4?"
Generated SQL Query
SELECT COUNT(title) AS number_of_movies
FROM movie
WHERE title IN (
    SELECT movie_title
    FROM role_mapping
    WHERE category = 'actor'
      AND name = 'John Abraham'
    GROUP BY movie_title
  )
  AND title IN (
    SELECT movie_title
    FROM ratings
    WHERE median_rating > 5
      AND avg_rating > 4
  )
Required Reasoning
  1. Find movies where John Abraham is an actor (Text: role_mapping)
  2. Find movies with median_rating > 5 and avg_rating > 4 (Table: ratings)
  3. Intersect both conditions and count matching movies (Breadth-2 join)
Answer
1
Natural Language Question NBA Height: 2 Breadth: 1
"What is the maximum height of the Lakers players who play as center, were drafted after 1995 and have a salary greater than the highest salary of the Suns and greater than 20,000,000?"
Generated SQL Query
SELECT MAX(height)
FROM nba_player_information
WHERE player_name IN (
    SELECT player_name
    FROM nba_player_affiliation
    WHERE salary > (
        SELECT MAX(salary)
        FROM nba_player_affiliation
        WHERE team_name = 'Suns'
      )
      AND salary > 20000000
      AND team_name = 'Lakers'
  )
  AND position = 'Center'
  AND draft_year > 1995
Required Reasoning
  1. Compute maximum salary of Suns players (Nested aggregation: nba_player_affiliation)
  2. Find Lakers players with salary above that max and >20M (Table: nba_player_affiliation)
  3. Filter for Centers drafted after 1995 and return max height (Table: nba_player_information)
Answer
84
Natural Language Question NBA Height: 1 Breadth: 3
"What are the names of the teams that scored more than the highest points scored by the Thunder when they scored more than 25 points in the first quarter and scored more than the highest points scored by teams that scored more than 100 points and had a three point field goal percentage of more than 30 and have an arena capacity of more than 20,000 and are not the Pistons?"
Generated SQL Query
SELECT team_name
FROM nba_team_game_stats
WHERE team_points > (
    SELECT MAX(team_points)
    FROM nba_team_game_stats
    WHERE team_name = 'Thunder'
      AND team_points_in_quarter1 > 25
  )
  AND team_points > (
    SELECT MAX(team_points)
    FROM nba_team_game_stats
    WHERE team_points > 100
      AND team_percentage_of_three_point_field_goal_made > 30
  )
  AND team_name IN (
    SELECT team_name
    FROM nba_team_information
    WHERE arena_capacity > 20000
      AND team_name <> 'Pistons'
  )
Required Reasoning
  1. Find max points of Thunder when Q1 > 25 (Subquery 1, Text: nba_team_game_stats)
  2. Find max points of teams with >100 pts and >30% 3pt FG (Subquery 2, Text: nba_team_game_stats)
  3. Find teams with arena capacity >20,000 excluding Pistons (Subquery 3: nba_team_information)
  4. Combine all three conditions (Breadth-3 join)
Answer
Bulls
Natural Language Question Movie Height: 1 Breadth: 3
"Which movies directed by Vivek Athreya have a median rating greater than 5 with more than 100 total votes, and do not feature Matt Smith as an actor?"
Generated SQL Query
SELECT title
FROM movie
WHERE title IN (
    SELECT T2.movie_title
    FROM director_mapping AS T2
    WHERE T2.name = 'Vivek Athreya'
      AND movie.title = T2.movie_title
  )
  AND NOT title IN (
    SELECT movie_title
    FROM role_mapping
    WHERE category = 'actor'
      AND name = 'Matt Smith'
  )
  AND title IN (
    SELECT movie_title
    FROM ratings
    WHERE median_rating > 5
      AND total_votes > 100
    GROUP BY movie_title
  )
Required Reasoning
  1. Find movies directed by Vivek Athreya (Subquery 1, Text: director_mapping)
  2. Exclude movies featuring Matt Smith as actor (Subquery 2, Text: role_mapping)
  3. Find movies with median_rating > 5 and >100 votes (Subquery 3: ratings)
  4. Combine all three conditions (Breadth-3 join)
Answer
Brochevarevarura, Mental Madhilo
1 / 13
Complexity

Key Contributions

Reference Fact Database

Unifies heterogeneous evidence (tables + text) inside a single relational store, making all facts uniformly queryable via SQL.

Provenance-Based Refinement

Uses "why-not provenance" to identify and fix overly selective predicates, ensuring every generated query returns valid results.

Realistic-Structure Enforcement

Constrains generation to post-order traversal of query graphs, producing human-like SQL that mirrors how analysts actually write queries.

Quality-Assured Generation

Achieves 0% annotation error rate over 100 sampled queries with lightweight human validation, compared to 21-30% error rates in existing benchmarks.

Framework Overview

SPARTA Pipeline Overview

Figure 2: Overview of SPARTA pipeline: (1) Reference Fact Database Construction, (2) Query Generation, (3) Question Verbalisation.

1

Reference Fact Database Construction

SPARTA unifies all evidence—structured and unstructured—into a single relational store called the reference fact database. Textual facts are decomposed into atomic propositions and stored as tuples, making them directly addressable via SQL alongside structured data.

2

Query Generation with Provenance-Based Refinement

An LLM generates SQL queries with controlled hop counts. Two safeguards ensure quality: provenance-based refinement fixes empty-result queries using "why-not" analysis, and realistic-structure enforcement follows post-order traversal for human-like SQL.

3

Question Verbalisation

Each validated SQL query is converted to fluent natural language using AST-ICL, a state-of-the-art SQL-to-text model. Lightweight human validation ensures correctness with 75% less annotation time than traditional methods.

Provenance-Based Refinement

Our novel technique that identifies and fixes overly selective predicates using database provenance

Provenance-based refinement process

Figure 3: Provenance-based refinement process. When a query returns empty results, the system traces the failure using "why-not provenance", identifies the blocking predicate, and guides the LLM to relax overly restrictive conditions.

Experimental Results

State-of-the-art models experience dramatic performance drops on SPARTA:
ODYSSEY drops from 69.5 F1 (HybridQA) to 35.6 F1 (SPARTA)
HProPro drops from 70.5 F1 (HybridQA) to 40.4 F1 (SPARTA)

F1 scores across tree configurations

Figure 4: F1 scores across query tree configurations. Performance degrades as depth and breadth increase.

F1 scores across SQL operations

Figure 5: F1 scores across analytical operations. Models struggle with GROUP BY, ORDER BY, and aggregation.

Query naturalness comparison

Figure 6: Query naturalness evaluation comparing different generation methods. Our execution-guided approach with post-order traversal achieves the highest naturalness scores.

Case Study

Case study showing model failures

Figure 7: Case study illustrating typical failure modes of current Table-Text QA models on SPARTA benchmark questions.

BibTeX

@inproceedings{
park2026sparta,
title={{SPARTA}: Scalable and Principled Benchmark of Tree-Structured Multi-hop {QA} over Text and Tables},
author={Sungho Park and Jueun Kim and Wook-Shin Han},
booktitle={The Fourteenth International Conference on Learning Representations},
year={2026},
url={https://openreview.net/forum?id=8KE9qvKhM4}
}