LLMs Boost SQL Query Accuracy by 33%

Critical Barriers to Production SQL Generation

CHASE-SQL: Parallel Pipeline Selection

CHASE-SQL combines multiple SQL generation techniques running in parallel pipelines. A selector model evaluates all outputs and chooses the best answer. The system incorporates:

• Advanced prompt engineering: Multiple prompt strategies generate diverse candidates
• Self-reflection: Models analyze their own outputs to identify and fix syntax errors
• Ensemble selection: Voting mechanism across pipeline results

Trade-off: Prioritizes benchmark accuracy over accessibility and speed—requires significant compute resources for parallel pipelines.

SQLFuse: Modular Pipeline Architecture

SQLFuse integrates prompt engineering, multi-module pipelines, and fine-tuning across four specialized modules:

1. Schema Mining: Crafts context including table structures, column definitions, constraints, and relationships
2. Schema Linking: Uses a specialized trained LLM to identify relevant columns and tables for the query
3. SQL Generation: Core model equipped with curated context and user question generates SQL
4. SQL Critic: Validation module reviews generated SQL for logical and syntactic errors

SkyRL-SQL: Multi-Step Reinforcement Learning

SkyRL-SQL trains a base model using Reinforcement Learning (RL) to solve tasks in multiple sequential steps. This approach shares conceptual similarities with the GGPO method detailed later, demonstrating that RL effectively optimizes for execution accuracy rather than just syntactic correctness.

SFT Problems for Long Reasoning Tasks

• Lack of Reasoning Datasets: SQL requires multi-step logical reasoning (schema selection → join logic → aggregation → filtering). Training data rarely captures this reasoning process explicitly.
• Misalignment with End Goals: SFT optimizes for producing text that looks like the training examples, not for generating SQL that executes correctly and returns accurate results.
• Writing Quality ≠ Problem Solving: SFT can improve syntactic quality without improving semantic correctness. A query can be beautifully formatted but return wrong results.

The “Reverse-Mining” Chain of Thought Problem

A common practice involves “reverse-mining” the Chain of Thought (CoT)—having a model reconstruct the reasoning behind a given correct answer. This approach is problematic:

• Expensive: Requires running inference on every training example to generate reasoning chains
• Logically Inconsistent: Models often fabricate plausible-sounding but incorrect reasoning that happens to reach the right answer
• Prior Knowledge Leakage: Hints in prompts allow models to guess answers without genuine reasoning, then backfill explanations

How RL Works for SQL Generation

The Core Concept: Treat the model as a policy that selects actions (generates tokens) to maximize a reward (passing unit tests, returning correct results, receiving positive user ratings).

Key Advantages:

• Direct Optimization: Trains for execution accuracy, not text similarity to training examples
• Real Feedback: Rewards based on whether queries actually work when run against databases
• Multi-Step Reasoning: Model learns strategies for complex queries through trial and reward
• Self-Correction: RL naturally encourages exploration of alternatives when initial approaches fail

The Training Loop:

1. Model generates SQL query for a natural language question
2. Query executes against database (or simulated environment)
3. Reward signal based on execution success and result correctness
4. Model weights update to increase probability of actions that led to high rewards
5. Process repeats across thousands of examples

The GGPO Framework

Key Innovation: Integrate Grammar Based Neural Fuzzer (GBNF) grammar into sampling parameters in vLLM, preventing generation of syntactically incorrect SQL.

How It Works:

1. Grammar Constraint: GBNF grammar defines valid SQL syntax for target dialect
2. Guided Decoding: During generation, model can only select tokens that maintain syntactic validity
3. RL Training: Model learns to maximize execution accuracy within grammar constraints
4. Guaranteed Syntax: Output is always parseable SQL—no syntax errors possible

Why This Combination Matters:

Traditional RL for text generation can explore invalid syntax, wasting training time on queries that fail to parse. GGPO ensures every generated query is syntactically valid, focusing RL exploration on semantic correctness—choosing right tables, joins, filters, and aggregations.

Interpreting the Results

Why challenging queries improved while simple queries didn’t:

• Simple queries: Both models already perform well (~24% accuracy). Little room for improvement.
• Moderate queries: Slight decline suggests model may be overfitting to complex patterns at expense of middle-tier queries.
• Challenging queries: Dramatic improvement (2.744 → 3.659) demonstrates RL’s strength—learning multi-step reasoning for complex joins, subqueries, and aggregations.

The validation set showed approximately 11% overall improvement, with the real gains concentrated where they matter most: queries that traditionally stump SQL generation systems.

Simple Queries (Single Table, Basic Operations)

Q: How many heads of the departments are older than 56?

SELECT COUNT(*) FROM head WHERE age > 56

Q: List the creation year, name, and budget of each department.

SELECT creation, name, budget_in_billions FROM department

Q: What is the maximum and minimum budget of the departments?

SELECT MAX(budget_in_billions), MIN(budget_in_billions) FROM department

Moderate Queries (Joins, Aggregations, Date Functions)

Q: In 2012, who had the least consumption in LAM segment?

SELECT T1.CustomerID 
FROM customers AS T1 
INNER JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID 
WHERE T1.Segment = 'LAM' 
  AND SUBSTR(T2.Date, 1, 4) = '2012' 
GROUP BY T1.CustomerID 
ORDER BY SUM(T2.Consumption) ASC 
LIMIT 1

Q: Average monthly consumption of SME customers in 2013?

SELECT AVG(T2.Consumption) / 12 
FROM customers AS T1 
INNER JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID 
WHERE SUBSTR(T2.Date, 1, 4) = '2013' 
  AND T1.Segment = 'SME'

Challenging Queries (Complex Logic, Case Statements, Multiple Joins)

Q: What is the ratio of customers who pay in EUR vs. CZK?

SELECT CAST(SUM(CASE WHEN Currency='EUR' THEN 1 ELSE 0 END) AS DOUBLE) / 
       SUM(CASE WHEN Currency='CZK' THEN 1 ELSE 0 END) 
FROM customers

Q: Difference in gas consumption between CZK and EUR customers in 2012?

SELECT SUM(CASE WHEN T1.Currency='CZK' THEN T2.Consumption ELSE 0 END) - 
       SUM(CASE WHEN T1.Currency='EUR' THEN T2.Consumption ELSE 0 END) 
FROM customers AS T1 
INNER JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID 
WHERE SUBSTR(T2.Date, 1, 4) = '2012'

Q: Which year recorded the most gas consumption paid in CZK?

SELECT SUBSTR(T2.Date, 1, 4) 
FROM customers AS T1 
INNER JOIN yearmonth AS T2 ON T1.CustomerID = T2.CustomerID 
WHERE T1.Currency = 'CZK' 
GROUP BY SUBSTR(T2.Date, 1, 4) 
ORDER BY SUM(T2.Consumption) DESC 
LIMIT 1

Two-Stage Fine-Tuning

Implement preliminary SFT on reasoning datasets before applying RL. This gives the model a stronger baseline for understanding SQL structure, then RL refines for execution accuracy.

Rationale: SFT establishes syntactic foundations; RL optimizes semantic correctness. Together, they address both aspects effectively.

Dataset Calibration

Calibrate training dataset for more uniform distribution across difficulty levels. Current results show model improves on challenging queries but regresses slightly on moderate ones—suggesting dataset skew.

Approach: Oversample moderate-difficulty examples or use curriculum learning that progressively introduces complexity.

Architectural Improvements

• Increase LoRA Rank: Higher-rank Low-Rank Adaptation captures more complex patterns without full fine-tuning costs
• Extend Training Time: More RL episodes allow deeper exploration of query space
• Enhanced Grammar Integration: Refine GBNF grammar to handle edge cases and dialect-specific syntax
• Multi-Dialect Support: Train separate grammar-guided models for PostgreSQL, MySQL, T-SQL, BigQuery

Why GGPO Works:

1. Grammar Constraints: Guarantee syntactic correctness, eliminating entire class of errors
2. RL Optimization: Directly optimizes for execution accuracy, not text similarity
3. Focused Learning: Model explores semantic space (right tables, joins, filters) not syntax space
4. Challenging Query Strength: 33% improvement where traditional approaches fail most
5. Statistical Significance: Hedges’ g of +0.692 confirms real, reproducible gains