Text-to-SQL and Verifiability

2026-07-28

A sales manager wants to know which of her German customers ordered little or nothing last year. She types the question into an analytics tool, gets back a list, and the list turns into a marketing campaign. Nothing about this is remarkable — except for one question that is rarely asked: how does she know the list is right?

She can't read the generated SQL. That isn't a shortcoming on her part — it is the very purpose of the analytics tool: it exists so that she doesn't have to. So the system supplies an explanation along with it — in natural language, easy to read. And this is exactly where I see a gap, the one that led me to start the koryki project.

One thing up front

I consider SQL one of the most successful languages there is — five decades in production are no accident. koryki does not compete with SQL; quite the opposite, it is built on generating SQL. Every query is translated and executed by the database; SQL is used exactly where it belongs. Anyone who can and wants to use SQL should do so. koryki is there to help those who can't or don't want to.

A good benchmark is not the same as good verifiability

The obvious reply is that the generators have become extremely good, and that is true. On the established benchmarks, current systems reach numbers that seemed out of reach not long ago. On Spider 2.0, which works with real enterprise databases — some with more than a thousand columns, spread across several systems — the best approaches land at roughly 60 to 70 percent execution accuracy, depending on the variant. On simpler benchmarks they score far higher.

But that does not answer the sales manager's question. A benchmark measures how often a system is right across many questions. She has a single question, and she doesn't need a statement of probability, she needs a basis for a decision. Even at 99 percent accuracy it remains open whether her query is correct.

How often correct SQL is generated and whether that SQL can be verified are two different things. The first improves with every new AI model. Verifiability is largely independent of model quality, because it does not depend on how often the system is right, but on whether anyone understands the generated query.

The expectation lives in the asker's head

A query is never just a technical construct. It is the translation of an expectation — and expectations are subjective. Whoever asks about "our German customers" has a particular picture in mind: customers headquartered in Germany? With a German billing address? The ones served by the German office? Whoever says "last year" means either the calendar year or the past twelve months. None of that is in the database. It is in the head of the person asking.

The obvious move is to bring someone in. A colleague from the data team knows the schema, spots an implausible join at once, and asks exactly the questions the sales manager would never think of on her own. That is good practice, and good analyses come out of that collaboration.

But the collaboration costs time and money, and it adds a communication problem. A question becomes a ticket, the ticket becomes a meeting, and if the ticket doesn't answer the question correctly, the process starts over. On top of that come the questions that are never asked at all, because the effort isn't worth it.

The communication problem is the subtler one: the expectation gets translated a second time. First from a mental picture into words, then from those words into the colleague's understanding — and the same ambiguities arise there as everywhere else. Two experts who agree that they have understood "German customers" can still mean different things.

And on one point even the best collaboration doesn't help: whether the query answers the question that was asked is something the data team cannot know as long as the expectation has not been stated in full — and it never is stated in full. Correctness here is not a property of the query alone, it is a relation between query and intent, and only the sales manager knows both.

That makes the value of a readable query language fairly clear, and it isn't limited to verifiability. What the domain expert can follow herself in a few readable lines, nobody else has to read for her: the follow-up question is gone, the waiting time is gone, the second translation is gone — and with it a source of errors.

The technical check can be delegated. The substantive check, whether the query matches the expectation, cannot — and that is why readability and clarity decide whether the sales manager is put in a position to make that judgment.

Three ways to check a query

Sanity-check the result. Order of magnitude, a comparison against last year, a few spot checks — this works better than many assume. Experienced practitioners have a good sense for when a result isn't right. But errors that produce plausible results won't surface this way: a list showing 83 customers instead of 91, a four percent deviation in a total — neither is easy to spot.

Read the SQL. The most reliable route, and the right one for anyone who knows SQL. It requires exactly the knowledge this is all about. And it costs more than commonly assumed, even for those who read SQL every day.

Read the explanation. The route today's tools offer — explanations in natural language. It is the most interesting of the three, because it is the one that offers her the content of the query in a form she can judge.

What the explanation delivers — and what it doesn't

The idea of translating a query back into natural language is neither new nor naive. Alkis Simitsis and Yannis Ioannidis proposed it at CIDR in 2009 under the title DBMSs Should Talk Back Too, explicitly for the purpose of verification: the system should translate its internal result back into language and put it in front of whoever asked, for checking. In the same paper the authors name the conditions under which this works. The output has to be expressive — reproducing exactly what the query does — and at the same time effective, meaning quick and unambiguous to interpret. Achieving both at once, they write, is the real challenge.

Today it still is. An explanation in natural language that correctly conveys everything an SQL statement contains becomes as hard to read as the SQL it explains. One that reads fluently has to leave things out — a conflict of goals.

A second, less comfortable finding comes on top of that. A user study by Vardhan Palod, Upasana Biswas and Subbarao Kambhampati from May 2026, so far not peer-reviewed (Evaluating the False Trust Engendered by LLM Explanations), measured what explanations do to the judgment of the people reading them. The result: reasoning traces and post-hoc explanations are "persuasive but not informative" — they increase acceptance of the answer regardless of whether the answer is correct. Only one variant improved people's ability to tell right from wrong: the contrastive double explanation, which gives arguments for and against its own answer. The study looks at LLM explanations in general, not specifically at SQL — but the finding is unambiguous, and it matches what anyone recognises who has once trusted a well-phrased, wrong answer.

That leaves the ambiguity itself. That the same question admits several plausible SQL variants is not a hunch, it is the subject of dedicated benchmarks: AmbiQT collects more than 3000 questions that each translate two plausible ways, and AMBROSIA organises three types of ambiguity across realistic multi-table databases.

The explanation is phrased in the same language as the question — and inherits its properties. Natural language in, natural language out. The checking step never leaves the medium that produced the ambiguity in the first place. And this is precisely where formal languages are strong: they determine what is executed rather than describing it afterwards, and they permit only one interpretation — expressive and unambiguous. What remains open is the second half of effective: being quick to grasp. This is where SQL has its weaknesses.

An example

Northwind, the sample database that has been around for ages, and a question of the kind sales teams ask every day:

How many orders did each of our German customers place in 2023 — including the ones that placed none?

That last clause is the crux. A customer who ordered nothing is the most interesting audience for a win-back campaign.

Four queries, of the kind a text-to-SQL generator could plausibly return. The first is the intended query in SQL, the other three deviate in one place each. This is what checking would ask of the sales manager: read them, and decide which one answers her question.

Variant A — the intended query:

SELECT c.company_name, count(o.order_id)
FROM customers c
 LEFT OUTER JOIN orders o ON c.customer_id = o.customer_id
  AND o.order_date >= DATE '2023-01-01'
  AND o.order_date <  DATE '2024-01-01'
WHERE c.country = 'Germany'
GROUP BY c.company_name
ORDER BY c.company_name DESC

Variant B — date range in the WHERE instead of the ON:

SELECT c.company_name, count(o.order_id)
FROM customers c
 LEFT OUTER JOIN orders o ON c.customer_id = o.customer_id
WHERE c.country = 'Germany'
  AND o.order_date >= DATE '2023-01-01'
  AND o.order_date <  DATE '2024-01-01'
GROUP BY c.company_name
ORDER BY c.company_name DESC

Variant C — count(*) instead of count(o.order_id):

SELECT c.company_name, count(*)
FROM customers c
 LEFT OUTER JOIN orders o ON c.customer_id = o.customer_id
  AND o.order_date >= DATE '2023-01-01'
  AND o.order_date <  DATE '2024-01-01'
WHERE c.country = 'Germany'
GROUP BY c.company_name
ORDER BY c.company_name DESC

Variant D — BETWEEN with an inclusive upper bound:

SELECT c.company_name, count(o.order_id)
FROM customers c
 LEFT OUTER JOIN orders o ON c.customer_id = o.customer_id
  AND o.order_date BETWEEN DATE '2023-01-01' AND DATE '2023-12-31'
WHERE c.country = 'Germany'
GROUP BY c.company_name
ORDER BY c.company_name DESC

In practice the sales manager would see one variant and would have to decide whether it is the one she wanted — with no way to compare it against the others, and without comments pointing out the differences.

All of them show a list of German customers with the number of their orders from 2023. For customers who did order something in 2023, A, B and C return the same result. Differences only appear when there is no order in 2023 — or, in the case of variant D, when an order falls on 31 December 2023.

A — the intended query. Every German customer is in the list, with the number of their 2023 orders next to them. Anyone who ordered nothing in 2023 shows up with a 0 — exactly as the question was meant.

B — the customers without orders in 2023 are missing entirely. The date condition sits in the WHERE instead of the join condition. The outer join first does the right thing: for every customer without an order it produces a row and fills the order columns with NULL. Then the WHERE checks whether that NULL falls within 2023 — and a comparison with NULL never comes out as "true". The row is removed from the result. The outer join has effectively become an inner join.

C — customers without orders in 2023 are reported with a count of 1 instead of 0. count(*) counts rows, not orders — and the outer join has already produced a row for this customer in which all order columns are NULL. That one row is counted. count(o.order_id) in variant A counts values instead, and skips NULL.

D — 31 December is left out. BETWEEN DATE '2023-01-01' AND DATE '2023-12-31' reads like "the whole year", but means "up to 31 December, 00:00" as soon as the column carries a time — all orders from that day are then missing.

Variant B has the largest error, but with experience it can probably be identified as wrong, because in practice there is almost always at least one customer who ordered nothing.

And now the counter-check: a description in natural language, of the kind a generator could supply for all four variants:

Number of orders per German customer in 2023, including customers without orders, sorted in descending alphabetical order.

To be fair: a good generator can describe the variants differently, and some do.

In this example, the errors in variants C and D are of little practical consequence.

When the same error starts to matter

Same structure, a different example. A company has to give its employees a safety briefing every year, and two decisions hang on those briefings: anyone who has completed at least the first session may be put to work. Anyone who has all four modules may work in a designated position. The question put to the analytics tool reads:

Show me, for every employee, the number of briefings completed by the cut-off date — including those who completed none.

The construction is the same as above: a list of people, an optional link to the training records, a count, a date range. Only here the thresholds are not a matter of judgment, they govern how the operation runs.

B — the untrained disappear. Exactly the people the analysis was made for are missing from the list. What remains reads like a complete workforce in which everyone has been briefed.

C — 0 becomes 1. Anyone without a single briefing shows up with one, and thereby crosses precisely the threshold that decides deployment. They get scheduled, and the documentation records a briefing that never took place. Between zero and one there is no counting error here, there is the line between "may work" and "may not work". Ask the question the other way round — "show me everyone without a briefing" — and write HAVING count(*) = 0 for it, and you get an empty list back: not because everyone has been trained, but because a group with no rows cannot exist in the first place. An empty list of deficiencies reads like a clean bill of health.

D — 4 becomes 3. Briefings are rarely completed early; the stragglers catch up shortly before the cut-off date. It is precisely their last session that falls outside the range. Whoever completed the fourth module on the cut-off date itself appears in the list with three and may not take the designated position, even though the requirement was met. Here the difference between three and four weighs just as heavily as the one between zero and one did before.

As long as a number measures a quantity, an error of one is noise. As soon as a threshold hangs on it — deployed or not, cleared or not — the same error tips a decision. And both errors sit exactly where the thresholds are: C at the line between none and one, D on the last day of the deadline, the day the cases pile up.

What verifiability requires

Four requirements can be derived from the example:

  1. Authority. What is checked has to be what determines the execution — not a description alongside it that can drift away from it.
  2. Unambiguity. Exactly one reading, no room for interpretation.
  3. Brevity. Short enough that it actually gets read and not just skimmed.
  4. Readable without SQL knowledge. Reading is a considerably lower bar than writing.

That is not a law of nature, it is the list of requirements I started from. Anyone who draws up a different one will arrive at a different solution — and arguing about that list is more productive than arguing about benchmark percentages.

The same query, in a language that is easy to check

Here is the query in KQL, a query language whose goal is to be easy to check:

FIND customers c, c + orders o
FILTER c.country = 'Germany'
  AND o.order_date BETWEEN "2023-01-01" AND "2023-12-31"
FETCH c.company_name DESC, count(o)

Variant A above is what these four lines compile to — not a reconstruction for this article, but the output the transpiler produces, checked by a test case in the repository. You can run the comparison yourself.

The difference between variants A and B is a single character here. The + in c + orders o means "including those without" — and that is the entire vocabulary the sales manager has to learn for this. She has to be able to read the query, not write it. The AI wrote it.

We can make it easier to understand still. Because the business terms come from the semantic model, the same query can also be phrased like this for German-speaking users — only the entity and attribute names change, the keywords of the language stay as they are:

FIND kunden c, c + bestellungen o
FILTER c.land = 'Germany' 
  AND o.bestelldatum BETWEEN "2023-01-01" AND "2023-12-31"
FETCH c.firma DESC, count(o)

Four lines, in the vocabulary of the business.

How koryki does it

Joins come from the model. Besides the different syntax, one thing stands out: KQL names no join criteria for the entities customers and orders here — how can that be?

koryki requires a semantic layer in which the permitted joins are defined in advance: which entities may be joined at all, and on which columns. All four SQL variants carried ON c.customer_id = o.customer_id; in the KQL that pair appears nowhere.

As long as the join between two entities is unambiguous, it can be omitted from the query entirely. If it is not unambiguous, it has to be named — named, not spelled out. In KQL such a relationship is called a link, and what is named here is its link name. The example would then look like this:

FIND kunden c, c VIA gleicher_kunde + bestellungen o
FILTER c.land = 'Germany' 
  AND o.bestelldatum BETWEEN "2023-01-01" AND "2023-12-31"
FETCH c.firma DESC, count(o)

The keyword VIA precedes the link name — it selects one of the relationships defined in the semantic layer, not columns. As a result the query can do without any information about the join itself. The + for optional joins is evaluated and the associated filters are assigned to either WHERE or HAVING. If a filter cannot be assigned unambiguously, this is detected as an error and execution is aborted.

Less language, more readability. The next reason why KQL can be simpler than SQL — reduction. KQL does what is needed most often in daily work:

  • filtering
  • linking
  • grouping
  • aggregating
  • sorting
  • date ranges
  • analytical functions
  • set operations
  • existence checks
  • sub-queries, including recursive ones

That makes a substantially simpler grammar possible: fewer keywords, simpler structure, better comprehensibility.

This is a deliberate decision; every additional feature comes at the expense of brevity and readability. If features beyond that are needed, SQL is the right way to go.

The idea is not new. Markdown can do considerably less than LaTeX or HTML — and that is exactly why it is read and written by people who have no need for a typesetting system, and Markdown never set out to replace LaTeX or HTML. But the comparison only goes so far: a misplaced paragraph is annoying, a wrong query can have serious consequences.

Grouping happens automatically. koryki implements the rules for handling aggregation. The function count is recognised as an aggregation, and the translation into SQL accounts for that in the GROUP BY for the non-aggregated outputs. This happens automatically; the user does not have to write it down, or even check it. Sorting is specified optionally after the output definition in the FETCH, and the ORDER BY clause of the SQL follows from it.

The mechanical errors from the SQL variants are gone. Where a condition belongs, how a count is formed, how an interval boundary is read — none of that is decided by the AI in koryki, it follows from the semantic layer and the rules of the transpiler. The semantic layer is predefined, the rules are documented and covered by tests — you don't need to know them in order to read the four lines above, but you can look them up, and they are deterministic.

What does not go away is the intent. Whether the customers without orders belong in the list is still the generator's decision: it writes the +, or it doesn't. The difference is that this decision is no longer buried in the placement of a clause. It is one character, in a line the sales manager reads herself.

One objection remains. What the database executes is SQL, not KQL — so isn't the KQL just another description sitting next to it? It is not, and the difference is the direction. An explanation is derived from the SQL and ends there: nothing depends on it, and if it is wrong the query still runs exactly as before. The KQL sits in front of the SQL, and the SQL is produced from it, deterministically and by documented rules. Reading the KQL is therefore not reading a description of the execution, it is reading what determines the execution. Nobody argues that reviewing C source code is pointless because the processor runs machine code; it is deterministic compilation that makes the source authoritative.

That covers the most important concepts in koryki — no magic, no vagueness. Anyone who wants to know more precisely can look into the documentation or the source code.

How well are the requirements met?

  1. Authority. This text determines the SQL that runs; there is no second version beside it.
  2. Unambiguity. koryki transforms it into SQL by rules, unambiguously.
  3. Brevity. Considerably shorter than the corresponding SQL from variant A.
  4. Readable without SQL knowledge. This remains a matter of judgment and will show once domain users work with the language.

There is a user study from NYU Stern, so far not peer-reviewed (Ipeirotis and Zheng, 2025). It compared an NL2SQL system with a conventional SQL interface and found clear advantages for generation — higher success rate, more trust, lower cognitive load.

The decisive detail is the sample: all participants knew SQL. What was measured, then, is the combination of a generated query and a person who can read that query — and this combination works.

My assumption is that what matters here is less SQL than readability: if a formal language is simple enough, a domain expert should have the same advantage — for the same reasons.

Whether this also holds for KQL remains an open question for now.

Conclusion

Trust does not come from a system being right more often. It comes from someone being able to check it. Checking SQL requires expert knowledge, and it creates new sources of error and new costs as soon as further people are brought in. As long as the checking step takes place in the same ambiguous language the question was asked in, the domain expert depends on the system's explanation — and a well-phrased explanation increases her acceptance, but whether it improves the quality of her check remains open.

That was the starting point for koryki: not a better SQL generator, but a language for queries that are easy to read and easy to check.

Sources