Claim: On 554 WikiTableQuestions specs, Tablebeam returns a silently wrong number 0.4% of the time against 13.7% for pandas, 17.2% for SQLite and 11.7% for DuckDB, and it gets there partly by refusing 20.6% of the specs. The committed results at f141ee7 follow from the pinned inputs.
InputPinned CSVs and questions in eval/inputs/, checked by SHA-256
NeedsGit, Python 3.12 with SQLite 3.54.1, no model, no API key
EvidenceRates printed by run_eval.py, an unchanged git status, total-row counts

Prerequisites

  • Git;
  • Python 3.12, ideally with SQLite 3.54.1 (check with step 2);
  • network access for git clone and pip install. The evaluation itself reads only local files.
Nothing in this recipe loads or calls a model.

1. Clone the repository at the evaluated commit

2. Install the pinned versions

eval/requirements.txt pins DuckDB 2.0.0, numpy 2.5.4 and pandas 3.1.0. The last line should print 3.54.1. SQLite comes with your Python build, and another version can change the SQLite column and make the files in step 4 differ.

3. Run the evaluation

It checks every input against eval/inputs/manifest.json, asks Tablebeam’s browser engine (web/engine.py) each documented question, runs the same calculation in pandas, SQLite and DuckDB, and rewrites eval/results.md and eval/results.json. It took 36 seconds on an Apple Silicon laptop.

4. Check the output matches the committed files

This prints nothing when both result files are byte-identical to the ones committed at f141ee7.

5. Count the wrong answers on total-row tables

The script reads each WikiTableQuestions spec’s scores. The grep shows one such spec: the table ends in a Total row, the truth is 36, and pandas, SQLite and DuckDB each return 72.

Expected evidence

  • step 2 prints 3.54.1;
  • step 3 ends with these lines (the wtq rows are the WikiTableQuestions specs, numeric the 162 hard-format CSV specs):
  • step 4 prints nothing;
  • step 5 prints pandas: 76 silent wrong, 32 on tables Tablebeam refuses for a total row, sqlite: 95 ..., 32 ... and duckdb: 65 ..., 32 ..., then the nt-61 row with wrong: 72 three times.
These matched on 23 October 2026 in a fresh clone at f141ee7.

This recipe does not prove

  • that Tablebeam answers more questions correctly. It answers fewer: 79.1% correct against 85.0% for pandas on the same specs, and on the hard-format files it refuses 37.0%, including 60 of the 65 Montpellier specs;
  • anything about pandas, SQLite or DuckDB used with explicit column types or a decimal separator. Every executor runs with its defaults;
  • that people can type these questions. None of the 954 WikiTableQuestions questions parsed in their original wording, and the 554 specs are re-asked in Tablebeam’s documented form;
  • that the live demo behaves this way. On 18 November 2026 it served the same engine modules as main, but this recipe runs only your local clone;
  • anything about the optional Ask AI path, which the evaluation never calls.
Remove the clone when finished: