f141ee7 follow from the pinned inputs.
InputPinned CSVs and questions in
eval/inputs/, checked by SHA-256NeedsGit, Python 3.12 with SQLite 3.54.1, no model, no API key
EvidenceRates printed by
run_eval.py, an unchanged git status, total-row countsPrerequisites
- Git;
- Python 3.12, ideally with SQLite 3.54.1 (check with step 2);
- network access for
git cloneandpip install. The evaluation itself reads only local files.
1. Clone the repository at the evaluated commit
2. Install the pinned versions
eval/requirements.txt pins DuckDB 2.0.0, numpy 2.5.4 and pandas 3.1.0. The last line should print 3.54.1. SQLite comes with your Python build, and another version can change the SQLite column and make the files in step 4 differ.
3. Run the evaluation
eval/inputs/manifest.json, asks Tablebeam’s browser engine (web/engine.py) each documented question, runs the same calculation in pandas, SQLite and DuckDB, and rewrites eval/results.md and eval/results.json. It took 36 seconds on an Apple Silicon laptop.
4. Check the output matches the committed files
f141ee7.
5. Count the wrong answers on total-row tables
grep shows one such spec: the table ends in a Total row, the truth is 36, and pandas, SQLite and DuckDB each return 72.
Expected evidence
- step 2 prints
3.54.1; - step 3 ends with these lines (the
wtqrows are the WikiTableQuestions specs,numericthe 162 hard-format CSV specs):
- step 4 prints nothing;
- step 5 prints
pandas: 76 silent wrong, 32 on tables Tablebeam refuses for a total row,sqlite: 95 ..., 32 ...andduckdb: 65 ..., 32 ..., then thent-61row withwrong: 72three times.
f141ee7.
This recipe does not prove
- that Tablebeam answers more questions correctly. It answers fewer: 79.1% correct against 85.0% for pandas on the same specs, and on the hard-format files it refuses 37.0%, including 60 of the 65 Montpellier specs;
- anything about pandas, SQLite or DuckDB used with explicit column types or a decimal separator. Every executor runs with its defaults;
- that people can type these questions. None of the 954 WikiTableQuestions questions parsed in their original wording, and the 554 specs are re-asked in Tablebeam’s documented form;
- that the live demo behaves this way. On 18 November 2026 it served the same engine modules as
main, but this recipe runs only your local clone; - anything about the optional Ask AI path, which the evaluation never calls.