Corpus version 210 records · 2026-08-23

The data

Name: Pakistan Provincial Textbook Evidence Corpus
Creator: Textbook Evidence Project
License: https://creativecommons.org/licenses/by/4.0/

Every figure on this site is computed from the files below. They are published so that anyone who doubts a number can recompute it, and so that anyone who wants to extend the study does not have to rebuild the catalogue first.

Downloads

File	Contains	Format
data.json	Everything: 210 catalogue records with scores and evidence, 46 findings, the seven axis definitions, and the annotated bibliography.	JSON
data.csv	The catalogue as a flat table — one row per book, one column per axis. For spreadsheets.	CSV
`/books/<id>.json`	A single record, e.g. kptbb-physics-10-en.json. Every book page has one at its own address plus `.json`.	JSON

Licence

Everything this project produced is CC BY 4.0. Use it, quote it, republish it, build on it commercially — attribute it to the Textbook Evidence Project and link back so a reader can check the version you used.

The textbook PDFs are not included and are not ours to license. They carry explicit reservations of rights from the boards that publish them. Each record carries the bibliographic detail and a link to the source we obtained it from, so you can retrieve the original yourself. That distinction is set out in full in about.

What you are getting, and what it will not support

Catalogue records	210 — 167 current, 43 historical
Carrying at least one score	137 (historical editions are never scored)
Findings	46
Corpus version	210 records, generated 2026-08-23

Scores are measurements, not verdicts. Every axis runs 1–5 and 1 is the secular, pluralist or open pole. A high number records a property of a text; it is not a judgement of a board, an author or a teacher.
Check the n before quoting any mean. Axis coverage is deliberately uneven — a lens scored the books where its question is live. A4 rests on as few as one book for some boards and must not be quoted as a board figure. The axes pages print every denominator.
Nothing is quotable from Urdu or Sindhi records. Those extractions are theme-grade: good for structure, chapter coverage and proper nouns, never for wording. Roughly a quarter of the extracted text falls here.
Historical records are not comparable to current ones. They are catalogued to document change and carry no scores by design.
The acquired date is a local retrieval date, not a formal web-archive capture. Source links rot; the bibliographic record is what identifies a book. Every record also carries a sha256 of the analysed file, so you can confirm you are working from identical bytes.

Reproducing the study from scratch

The pipeline is a handful of shell and Python scripts, and it is deterministic apart from the boards reprinting under you. In order: acquire from the recorded source URLs with a magic-byte and trailer check; resolve each book's medium empirically rather than from its filename; extract native text where a publisher's text layer exists; OCR the rest at 300 dpi with language-specific models; rebuild the manifest; assemble the dataset; then build the site, which regenerates all of its own content from the analysis directory.

The pipeline itself

Served as plain text so they can be read in the browser. These are the exact scripts that produced everything above.

Artefact	What it does
download.sh	Acquisition. Verifies the magic bytes and the trailer, because one origin in this project appended an HTML error page mid-stream to a file that otherwise looked complete.
detect_medium.sh	Resolves each book's medium empirically rather than trusting its filename.
extract_text.sh	Native text where a publisher's layer exists.
ocr.sh	300 dpi rendering and OCR with language-specific models. Refuses to run on a book whose medium is unresolved, and refuses to write an extraction under 200 characters.
build_manifest.sh	Inventory: pages, producer, extractability, per PDF.
build_dataset.py	Assembles the dataset — vintage table, lens-table score parsing, evidence classification, checksums.
sync_site.sh · sync_studies.py	Regenerates everything the site reads from the analysis directory, so the two cannot drift.
README · editorial guideline · URL policy	The written rules the project holds itself to, including the identifier scheme.

Two failure modes are worth knowing before you run it. A PDF can be a complete-looking file and still be truncated — one origin in this project appended an HTML error page mid-stream — so completeness is verified by parsing the page tree, never by file size. And OCR must never be run on a book whose medium is unresolved, because the wrong language model produces confident noise rather than an obvious failure. Both checks are in the scripts.

Citing the dataset

Textbook Evidence Project, Pakistan Provincial Textbook Evidence Corpus, corpus version 210 records, 2026-08-23. CC BY 4.0. Retrieved from /data.json

Cite the corpus version. The catalogue grows and findings are revised; a citation without a version cannot be checked against what you actually read.

← How the data was produced · corrections log →