# Pakistan Provincial Textbook Corpus A locally-held, comparison-ready corpus of school textbooks from all four of Pakistan's provincial textbook boards, with extracted text, a seven-axis scored analysis, and a static research site. **163 PDFs · ~6.7 GB · classes 6–10 · English medium preferred throughout** 120 current textbooks across four boards, all scored, plus 43 historical Punjab editions (1949–2005) held to document curricular change and deliberately left unscored. ## Boards | Directory | Board | Books | |---|---|---| | `pctb-punjab/` | Punjab Curriculum and Textbook Board, Lahore | 32 | | `stbb-sindh/` | Sindh Textbook Board, Jamshoro | 27 | | `kptbb-kp/` | Khyber Pakhtunkhwa Textbook Board, Peshawar | 33 | | `btbb-balochistan/` | Balochistan Textbook Board, Quetta | 28 | | `ptb-historic/` | Punjab Textbook Board — withdrawn editions, 1949–2005 | 43 | Files are named `--.pdf` (`-en` English, `-ur` Urdu) inside `/class-/`. 75 current books are English medium, 45 Urdu — Urdu is used where no English-medium edition is published (Islamiyat in Punjab and KP; the Urdu-subject readers everywhere; KP's class 6–8 History and Geography; Sindh's class-6 General Science and class-9 Pakistan Studies). ## Subject coverage - **Classes 6–8:** English, Urdu, Islamiyat, General Science, and Social Studies. Note the boards disagree on what "Social Studies" is: Sindh publishes one integrated volume per class, while Punjab and KP each split it into separate **History** and **Geography** books. - **Class 9:** English, Urdu, Islamiyat, Pakistan Studies, Biology, Chemistry, Physics. - **Class 10:** Pakistan Studies (all three boards) — no board publishes Pakistan Studies below class 9, and Sindh's is a combined 9–10 volume issued separately in each medium. ## Sources Every URL used is recorded in `analysis/sources-*.json` and every download verified as a real PDF. - **Sindh** — the only board with a working official e-book portal: `portal.stbb.edu.pk/ebooks/pdf_proxy.php?id=&download=1`. No headers or auth required. - **Punjab and KP** — neither official site serves textbook PDFs (`pctb.punjab.gov.pk` was unreachable; `tbb.kp.gov.pk` is a React shell whose downloads API carries only tenders and price lists). Books come from free mirrors: literaria.edu.pk, ustad360.com, taleem360.com, awazeinqilab.com, waqasabid.com, and ilmkidunya.com's CDN. - Google-Drive-hosted mirrors can hit a per-file download quota; `download.sh` falls back to the `drive.usercontent.google.com/download?...&confirm=t` form, and directly-hosted copies are preferred where found. ## The OCR problem **Only 12 of 120 current books ship with a usable text layer.** The rest — including current official editions from every board — are scanned page images with no embedded fonts. The pipeline therefore renders and OCRs them: 300 dpi grayscale via `pdftoppm`, then tesseract with the project-local models in `scripts/tessdata/` (English and Urdu, from `tessdata_best`). Quality tiers, which matter for how each book may be used: - **Native text (12 books)** — quotation-grade, exact. Tiers are computed from whole-book text density, not a page sample: four KP class-8 books previously read as "native" carried only a watermark text layer at an identical 86 characters per page, and are correctly scanned. - **English OCR** — quotation-grade with care; typical ligature and column-interleave errors. Punjab books carry a diagonal "Web Version / Not for sale / PCTB" watermark that shreds words mid-line; KP mirror copies carry a `perfect24u.com` watermark and mirror-site ad pages. - **Urdu (Nastaliq) OCR** — **theme-grade only**. Good enough for chapter structure, topics and recognizable names; never quote from it verbatim. All Urdu-medium books fall here. ## Layout ``` /class-/--.pdf the corpus extracted-text//class-/ .txt = native, .ocr.txt = OCR, .scanned-stub = image-only marker analysis/ manifest.csv, manifest.jsonl per-book: pages, producer, extractability sources-*.json verified source URLs per board track-*.md per-subject comparative reports raw-profiles-social-studies.md per-book profiling notes with page cites scripts/ download.sh detect_medium.sh acquisition extract_text.sh ocr.sh text layer build_manifest.sh inventory ``` ## Reproducing ```bash ./scripts/download.sh analysis/sources-stbb.json # verifies %PDF, retries, Drive fallback ./scripts/detect_medium.sh # resolves unknown-medium files empirically ./scripts/extract_text.sh # native text where it exists ./scripts/ocr.sh --all-missing # OCR everything else ./scripts/build_manifest.sh # rebuild the inventory ``` `extract_text.sh` treats a book as scanned when extraction yields fewer than ~250 characters per page. That threshold matters: KP mirror PDFs have a watermark-only text layer that yields a few thousand characters for a whole book and would otherwise pass as genuine text. ## Caveats for anyone using this corpus - **Editions differ by years, not months.** Punjab's books are SNC 2022–2023 and 2025–26 printings; KP's class 9–10 are National Curriculum 2006 books; Sindh's Pakistan Studies is an edition approved in **2004**. Cross-board differences are as much generational as provincial. - Page citations in the analysis refer to PDF page markers (`===== PAGE N =====`) unless a book had native text with printed folios, which is noted per book. - Mirror-sourced files carry no imprint guarantees; edition years are as stated by the mirror or the book's own imprint page, and several KP volumes have no recoverable imprint at all. - Statistics read out of OCR'd tables (reserves, tonnages, populations) are unreliable at the digit level and should be checked against the PDF before being quoted.