Extracting contamination results from four UK lab PDF formats — and refusing to invent a value when the table would not yield one.
A UK geoenvironmental practice needed contamination results off laboratory Certificates of Analysis and into a screening schedule. Four lab house styles. Same scientific intent — determinands, units, LODs, sample IDs — different geometry on the page. This is the concrete build behind our geotechnical report analysis work.
We tried the usual stack first: lattice detection (needs ruled lines), stream grouping (needs stable gutters), and a generic OCR table pass. The CoAs that mattered were borderless. Headers wrapped onto two lines (“Benzo(a)pyrene” under “PAH-16”). Sample IDs repeated in a left margin that was not a column in the PDF’s drawing order. Lattice found no cells. Stream merged adjacent determinands. OCR invented a decimal place more than once. That is unacceptable when the number becomes a screening exceedance.
We rendered each page with pdf.js, took every text item’s x, y, width and string, and clustered x-centres into columns. Rows were bands in y. Wrapped headers were stitched by proximity and a small synonym list (As / arsenic, TPH CWG bands). Units stayed attached to the result token, not to the next column’s LOD. When two clusters overlapped within a tolerance, the page went to the exception queue instead of a merged fantasy column.
We hand-verified a gold set: 148 cells across the four formats — sample ID, determinand, result, unit, LOD where present. The extractor had to match 148/148 on that set before we would screen anything. Misses were classification errors (wrong column), not “close enough.” The gold set is the product contract.
A second, independently written pass (different grouping heuristic) had to agree on every certified cell. Disagreement → exception, not average. Dual-parse is cheaper than a contaminated land report that cites a number nobody typed.
If the system cannot certify a cell, it does not emit a mg/kg value. Consultants already know how to read a smeared scan. They should not have to discover that the software invented 12 mg/kg of lead. That rule is also the spine of our document intelligence work.
More on this kind of work: geotechnical report analysis, document intelligence and the compliance software overview. Most builds start as a 30-day first slice.
We wrote up the general approach, including AGS4 output and validation, in how to extract AGS data from lab PDFs.