Extract AGS Data From Lab PDFs: A Practical UK Guide
Lab results still arrive as PDFs that look like spreadsheets and behave like posters. Here is how to get certificate of analysis data into AGS4 without re-keying it, and without letting software invent a number.
First ask the lab for an AGS file. If you only have the PDF, extract sample ID, determinand, result, unit and detection limit per cell with a layout-aware parser that reads word positions, not ruled lines. Cross-check with a second independent pass and send any uncertain cell to a person instead of guessing. Map chemistry to the ELRG group (ERES was removed in AGS 4.2), keep geotechnical tests in their own groups, then run an AGS validator. A validator checks format, not values, so reconcile against a hand-checked gold set too.
Why Lab Results Still Arrive as PDFs
A UK site investigation sends soil and groundwater samples to a laboratory, often an MCERTS-accredited one for chemistry. What comes back is a Certificate of Analysis: sample IDs, matrices, determinands, methods, limits of detection and results. It is usually a PDF, because the PDF is the signed, issued record. The AGS itself says as much: the notes to the ELRG group in the AGS 4.2 document describe the lab certificate as "currently provided in Adobe Portable Document Format".
The consultant then needs those results somewhere useful: a screening schedule against Generic Assessment Criteria, S4ULs or C4SLs, a project database, or an AGS4 file for the client. The bottleneck is not the science. It is moving arsenic, TPH CWG bands, PAH-16, asbestos, pH and SOM from several lab house styles into one table without a column shifting.
Re-keying is the default, and its risk is specific: a typing slip does not look like an error. A value of 1.2 typed as 12 mg/kg is still a believable lead result, and it can create or hide an exceedance. Nobody notices until someone reconciles the schedule against the certificate line by line.
What an AGS4 File Actually Is
The AGS Format is the Association of Geotechnical and Geoenvironmental Specialists' standard for moving ground investigation, laboratory and monitoring data between organisations without re-keying, independent of any software. The first edition was published in 1992; the current one is AGS 4.2, issued December 2025.
An AGS4 file is plain text in a strict CSV layout. Each GROUP (a four-character name such as PROJ, LOCA or SAMP) is introduced by header rows, in this order: a GROUP row, a HEADING row, a UNIT row and a TYPE row, then one or more DATA rows. Every value is in double quotes. Headings are defined as KEY, REQUIRED or OTHER. Every child record's key fields must match a record in its parent group, so a lab result must point to a sample that exists in SAMP, which in turn points to a location in LOCA.
Five groups must appear in every file: PROJ (project), TRAN (transmission details, including the AGS edition in TRAN_AGS), ABBR (abbreviations), TYPE (data types) and UNIT (units). Laboratory test groups hang off SAMP; in situ tests hang off LOCA.
Illustrative fragment only. Not a complete or valid file: it omits PROJ, TRAN, ABBR, TYPE, UNIT and LOCA, and the determinand codes are placeholders, not real ELRG codes.
"GROUP","SAMP"
"HEADING","LOCA_ID","SAMP_TOP","SAMP_REF","SAMP_TYPE","SAMP_ID"
"UNIT","","m","","",""
"TYPE","ID","2DP","X","PA","ID"
"DATA","BH01","0.50","ES1","ES","BH01-ES1"
"GROUP","ELRG"
"HEADING","LOCA_ID","SAMP_TOP","SAMP_REF","SAMP_TYPE","SAMP_ID","SPEC_REF","SPEC_DPTH","ELRG_CODE","ELRG_METH","ELRG_MATX","ELRG_RTYP","ELRG_TADE","ELRG_TICN","ELRG_RUNI","ELRG_LQLF","ELRG_RVAL","ELRG_RTXT","ELRG_NAME","ELRG_DETF","ELRG_RDLM"
"UNIT","","m","","","","","m","","","","","","","","","","","","",""
"TYPE","ID","2DP","X","PA","ID","X","2DP","PA","X","PA","PA","PA","X","PU","X","U","X","X","YN","U"
"DATA","BH01","0.50","ES1","ES","BH01-ES1","","","CODE-A","LAB-METHOD-1","SOLID","INITIAL","","","mg/kg","","42","42","Lead","Y","1"
"DATA","BH01","0.50","ES1","ES","BH01-ES1","","","CODE-B","LAB-METHOD-2","SOLID","INITIAL","","","mg/kg","<","","<0.05","Benzo(a)pyrene","N","0.05"Note the second row's "less than detection" result: the text as reported goes in ELRG_RTXT, the qualifier in ELRG_LQLF and the reporting limit in ELRG_RDLM. The AGS says providers and receivers must agree the ELRG specification at the outset, so do not let an extractor decide that split silently.
Extraction Approaches Compared
There are five realistic approaches. None is risk-free; the question is where the risk lands and whether you can see it.
| Approach | Accuracy risk | Effort | When it fits |
|---|---|---|---|
| Manual entry | Silent typing slips that look like real values; depends on a second person checking | Low to set up, high per report, forever | Occasional small jobs; one-off scans |
| PDF table parsers (lattice / stream) | Lattice finds nothing without ruled lines; stream merges adjacent columns when gutters are narrow | Low to try, rising fast as you patch each layout | Ruled, consistent tables from one lab |
| OCR | Highest. Can misread or invent digits and decimal points | Medium | Scanned certificates only, and only with a cross-check |
| LLM extraction | Plausible values that are not on the page, with no warning | Low to demo, high to make trustworthy | Proposing column mappings and synonyms for a human or parser to confirm |
| Layout-aware parsing with cross-checks | Lowest that we have found, because uncertain cells are refused rather than guessed | Highest up front: templates, gold set, exception queue | Recurring volume from a known set of labs |
We learned this order the hard way on SI Report Analyser, a build for a UK geoenvironmental practice that needed contamination results from four lab house styles in one screening schedule. We tried lattice detection, stream grouping and a generic OCR table pass first. The certificates that mattered were borderless: lattice found no cells, stream merged adjacent determinands, and OCR invented a decimal place more than once.
What worked was going one level lower. We rendered each page with pdf.js, took every text item's x, y, width and string, clustered the x-centres into columns and treated bands of y as rows. The table is rebuilt from geometry, not from the PDF's idea of a table.
Where Lab PDF Extraction Breaks
These are the failure modes that matter on UK certificates. Most break generic parsers quietly.
- Borderless columns. No ruled lines, so anything looking for cell borders sees one block of text. Columns exist only as gaps between words.
- Merged and wrapped headers. A determinand name wraps onto two lines, or sits under a group header, such as "Benzo(a)pyrene" under "PAH-16". Read naively, you get two half-headers or a header attached to the wrong column.
- Units and detection limits. mg/kg, µg/kg and % w/w can share one certificate. Results like "<0.05" carry a qualifier that is not a number, and the LOD column sits beside the result, so a sloppy parser attaches the unit or limit to the wrong value.
- Multiple samples per page. Several samples share a page and sample IDs repeat in a left margin which, on the certificates we parsed, was not a column in the PDF's drawing order at all.
- Lab-specific layouts. Every lab has a house style with the same scientific intent and different geometry on the page. On top of the regular formats there is always the one-off scan from years ago. A template tuned to one lab tells you nothing about the next.
- Synonyms. "As" and "arsenic" are one determinand, and the TPH CWG bands need the same care. Unresolved synonyms turn one determinand into several.
A blank cell gets chased. An invented 12 mg/kg of lead does not. Consultants can read a smeared scan; they should not have to discover that software made up a number.
Guards That Make It Safe
Each failure mode above has a matching guard. The principle behind all of them is the same: refuse to guess. If the system cannot certify a cell, it does not emit a value. That rule is the spine of our document intelligence work.
- Keep units attached to results. The unit stays with the result token, not the next column's LOD. Normalise mg/kg, µg/kg and % w/w before any comparison or export.
- Send overlaps to an exception queue. When two column clusters overlap within a tolerance, the page goes to a person instead of producing a merged column.
- Stitch headers explicitly. Join wrapped headers by proximity, and resolve names through a small synonym list (As and arsenic, the TPH CWG bands) rather than letting a model guess.
- Dual parsing. A second, independently written pass with a different grouping heuristic has to agree on every certified cell. If they disagree, the cell becomes an exception. Never average them.
- A gold set as the contract. On SI Report Analyser we hand-verified 148 cells across the four formats: sample ID, determinand, result, unit, and LOD where present. The extractor had to match all 148 before we screened anything. The misses we fixed were classification errors, a value in the wrong column, not "close enough".
- Record who accepted each cell. When a person resolves an exception, log who and when. That audit trail makes the output defensible.
Mapping Extracted Values Into AGS Groups
Once each cell is certified, mapping is mostly bookkeeping, with a few traps.
Use ELRG, not ERES, for chemistry. Older guides still talk about ERES. AGS 4.1 introduced ELRG (Environmental Laboratory Reporting) to replace it, and the AGS 4.2 document says ERES has been removed and ELRG should be used instead. Older 4.0.x files will contain ERES; new files should use ELRG and declare the edition in TRAN_AGS.
Key fields come from SAMP. An ELRG row carries the sample keys (LOCA_ID, SAMP_TOP, SAMP_REF, SAMP_TYPE, SAMP_ID) plus specimen fields and its own keys, including ELRG_CODE, ELRG_METH, ELRG_MATX and ELRG_RUNI. Those sample keys must match a SAMP record, and every SAMP must match a LOCA. The lab's own sample reference may differ from the consultant's sample ID, so build that cross-reference deliberately; ELRG_LSID holds the lab's ID alongside.
Determinand codes come from the AGS list. ELRG_CODE is an abbreviation type drawn from the geoenvironmental code list on the AGS website. Map your synonym table to those codes rather than inventing your own.
Keep geotechnical tests out of ELRG. The AGS is explicit that geotechnical chemistry (for example water-soluble sulfate to BS 1377) goes in GCHM, and that geotechnical tests run by an environmental lab, such as Atterberg limits, go in their own group (LLPL), not in ELRG. Other geotechnical lab sheets have their own groups too: moisture content (LNMC), particle size distribution (GRAG/GRAT), triaxial (TRIG/TRIT, TREG/TRET), shear box (SHBG/SHBT), compaction (CMPG/CMPT) and consolidation (CONG/CONS).
Units go in UNIT; abbreviations go in ABBR. Every unit used in ELRG_RUNI must be defined in the UNIT group, and where there is no unit the AGS says to use "UNITLESS". Every abbreviation used in a PA field, such as sample type, matrix and determinand code, must be listed in ABBR.
Validating the Output
Format validation. The AGS Data Format Working Group maintains an open-source Python library, published on PyPI as python-ags4. It reads, writes and checks AGS4 files, and its command-line tool (ags4_cli) can validate and convert files. The AGS publishes a free Windows desktop AGS Validator built on that library, which validates against AGS 4.0.3, 4.0.4, 4.1.0, 4.1.1 and 4.2. The British Geological Survey runs a web validator (the open-source pyagsapi project) that also checks files against BGS data submission requirements and converts between .ags and .xlsx. Run one on every file before it leaves the building.
Content reconciliation. No format validator knows what the certificate said; a file can pass every rule and still hold the wrong lead result. That is what the gold set and dual parser are for. Re-run the gold set whenever you add a lab template or change a parsing rule, and treat any regression as a blocker.
Extract, certify (dual parse plus gold set), resolve exceptions with a named reviewer, map to groups, write the file, validate it, then spot-check rows against the PDF. If any step fails, the file does not go out.
Build vs Buy
Start by asking the lab for AGS output; that beats any extraction you build. If certificates come in volume from a stable handful of labs, a custom extractor is worth costing against the hours spent re-keying and reconciling. If every job brings one-off scans from a different lab, careful manual entry with a second checker may honestly be right.
When evaluating a product, ask three questions. When it cannot read a cell, does it flag it or fill it in? Can it show the source position on the page for every value? Will it run against your own gold set before you commit? A demo on a clean, ruled certificate tells you nothing about borderless ones.
If you build, the shape we use for geotechnical report analysis is a bounded corpus (typically the four lab formats you actually receive), a hand-verified gold set, layout-aware parsing, dual checking and an exception queue. With known labs, a first slice takes about 30 days, and adding a fifth lab later is not a rewrite. The output can be a screening schedule, an AGS4 file or both: the hard part, certifying each cell, is the same.
Bring one lab PDF
Send the certificate your team dreads re-keying. On a 30-minute call we will tell you honestly whether it can be extracted to AGS4 with a gold set, or whether it should stay manual.
Book a 30-Minute Call →FAQ
What is the AGS file format used for?
It is the UK standard for transferring ground investigation, laboratory testing and monitoring data between organisations as plain-text .ags files, independent of any software. Contractors, labs and consultants use it to exchange logs, sample records and test results without re-keying. The current edition is AGS 4.2, issued December 2025.
Can I use an LLM to extract data from a PDF?
Yes, and a language model is good at reading messy layouts and suggesting which column is which. But it can also return a plausible number that is not on the page, without warning. Every value must be verified against the page text, and the system should fail closed: a value that cannot be confirmed goes to a person, not into the AGS file. Let the model propose a mapping, not write a concentration.
Is ERES still used in AGS4?
Not in the current edition. AGS 4.1 introduced ELRG to replace ERES, and AGS 4.2 states that ERES has been removed. Older 4.0.x files still contain ERES, so a converter may need to read both.
How do I check an AGS4 file is valid?
Use a validator built on the AGS working group's python-ags4 library: the free AGS Validator desktop app, the BGS web validator at agsapi.bgs.ac.uk, or the library itself. These check format rules and the dictionary, not whether the numbers match the certificate, so you still need your own reconciliation.
Will OCR work on scanned lab certificates?
Sometimes, but treat it as the highest-risk route. On the CoAs we worked with, a generic OCR table pass invented a decimal place more than once. Cross-check OCR output with a second method or a person before any value is screened or exported.
How long does it take to build a lab PDF to AGS extractor?
For a bounded corpus where the labs are known, we scope a first working slice in 30 days. The time goes into the gold set and exception handling, not the happy path. Adding another lab later is retainer work, not a rewrite.
Sources
Checked October 2026. AGS format details are taken from the AGS 4.2 document and the AGS and BGS pages below. Project details come from our own build, described in the SI Report Analyser case study.
- AGS: Data Format overview, ags.org.uk
- AGS 4.2 release page, ags.org.uk
- Electronic Transfer of Geotechnical and Geoenvironmental Data, AGS4 Edition 4.2 (December 2025), PDF: foreword (ERES removed, ELRG used instead), section 3 (group hierarchy, group list, ELRG and GCHM definitions), section 4.1.1 (CSV file rules)
- AGS Validator, ags.org.uk
- AGS Python library (python-ags4), GitLab, and python-ags4 on PyPI
- python-ags4: A Python library to read, write, and validate AGS4 geodata files, Journal of Open Source Software
- pyagsapi, British Geological Survey, GitHub; hosted at agsapi.bgs.ac.uk