LLM-driven materials knowledge extraction: multimodal parsing, ontology, and agentic systems

Evolution of technical paradigms

Early efforts in scientific document parsing were dominated by modular pipeline architectures, which decomposed the task into PDF text extraction, OCR, layout detection, table recognition, figure extraction, and rule-based or statistical entity extraction. In chemistry and materials science, OSCAR4 provided chemical tokenization and named-entity recognition, ChemicalTagger parsed experimental language into phrase-level semantics, and ChemDataExtractor coupled text and table parsing to populate chemical and materials records^[3-6]. These systems established the importance of domain-specific grammar and dictionaries, but their staged architecture made them vulnerable to error propagation. A missed caption, a split table cell, or an incorrectly recognized chemical formula can propagate into the final database without an explicit mechanism for later correction^[23].

Between 2019 and 2022, the field entered a dataset-driven deep-learning era for layout and table understanding [Figure 3A]. PubLayNet and DocBank enabled layout detectors to recognize regions such as text, titles, lists, figures, and tables at scale^[24,25], while SciTSR, PubTabNet, and PubTables-1M supported table-structure recognition using metrics such as tree-edit-distance similarity (TEDS) and cell-level F1^[26-28]. These datasets improved modular evaluation and made the perception layer more reproducible, but they did not solve the scientific grounding problem. Recognizing a table cell is different from understanding that the value belongs to a specific alloy composition under a specific annealing schedule. Similarly, recognizing a plotted curve is different from binding it to the correct legend entry and sample code.

Figure 3. Two document-parsing paradigms for materials-science papers. (A) Modular OCR- and rule-based pipelines separate layout, table, chart, and chemical-structure extraction; (B) Unified end-to-end multimodal LLM parsing directly maps page images or parsed blocks to machine-readable structured output. The two routes differ in controllability, error propagation, cost, and traceability. OCR: Optical character recognition; LLM: large language model.

From 2022 onward, two converging research directions reshaped the technical landscape. The first involved document-specific multimodal Transformers, exemplified by LayoutLMv3, which unified text, layout, and visual features through joint pre-training objectives^[29] [Figure 3B]. The second pursued OCR-free, end-to-end image-to-structure generation, as demonstrated by Donut and Pix2Struct, which directly map document images to structured representations^[30,31]. Since 2024, document-focused vision-language models (VLMs) such as DocLayLLM and DocVLM have further targeted the ‘seeing but not reading closely’ limitation of general vision-language models by improving structural resolution and efficiency for text-dense pages^[32-34]. These advances move the field from loosely connected modules toward end-to-end perception systems, but they also make faithfulness evaluation more important because generative parsers can produce plausible structure even when the source page is ambiguous.

Beyond representative academic models, these paradigms have rapidly translated into a diverse engineering ecosystem. Rather than enumerating incommensurable implementations spanning research models, commercial application programming interfaces (APIs), OCR engines, and format-conversion utilities, we adopt a functional taxonomy that groups tools by their architectural trade-offs. (1) Modular pipeline tools (e.g., MinerU, Marker, Unstructured.io) prioritize throughput and deployability, serving as the workhorse for large-scale ingestion, but inherit the error propagation and weak cross-modal integration of staged architectures; (2) General-purpose vision–language models (e.g., GPT, Qwen-VL, Gemini, InternVL families) offer strong zero-shot layout and table understanding across document types, at the cost of higher inference expense, limited fine-grained numerical fidelity, and weaker traceability; (3) Domain-specialized document VLMs (e.g., dedicated scientific OCR/parsing models) trade generality for higher structural resolution on text-dense pages. The optimal choice is dictated by the target balance among accuracy, cost, controllability, and traceability rather than by raw capability alone. A categorized inventory is maintained in the project repository (https://github.com/sShuaiYang/AI4Mat_KnowExtraction). Across all categories, challenges of system stability, interpretability, result traceability, and standardized evaluation remain open^[22].

In materials informatics, parsing should be evaluated by how well the evidence structure is preserved, not merely visual reconstruction quality. A parser that perfectly reads a table but loses multi-row header semantics may fail for catalysis or battery data; a chart reader that extracts a curve accurately but ignores the caption may bind data to the wrong sample; and an optical chemical structure recognition (OCSR) model outputting a plausible SMILES string must still be verified against the source image and downstream chemical validity. These considerations motivate the modality-specific discussion in Section “Key subtasks and domain-specific practices in materials science” and the reliability framework in Section “Reliability, Quality Control, and Evaluation”.

Key subtasks and domain-specific practices in materials science

Multimodal parsing can be organized around five interdependent subtasks. Layout understanding provides the spatial and logical scaffold for identifying sections, captions, tables, figures, equations, and supplementary links, enabled by large-scale datasets such as PubLayNet^[24] and DocBank^[25]. Table parsing recovers nested headers, merged cells, units, and condition-dependent columns; materials-focused approaches such as MaTableGPT^[9] report F1 up to 96.8% on water-splitting catalysis tables, while graph-based frameworks like MatSKRAFT^[14] encode scientific constraints for high-throughput processing. Chart and curve parsing extracts quantitative trends from phase diagrams, thermal-analysis curves, and electrochemical impedance spectra^[35]; general tools such as DePlot^[36] convert plots to approximate tables, but materials-specific benchmarks like MatViX^[37] evaluate curve similarity and hierarchical alignment rather than plain text overlap. Recent advances include PlotExtract^[38], which recovers two-axis plot data with over 90% precision/recall and ~5% positional error via chain-of-thought prompting, and DIVE^[17], a multi-agent workflow that improves figure-based extraction accuracy and coverage by 10%-15% over commercial models and > 30% over open-source baselines in solid-state hydrogen-storage literature. OCSR converts molecular depictions into machine-readable SMILES or InChI; recent models such as MolParser^[39], GTR-CoT^[40], and MolSight^[41] incorporate graph-based visual reasoning for reliable end-to-end conversion^[42]. Cross-modal alignment binds these elements to each other and to surrounding text so that extracted values remain scientifically interpretable, as benchmarked by DocGenome^[43], MMDocIR^[44], and M4DocBench^[45].

Materials literature makes these subtasks unusually demanding because scientific meaning is distributed rather than localized. A conductivity value in a plot may depend on a synthesis route described in the Methods section and on a composition code defined only in a table or caption. Figure–caption alignment is therefore a critical bottleneck^[46,47]: captions define sample identities, axis semantics, units, and measurement conditions, while in-text references (e.g., “Figure 3B”) connect the visual element to the surrounding argument. Pipeline-based workflows such as that of Sror et al.^[48] and multi-agent readers such as DIVE^[17] explicitly exploit caption grounding to mitigate misattribution errors, where a correctly read value is assigned to the wrong material or condition. Integrated pipelines such as KnowMat^[49] further combine specialized parsers, cross-modal verification, and standardized ingestion workflows for end-to-end materials knowledge extraction.

Parser faithfulness should be evaluated with task-matched, modality-specific metrics rather than a single aggregate score: layout detection with mAP/IoU^[24,25]; table recognition with TEDS and cell-level F1^[26-28]; OCR-free conversion with edit-distance or character accuracy; and chart extraction with curve similarity or tolerance-based positional error. Reporting modality-specific results is essential - Sror et al. report F1 above 99% for text and tables but only 85.8% for figures on the same 9,600-entry catalytic-performance corpus^[48], quantifying a persistent accuracy gap between visual and textual modalities. Because generative parsers are stochastic, these metrics should be paired with procedural safeguards: figure pre-extraction, source prioritization across modalities, repeatability checks (~87% run-to-run agreement^[48]), and DOI/page-level provenance for every parsed value. These parser-level checks feed into the downstream content-level quality-control mechanisms detailed in Section “Reliability, Quality Control, and Evaluation”.

Top-down approach: LLM-empowered collaborative ontology modeling

The top-down route starts with explicit domain requirements, mature scientific abstractions, and competency questions. It is appropriate when a domain already has relatively stable concepts, such as crystallographic descriptors, thermodynamic properties, electrochemical measurement protocols, or standardized process-chain entities. LLMs can accelerate early ontology design by summarizing domain corpora, proposing candidate classes and relations, and translating informal requirements into draft ontology components. This role is best described as semantic drafting rather than autonomous ontology construction: experts begin from a structured candidate rather than a blank schema, while retaining authority over scope, granularity, and logical commitments^[54-56].

Validation is therefore not optional. A practical validation loop should include source-span grounding for each candidate class or relation, competency-question testing, reasoner/SHACL consistency checking, and expert acceptance before ontology admission. This caution is essential because LLMs may fabricate plausible concepts, relations, references, or even DOIs; if admitted, such artifacts can contaminate subsequent extraction as false constraints. Top-down LLM-assisted ontology engineering is strongest when model-generated drafts are coupled with explicit provenance, formal consistency testing, and human-in-the-loop acceptance^[54,56-58]. Materials-domain workflows exemplify this pattern: the Schema-Miner pipeline for atomic layer deposition couples LLM-discovered schemas with ontology grounding and expert refinement^[15,54-56], while retrieval-augmented pipelines build ontologic knowledge graphs from selected papers^[61] or mine decades of literature into ontology-structured triples for expert review^[62].

A further challenge is granularity. Materials concepts can be represented at multiple levels: “thermal treatment”, “calcination”, “annealing under argon”, and “500 °C for 2 h” may all be valid depending on the use case. LLMs can surface these levels from literature, but they cannot decide alone which granularity best serves database integration or downstream modeling. Competency questions provide a practical criterion: if the ontology must answer queries about synthesis temperature, atmosphere, and dwell time, those fields must be explicitly represented; if it only supports high-level process classification, they may remain annotations. Ontology design is therefore a task-specific scientific modeling problem rather than a purely linguistic extraction problem.

Bottom-up approach: from data induction to knowledge enhancement

The bottom-up route addresses the opposite problem: emerging subfields often lack complete schemas, use inconsistent terminology, and report new attributes before formalization. Here, LLMs induce candidate concepts, attributes, and relations directly from a curated literature sample. A representative sample does not require loading an entire corpus into a single context window; documents can be processed iteratively in chunks until schema saturation, when additional documents introduce few new concepts. Schema-Miner and related workflows illustrate this process: LLMs identify domain attributes, after which human review and ontology grounding consolidate the induced schema^[15,59]. Schema induction is typically performed by prompting general-purpose LLMs, whereas subsequent large-scale named entity recognition and relation extraction are often delegated to smaller fine-tuned models for cost and throughput reasons^[8,10].

A practical workflow comprises five steps: (1) select and sample a representative corpus; (2) propose candidate fields, relations, and reporting patterns; (3) consolidate candidates through synonym merging, unit normalization, and mapping to reference ontologies; (4) populate records or graphs from larger corpora through schema-constrained extraction; and (5) apply provenance, consistency, and expert checks before using the knowledge graph for retrieval or reasoning^[60-62,66]. This sequence clarifies that bottom-up discovery expands coverage but does not remove the need for top-down validation.

Bottom-up extraction also creates knowledge graphs that enhance downstream LLM reasoning through retrieval-augmented generation (RAG), where relevant external evidence is retrieved at inference time and supplied as context^[67]. Ontology-grounded RAG (OG-RAG) goes further by retrieving not only passages but also class definitions, admissible relations, and graph neighborhoods^[60]. This is especially useful in materials science because many questions require both factual evidence and typed scientific meaning: a reported value may be a bandgap rather than an absorption peak, or a temperature may belong to a processing event rather than a measurement condition. Knowledge graphs supply this structure while preserving links to source documents^[61-64]. The same hybrid “knowledge graph (KG)-from-text” paradigm extends beyond materials science; in pharmaceutical engineering, Taskiran et al. combine ontology-based information extraction with generative LLMs to construct a literature-derived knowledge graph and expose it through a querying chatbot, using symbolic grounding to improve explainability and reduce hallucination^[68].

Bottom-up induction therefore complements rather than replaces top-down ontology modeling. Induced schemas may contain redundancy, semantic drift, or hallucinated elements, so alignment with reference ontologies, formal consistency checks, source-grounded provenance, and expert review remain mandatory. The most robust workflows are hybrid: bottom-up discovery expands coverage, while top-down ontologies determine admissibility, interoperability, and downstream utility. The field still lacks standardized task definitions, ontology-construction benchmarks, and evaluation protocols, limiting reproducibility^[58]. Closing these gaps is essential for advancing LLM-driven ontology engineering from exploratory schema discovery toward robust, large-scale deployment.

Schema-based generative extraction

Schema-based generative extraction is the most controlled LLM extraction paradigm. Domain experts define a target schema, usually as JSON Schema, ontology-aligned fields, or a domain template, and the model maps source evidence into that structure. This strategy is attractive because it transforms open-ended language generation into constrained record population. The model is still generative, but the schema specifies admissible fields, nesting, value types, and sometimes enumerated categories. As shown conceptually in Figure 6, a source sentence can be converted into a nested object that binds material identity, processing conditions, property values, units, and measurement conditions in a single record.

Figure 6. Example of schema-based materials data extraction. (A) Hierarchical schema structure for materials data extraction; (B) A source sentence is mapped into a nested schema that preserves material identity, synthesis and calcination conditions, property value, unit, and measurement conditions. This representation shows why schema-based extraction is useful for conditional materials properties whose meaning depends on experimental context.

Its main advantage is that hierarchical records can preserve conditional material facts that are difficult to capture with flat entity-relation labels. A catalytic activity, hydrogen-evolution rate, ionic conductivity, or battery capacity is meaningful only together with conditions such as illumination, electrolyte, temperature, current density, cycling rate, atmosphere, or sample history. Traditional sequence labeling can identify entities and relations, but it often fragments this context. Schema-based generation instead represents a conditional property as a nested object, which supports database construction and downstream modeling because values are stored together with the conditions under which they were measured^[8-10].

Reported performance shows why this paradigm has become a foundation for materials data production. In the structured scientific-text extraction study by Dagdelen et al., LLM-NERRE achieved host-dopant F1 of approximately 0.82, compared with roughly 0.50 for seq2rel and 0.39 for a MatBERT-proximity baseline (these figures are drawn from different studies with different corpora, annotation schemes, and matching rules, so they should be read as indicative rather than as a controlled head-to-head comparison), while producing parseable structured outputs in about 99% of cases and reaching high manual-scoring precision^[8]. MaTableGPT reached total F1 up to 96.8% for water-splitting catalyst tables and maintained more than 95% F1 with only ten labeled examples, enabling tens of thousands of catalytic-performance records to be extracted from thousands of tables^[9]. ChatExtract similarly demonstrated that prompt-only conversational extraction can reach approximately 90% precision/recall for selected materials properties, especially when multi-turn follow-up is used to reduce missing fields and hallucinated values^[10]. These results justify the use of schema-based LLMs for high-throughput, fixed-target data production.

The recent 2025-2026 wave of LLM-extracted materials datasets further illustrates the breadth of this paradigm. Reported applications span low-temperature alloys, magnetic materials, metal-organic frameworks, perovskites, nanomaterials, polymer catalysts, thermoelectrics, solid-state synthesis, solid-state batteries, organic photovoltaics, high-entropy-alloy catalysts, aluminum alloys, transition-metal sulfides, and mineral thermodynamics^{[10,66,72-91]}. Table 3 summarizes representative examples. These examples are valuable not only because they produce datasets, but because they expose how extraction tasks differ across materials domains. Metal-organic framework (MOF) extraction often requires topology, linker, metal-node, and adsorption-property fields; battery extraction requires electrode composition, electrolyte, cycling protocol, and capacity-retention context; catalysis extraction requires reaction conditions and normalization basis; synthesis extraction requires multi-step procedural information. A single generic schema is therefore unlikely to serve all subfields.

Schema-based extraction can also integrate limited normalization into the extraction step, such as correcting surface-form OCR variants, canonicalizing simple formulas, or standardizing units under schema constraints^[91]. This capability should be framed cautiously. LLMs learn statistical regularities from training data; they do not possess guaranteed chemical understanding. Mapping ‘Li Co O2’ to ‘LiCoO2’ may be a safe orthographic normalization, but inferring an oxidation state, assigning a CAS number, or resolving a complex molecular identity without source evidence can produce chemically plausible falsehoods. Such cases exemplify fabrication and misattribution and require external chemical databases, rule-based formula checks, or expert review rather than trust in the model alone.

The limitation of schema-based extraction is the mirror image of its strength. Fixed schemas create blind spots for unexpected properties, mechanisms, or experimental conditions. Deeply nested schemas can impose context-window pressure on long documents, and rigid fields may force uncertain evidence into overconfident categories. Schema-based extraction is therefore best suited to known targets and mature data models, not to exploratory discovery. It remains the most controllable paradigm, but its outputs are only as complete as the schema and only as reliable as the grounding and validation layers that surround it.

Schema-free and open knowledge discovery

Schema-free extraction addresses the opposite need: discovering attributes, relations, and reporting patterns that are not known in advance. It is valuable in emerging or interdisciplinary materials domains where predefined schemas lag behind actual reporting practice. Unlike schema-based extraction, schema-free methods aim at ‘unknown unknowns’: new parameters in synthesis recipes, unexpected structure-property relations, emerging device metrics, or reporting conventions that have not yet been formalized. This paradigm is related to Open Information Extraction (openIE), but materials facts are rarely simple subject-predicate-object triples. They are often high-order events with participants, conditions, temporal steps, measurement settings, and uncertainty^[92,93].

Generic OpenIE is not sufficient for this setting because reducing a materials statement to a flat triple can remove context that determines what material or property is actually being described. Dagdelen et al. give a concrete example: an “epitaxial La-doped thin film” of HfZrO4 is scientifically distinct from a “La-doped thin film” or a “La-doped” HfZrO4 sample; omitting the epitaxial and thin-film qualifiers changes the represented material state and can change its physical properties^[8]. The same issue arises for quantitative records: water-splitting overpotentials must be associated with the reaction [oxygen evolution reaction (OER) or hydrogen evolution reaction (HER)], electrolyte, and current density^[9], and property records may require temperature, pressure, or processing conditions in addition to a Material-Value-Unit triplet^[10]. Consequently, schema-free discovery should not terminate at unqualified triples but should preserve attributes and conditions in event-centric, frame-based, graph-based, or nested representations. These representations can later be consolidated into schemas or knowledge graphs while retaining the provenance and experimental context required for scientific interpretation.

Schema-Miner illustrates the transition from extraction to schema induction. Rather than requiring experts to specify all fields in advance, the system asks LLMs to analyze a curated corpus, propose candidate attributes such as precursor pulse time or purge duration in the atomic-layer-deposition literature, and then rely on expert review and ontology grounding to remove redundancy and enforce semantic consistency^[15,59]. Related panoramic or free-form summarization workflows, including LLM-assisted Starrydata curation, read full documents and reconstruct associations among samples, procedures, plots, and reported results rather than extracting isolated scalar fields^[70,71]. These systems are closer to scientific reading than to conventional field filling.

The advantage of schema-free extraction is coverage. It can reveal which variables authors actually report, how terminology varies across papers, and which attributes are repeatedly omitted from handcrafted schemas. This makes it useful for building first-generation data models in areas where community standards are immature. It also provides a feedback mechanism for ontology development: frequently recurring induced attributes can become candidates for formalization, whereas rare or ambiguous attributes may remain annotations. In this sense, schema-free extraction is not the absence of structure but a method for discovering structure from literature.

The trade-off is reduced controllability. Without explicit constraints, outputs may drift semantically across documents, duplicate equivalent concepts under different names, or mix source evidence with model inference. Schema-free extraction therefore requires post hoc structuring: entity alignment, unit normalization, ontology mapping, deduplication, contradiction detection, and provenance audit. Its reliability depends less on the first-pass generation and more on the consolidation pipeline that follows. This is why schema-free and schema-based methods are increasingly hybridized: open induction expands coverage, while schemas and ontologies restore comparability and validation.

A recurring practical difficulty is deciding when an induced schema is stable enough to drive large-scale extraction. In practice, saturation can be approximated by tracking the marginal rate at which new documents introduce previously unseen attributes: once this rate falls below a small threshold across a held-out sample, the schema can be treated as provisionally saturated. Semantic drift - where the same attribute is populated with subtly different meanings across documents or sub-domains - should be monitored explicitly, for example by clustering the values and contexts assigned to each field and flagging fields whose usage diverges over time or across sources. The tension between coverage and consistency is not fully resolvable: broader induction captures more of what authors actually report but admits more redundant or ambiguous fields, whereas tighter constraints improve comparability at the cost of missing emerging attributes. For this reason, induced schemas should be validated before large-scale use through ontology alignment, competency-question testing, expert review of a sampled subset, and consistency checks on populated records, rather than being deployed directly from the induction step.

Agentic systems for materials knowledge extraction

Because this is the most rapidly moving frontier, several of the systems discussed below are reported in recent preprints rather than peer-reviewed venues; we therefore present them as emerging directions whose quantitative claims await independent validation.

Materials knowledge extraction is transitioning from single-pass inference to autonomous agentic systems equipped with planning, reflection, and tool-use capabilities. We define an AI agent operationally: an LLM-based system in a control loop that decides on and performs actions - invoking retrievers, databases, code, or simulators - rather than emitting a single output. The minimal requirement is action: selecting and executing tool calls conditioned on intermediate inference. The defining capability is reflection: evaluating intermediate results, detecting errors or missing evidence, and revising plans accordingly. Single-pass models are therefore not agents; act-only systems qualify as basic agents, whereas the planning–act–reflect loop characterizes mature agentic systems^[94-96].

A note of caution is warranted when interpreting these capabilities. Distinguishing an agent that genuinely revises its plan on the basis of intermediate results from one that merely consumes a longer context window is difficult, and the field currently lacks agreed, principled metrics for making this distinction. Many of the systems discussed below are early prototypes evaluated on narrow tasks, and reported behaviors such as reflection or self-correction are often demonstrated anecdotally rather than measured against controlled baselines. Readers should therefore treat claims of agentic maturity as provisional, and the operational definition above as a functional description rather than as evidence that current systems reliably perform genuine plan revision.

This transition is necessary because materials science literature encodes knowledge through non-local, cross-modal dependencies. A performance metric may depend on synthesis details in supplementary sections, or require correlating text with phase diagrams and characterization plots. Single-pass models fail due to limited context windows, attention dilution, and inability to reconcile distributed evidence. The agentic paradigm introduces a closed-loop reasoning cycle - planning, execution, reflection^[96,97] - that decomposes tasks into subgoals, selectively retrieves additional evidence, and iteratively converges on evidence-supported outcomes. This transforms extraction from passive text processing into active problem-solving resembling human scientific reasoning.

By embedding tool use into this loop, agents extend their effective knowledge horizon. They invoke document retrievers, query structured databases, inspect figures or tables, and perform lightweight computations - mitigating context limitations and grounding knowledge in verifiable sources^[97].

Failure modes in LLM-based materials extraction

Evidence for these recurring failure modes and their mitigation strategies is drawn from prior work^[67,111-121]. Table 4 summarizes the taxonomy. M1, fabrication, occurs when the model invents a value, entity, relation, CAS number, DOI, or citation not supported by the source. M2, misattribution, occurs when a real value is attached to the wrong material, condition, figure panel, or legend entry. M3, unit and quantity-type error, occurs when the model confuses units, dimensions, or property categories, such as treating a temperature as a time or a gravimetric capacity as an areal capacity. M4, format and precision error, includes misplaced decimals, sign errors, lost significant figures, and malformed chemical strings. M5, coverage error, includes omission, duplication, and over-extraction. These categories are defined at the fact-record level rather than the model level because the downstream question is whether an extracted record is scientifically usable.

These modes often compound across the pipeline^[122]. A figure–caption alignment error in perception becomes a misattributed property in extraction; a normalization error splits equivalent materials into duplicate entities; and an unsupported fact may enter a knowledge graph, only to be retrieved later as apparent evidence. This makes provenance, versioning, and validation integral to the data lifecycle rather than optional post-processing.

The severity of these failures is context-dependent^[2]. A missing optional field may be tolerable for qualitative review, whereas a misattributed measurement condition can invalidate a machine-learning dataset. A unit conversion error corrupts derived descriptors; a fabricated DOI or CAS number misleads curators by appearing authoritative. Materials science is particularly vulnerable because many values are plausible over broad ranges: an incorrect bandgap, modulus, or synthesis temperature may look reasonable unless checked against source evidence and physical constraints. Reliability evaluation should therefore distinguish benign omissions from critical scientific errors, rather than treating all mismatches as equivalent.

Layered defenses: entity standardization, source grounding, ontology constraints, physical checks, and human review

The first defense layer is entity standardization and semantic normalization. LLMs can correct some surface-level variation during extraction, especially when constrained by examples or schema formats^[8]. For instance, LLM-NERRE shows that fine-tuned GPT-3 and Llama-2 models can learn annotator-consistent extraction and normalization patterns from only a few hundred examples, reaching F1 ≈ 0.82 for host–dopant extraction compared with ≈ 0.39 for a MatBERT-proximity baseline^[8,123]. However, this capability should be carefully delimited. Orthographic normalization is comparatively safe, whereas inferring unstated chemical attributes - oxidation states, identifiers, or registry numbers - is error-prone. More reliable normalization should therefore combine LLM outputs with deterministic or database-backed tools, including OPSIN for name-to-structure conversion, SMILES/InChI representations, STOUT V2.0 for bidirectional name–SMILES translation, ChemProps for polymer-name standardization, and stoichiometry-aware alignment frameworks such as Wikontic^[21,124-126].

The second layer is ontology and KG constraints. Entity normalization answers “what the object is”, whereas ontologies and KGs constrain “which relations are admissible”. Materials ontologies, including upper-level frameworks such as EMMO and domain-specific extensions, provide type systems and relation constraints that reduce semantic drift and support consistency checking^[50,114-117]. In practice, this can be implemented through schema-constrained generation, where JSON Schema or ontology-derived constraints restrict output structure and value types^[51-53], and through ontology-aware retrieval, where textual mentions are aligned to admissible ontology classes. Large-scale KGs such as MatKG^[65], MGED-KG^[127], and FMKG^[62] further convert isolated extractions into connected evidence structures, enabling provenance tracking, cross-document consistency checks, and downstream reasoning.

The third layer is source grounding and retrieval-based verification. Conversational extraction systems such as ChatExtract use multi-turn follow-up prompts and self-consistency checks to reduce mismatches, achieving approximately 90% precision/recall for representative property triplets such as bulk modulus and metallic-glass critical cooling rate^[10]. Cross-document validation can flag outliers when multiple sources report related facts^[120,121], while NLI-style entailment checks test whether generated claims are supported by cited source text^[111]. RAG further reduces fabrication by conditioning generation on retrieved evidence^[112,113]. In materials informatics, dual-source grounding is especially valuable: unstructured literature retrieval supplies mechanistic and experimental context, whereas structured databases such as the Materials Project provide numerical anchors and reference ranges^[128-130]. Citation-based generation strengthens auditability by requiring each claim to be linked to evidence^[67].

The fourth layer is physics-aware verification, which is the most materials-specific component of quality control. Unit and dimensional analysis can deterministically expose category errors, while enforced unit normalization has been shown to reveal inconsistencies in extracted records^[118]. Derived-property checks can recompute dependent quantities and test internal consistency - for example, recalculating thermoelectric zT from Seebeck coefficient, electrical conductivity, thermal conductivity, and temperature^[131]. Frameworks such as VIRF operationalize this idea through a mentor–apprentice loop, where symbolic or physics-based validators provide corrective feedback when LLM-generated recipes or values violate basic scientific constraints^[119]. Unlike self-consistency or RAG, these checks can provide definite pass/fail signals whenever the extracted fact admits deterministic validation.

The final layer is human-in-the-loop review. Expert review remains essential for ambiguous records, schema induction, ontology extension, and high-impact datasets. The goal is not exhaustive manual reannotation, but targeted curation supported by provenance, confidence scores, detected failure mode, and validator feedback. The most reliable pipelines are therefore hybrid: deterministic checks anchor facts that can be formally validated, probabilistic checks reduce unsupported generation, and human review resolves residual uncertainty^[132].

Evaluation protocols: from accuracy to scientific validity

Reliable extraction also requires transparent evaluation. A useful protocol should define the gold standard, matching rules, scoring level, and stratification strategy. Gold standards are typically constructed through expert annotation; for example, Wang et al. use multi-expert validation before benchmarking extraction quality^[133]. Precision, recall, and F1 remain useful, but they must be reported at both field level and tuple/record level. Field-level evaluation scores isolated attributes, whereas tuple-level evaluation requires material identity, conditions, property, value, and unit to be jointly correct. Tuple-level accuracy is more scientifically meaningful because a correct number assigned to the wrong material is unusable.

Matching rules should reflect physical equivalence rather than text overlap. Metrics such as BLEU or ROUGE are poorly suited to materials extraction because “773 K” and “500 °C” are textually different but physically identical^[134]. Evaluation should therefore use normalization-aware matching for units, chemical identifiers, synonyms, and numerical values. For figure-derived data, tolerance-based scoring is essential: Sror et al. define correctness using a 2% axis-scale tolerance, while PlotExtract reports roughly 5% positional error with over 90% precision and recall for two-axis plots^[38,48]. These examples show that modality-specific evaluation is necessary. Implementing normalization-aware matching consistently is, however, a substantial engineering undertaking rather than a solved problem. Different physical quantities require different normalization logic: band gaps in eV, synthesis temperatures in K vs. °C, lattice parameters in Å vs. nm, and capacities in mAh g^-1 vs. mAh cm^-2 each demand distinct unit systems, reference bases, and equivalence rules. Building and maintaining validators that correctly cover all such quantities and their unit variants across the many subfields of materials science therefore remains an open and labor-intensive challenge.

Results should also be stratified by modality and property type. Sror et al. report F1 > 99% for text and tables but only 85.8% for figures on the same corpus^[48]. Ghosh and Tewari similarly show property-dependent variation, with F1 ≈ 0.91 for thermoelectric properties but ≈ 0.82 for structural fields^[102]. Because LLM outputs are stochastic, repeatability should also be reported; in one study, approximately 87% of extracted items were identical across five runs, with remaining variance concentrated in figure reading and decimal placement^[48].

Evaluation is now expanding beyond extraction accuracy toward scientific validity. Physical-consistency scoring treats violations of chemical or physical constraints as critical errors. Source-tracked extraction allows each value to be audited against a cited passage, table cell, figure panel, or caption. LLM-as-a-judge methods, such as LeMat-Synth, can help scale rubric-based evaluation, but they should be treated as auxiliary triage tools because the judge may inherit the same biases and hallucination tendencies as the extractor^[135].

At the benchmark level, the field is developing a layered ecosystem. MatSciRE^[136], SuperMat^[137], and MatSciNLP^[138] evaluate core materials text understanding and extraction; LLM4Mat-Bench^[139] and LLM4Mol^[140] extend evaluation toward crystal, molecular, and predictive reasoning tasks; and multimodal benchmarks such as MatViX^[37], DocGenome^[43], and Macbench^[141] evaluate chart understanding, multimodal document reasoning, and long-context scientific interpretation. Together, these benchmarks mark a shift from string-level correctness toward source-supported and physically valid knowledge extraction. However, standardized task definitions, gold-standard construction procedures, tolerance rules, and reporting conventions remain insufficient. Establishing community-wide evaluation protocols is therefore essential for trustworthy materials databases, knowledge graphs, and agentic discovery systems.