Training a supervised ML model requires labelled data. Labelled data requires domain experts. In specific domains like infrastructural inspection, these experts are scarce, expensive and probably have better things to do than clicking through hundreds of images.
So, I asked a logical question: Could we make use of LLMs to evaluate rust? And would they perform better than an absolute amateur like me? I decided to give it a try and started an experiment with several Visual Language Models (VLMs), Large Language Models with vision encoder features.
The setup: 291 rust images from the DACL10k validation dataset, rated on a continuous 0–1 severity scale. Four raters: me (data scientist, zero inspection experience), GPT-4o, Claude Sonnet, and GLM-4.6V-Flash-9B running locally.
All four received the exact same prompt: “Act as an experienced structural inspector, rate rust severity according to DIN 1076 (the German standard for inspecting engineering structures, rating damage by its structural significance and urgency of action.), use the full 0–1 range across six anchor categories from insignificant surface discolouration to severe section loss, and return structured JSON.”
Comparing the outputs:
The severity ordering: OpenAI < Self < Claude ≲ GLM. GPT-4o is the most lenient while Claude and GLM share nearly identical mean ratings (bias = −0.009) despite only moderately agreeing on individual images (r = 0.60, MAE = 0.133). I sit somewhere in the middle (which I choose to interpret as being well-calibrated and not as being indecisive).
OpenAI and Claude have the strongest rank correlation (r = 0.77), they mostly agree on which images are more severe than others, but Claude rates about 0.18 higher on average. One thing that stands out is that GLM, an open-source model, ending up in the same severity areas as Claude is both surprising and relevant if you’re working with data that can’t leave your network.
The correlations across all pairs range from 0.40 to 0.77, showing some shared signal and not just random noise. But there’s also real disagreement in both average severity level and in how individual images get ranked. If raters only agree to this extent, that agreement sets a ceiling on what any downstream model can learn from these labels.
Should we let LLMs label our ground truth?
If I have zero professional experience in structural engineering, why shouldn’t we just accept Claude’s or OpenAI’s ratings as the absolute ground truth?
The short answer is: Because “consistently confident” is not the same as “correct.”
A visual model doesn’t feel the humidity; it doesn’t understand the physical context of the surrounding concrete. It is translating pixels into text based on probability, not physics. If we blindly accept VLM outputs as ground truth, we risk training computer vision models on “hallucinated urgency” or “hallucinated safety.”
What I’d like to explore next:
- Can an iterative approach help e.g. AI rates first, human reviews only the uncertain ones
- Whether the biases are prompt-correctable or architectural
- What inter-rater agreement looks like between actual human inspectors on the same images (could someone from the inspection world validate this? Genuine ask.)
If certified inspectors also agree at r ≈ 0.7, then the label noise isn’t a data quality problem, but a property of the domain itself.
The reason I’m stuck labelling rust images in the first place is RoboTUNN, a project where a robot dog inspects subway tunnels and feeds the data into a digital twin for predictive maintenance. Before any of that can work, someone has to decide what counts as severe damage. Right now, that someone is me, three LLMs, and a lot of disagreement.
If you work in infrastructure inspection and have thoughts on this, or access to labelled damage data, I’d love to talk.
