Engineering
From sensor to evidence record.
How the field capture system behind EdgeRaptor and Willow is built: the architecture, the decisions behind it, the problems met along the way and how they were solved. Designed and built end to end by one engineer, from device integration on the tablet to the backend, cloud storage and model layer.
- 4media types (photo, video, thermal and voice) from tablet, pole and thermal cameras
- 3inspection schemas on one engine: UK RICS, Florida forms, API 653
- 94automated backend tests passing
- TRL 4integrated rig tested end to end outdoors
The architecture
Six layers, from the devices in the inspector's hands to the structured record that leaves site. Each layer has one job, so a device, a schema or a model can change without rebuilding the rest.
-
Edge devices
Sensors integrated directly into the capture app, not imported afterwards.
-
Capture app
Live view, capture and spoken notes on one tablet, one location at a time.
-
Ingest and storage
Capture bundles uploaded over a mobile connection and stored against shared identifiers.
-
Structure
A domain profile supplies the elements, condition codes and phrase bank for the inspection type.
-
Infer and verify
A vision-language model reviews the evidence under schema constraints and asks for a recapture when it cannot assess it.
-
Hand-off
One reviewed record, with provenance on every line, rendered into the outputs the inspection needs.
The data model
Every photograph, clip, thermal frame and voice note is bound at source to its survey, session, location and capture group. Physical location, observed condition and evidence are kept distinct, and nothing is left floating in a camera roll.
Thermal data is treated as radiometric only when it carries temperature values, and is linked by the same identifiers as the image it belongs to.
- Photograph
- Video clip
- Thermal image and temperature data
- Voice note and transcript
Design decisions, and why
-
Capture before models
The structured capture layer was built before the model layer, so every item is labelled at the moment it is taken rather than handed to a model as loose files.
Why: context and labels recorded at acquisition remove the later reconstruction and ambiguity that loose files need before they can train or evaluate a detector.
-
The model is replaceable and untrusted
The vision-language model works inside a fixed schema, and its prompts are assembled with LangChain from the inspector's structured comments and a curated phrase bank. The inspector's words outrank its suggestions, and no field is filled silently.
Why: the evidence trail has to survive a change of model, and a professional must be able to trace every line to its source.
-
One engine, swappable schemas
Built first against UK RICS survey levels, with a 444-phrase bank, then carried to Florida inspection forms, with a 203-phrase bank, and then to API 653 external tank inspection.
Why: the capture, binding and provenance layers do not change between domains; only the profile does.
-
Voice as the primary input
Observations are spoken as they are captured, across pole camera, tablet video and stills, with a typed note optional. Transcription status is tracked separately from the evidence.
Why: fieldwork on Texas tank sites showed that inspectors working at height have both hands occupied.
-
Capture once, many outputs
In the Florida build, one reviewed record feeds both the client report and the insurer forms, such as the Citizens 4-Point and roof forms, with image, video and thermal evidence carried into the report.
Why: shared data is captured once and stays editable, instead of being re-keyed for every form.
-
Recorded supervision
A supervising inspector can follow the record live from anywhere and send prompts to the person on site, such as another image from the north side, before they move on.
Why: oversight is stored with the evidence it relates to, so it can be shown rather than assumed.
Problems met, and how they were solved
Field hardware does not behave like a laptop on a desk. These are the problems that shaped the build.
-
Problem
The thermal camera's wireless session dropped immediately after connecting.
FixTraced against FLIR's own sample application and rebuilt the connection order: stop Bluetooth discovery, join the camera's Wi-Fi, prefer that network, then connect and start the stream. The live adapter is specific to the FLIR ONE family; ingest stays camera-agnostic.
-
Problem
A race condition in the camera lifecycle.
FixA dedicated state machine now governs every camera transition, so capture can only happen from a known state.
-
Problem
Elevated capture depended on a proprietary action-camera SDK.
FixReplaced it with a hard-wired USB camera on a 10 m cable: live view and capture on the tablet during the inspection, feeding the same ingest, with one real-time path shared by both products.
-
Problem
Upload, transcription and AI review all need a connection on sites that have no Wi-Fi.
FixA belt-mounted 5G router travels with the rig, so evidence is reviewed on site, wherever there is 5G coverage, while the inspector is still there to recapture.
-
Problem
Demonstration records had to stand up to scrutiny line by line.
FixA hardening rebuild separated firm findings from system and model interpretation, and stamped every run with an evidence version and timestamp derived from its source. In the result, all 42 findings carried explanatory evidence, and the backend test suite passed.
Tested in the field
In September 2026 the integrated rig was run end to end outdoors on disused steel gasholders in southern England.
- Pole-mounted camera on a 10 m cable, tablet, thermal imager, battery pack and belt-mounted 5G router, worn as one rig.
- Evidence captured, uploaded to the cloud and transcribed on site.
- Live AI evidence review returned verification prompts to the inspector before moving on.
- Flask backend of about 8,350 lines with 94 automated tests, reviewed for race conditions, authentication gaps and asynchronous bottlenecks.
Technology Readiness Level 4 for the tank application. These results come from the developer's own testing; independent validation on a working tank site is the step to TRL 5.
What is built, and what comes next
Independent development and field testing ran from 2024 to September 2026. Further deployment depends on an industrial partner. The line between what is built and what would come next is kept explicit.
Built and tested
- Tablet capture app with direct USB pole camera and FLIR thermal integration
- Voice capture with transcription
- Evidence bound at source to survey, session, location and capture group
- Domain profiles for UK RICS, Florida forms and API 653
- Schema-constrained vision-language review with recapture requests
- Live remote supervision recorded in the evidence trail
- Cloud ingest, storage and backend test suite
Next
- A custom detector for a closed defect vocabulary, trained on the captured corpus
- A spoken flag during continuous video that anchors an observation and selects the surrounding frames
- Measurement evidence, such as wall thickness, on the same observation model
- A model gateway with full logging, an evaluation harness and an open-weights model run in shadow mode
- An operational trial on a working tank site
Related work
The same habits started in doctoral research on real-time intent prediction from video: build the dataset the problem needs, label from what actually happened, and decline to answer rather than guess.