Research

Reading a driver's intent from video, in real time.

Doctoral research at the University of Hertfordshire, 2020 to 2024. A real-time system watches a vehicle waiting at a UK T-junction and predicts whether it will stop or pull out, as an early warning for an approaching motorcyclist.

Built on a dataset collected for the problem, and evaluated on a junction the model never saw in training.

See the pipeline Jump to the results Read the thesis
TRACK 12 INTENT · EVERY 40 ms Stop Pull out Merge line
Illustration. The system works from a fixed camera overlooking the junction.

The pipeline, frame by frame

Every frame passes through five stages. The expensive step is detection; everything after it is light enough to run alongside.

  1. Detect

    YOLOv5m, fine-tuned from COCO

    Finds the vehicles in each frame, using a detector retrained on images of the target vehicles.

  2. Track

    DeepSORT

    Follows each vehicle from frame to frame, so the system reasons about one vehicle's history, not isolated boxes.

  3. Describe

    Per-frame feature vector

    Velocity, acceleration, distance to the merge line and traffic on the main road, measured for every frame.

  4. Predict

    Gaussian naive Bayes, every 40 ms

    Votes across the vehicle's history. When no outcome has a majority, it gives no prediction rather than a guess.

  5. Learn

    Outcome-based self-labelling

    What the vehicle actually did at the line labels its trajectory, which joins the training data.

Why naive Bayes: limited labelled trajectories, a model that had to update online as each vehicle was labelled, and the need to read why it made each call. It learns from little data, updates by adjusting a mean and a variance, and costs almost nothing to run.

Data built for the problem

No suitable public dataset existed, so the research began by collecting one.

  • 16 hof video recorded at UK T-junctions
  • 5junctions recorded
  • 5,381detector training images, augmented from a set of 1,139
  • 237vehicles at a junction held out entirely from training

Choosing the detector on evidence

Faster R-CNN, with ResNet50 and MobileNet v3 backbones, was benchmarked against YOLOv5 on accuracy and on CPU and GPU latency. Model size, input resolution and frame rate were then swept to find the operating point: the most accurate detector that still leaves time for tracking and prediction inside each 40 ms step.

Learning from outcomes, never from its own guesses

Hand-labelling every trajectory does not scale. Instead, the junction labels itself: once a vehicle reaches the line, what it actually did becomes the label for its whole approach. A wrong prediction therefore never enters the training data.

Tested on recorded junction video.

  1. ApproachEach frame adds a prediction to the vehicle's history.
  2. PredictionA majority vote across that history, or no prediction at all.
  3. Outcome at the lineThe vehicle stops or pulls out. That observed outcome is the label.
  4. Training dataThe labelled trajectory is added, and the model updates.

Results on a junction it had never seen

The test junction was held out entirely from training. Accuracy rises as the vehicle approaches, because drivers far from the line have not yet committed to a decision.

  • 0.86F1 score, stop
  • 0.85F1 score, merge (pulling out)
  • 237vehicles at the held-out junction

Limits, stated plainly

  • Real time on a desktop GPU, not on embedded hardware. The classifier costs almost nothing; the detector is the expensive part, and the resolution and frame-rate sweeps map out how to scale it down.
  • Fixed camera. The features are measured in pixels from a fixed viewpoint. A moving platform would need stabilisation and projection onto the ground plane, or features that do not depend on viewpoint.
  • Independent features. Naive Bayes treats velocity, acceleration and distance as independent, which they are not. With more data, a small sequence model over the trajectory, with calibrated confidence, is the next step.
  • Rare hazards. The case that matters most, a vehicle pulling out into traffic, is rare in real footage. Training data was balanced with SMOTE, but rare hazardous events need far more data, or synthetic data.

Where it led

The same disciplines run through Daisy Studio's field capture work: collect the data the problem needs, label from what actually happened, test on cases the system has not seen, and decline to answer rather than guess.

How the capture system is built EdgeRaptor Willow

The thesis

Real-Time Accuracy and Effectiveness of Computer Vision and Machine Learning in Vehicle Intent Prediction at T-Junctions

Dr Simon McCool. PhD, Artificial Intelligence (Computer Vision and Machine Learning), University of Hertfordshire, 2024.

Supervisors: Dr Christoph Salge, Dr Wei Ji and Prof. Volker Steuber.

Read the full thesis in the University of Hertfordshire Research Archive (open access, CC BY 4.0).