Robust visual perception in challenging real-world scenarios

Amirreza Rouhi

doi:10.17918/00011379

This dissertation investigates robust visual perception for detecting and recognizing difficult objects in challenging real-world environments, tracing a single arc from long-range small-UAV detection to open-ended object detection. The work began with the practical problem of detecting small flying objects at long range, where drones routinely occupy fewer than fifty pixels, blend into cluttered backgrounds, and span distances from a few metres to over a hundred. The deeper obstacle was data: no public drone-detection benchmark captured the operating conditions we needed, and none provided camera-to-drone range information. We therefore contributed a comprehensive survey of deep learning methods for drone detection, covering 16 datasets and 24 algorithms, and introduced two new benchmarks: the Long-Range Drone Detection Dataset (LRDD) with 22,500 images, and its enhanced successor LRDDv2 with 39,516 images and explicit camera-to-drone range annotations for over 8,000 images, the first such information in any public drone-detection dataset. On standard benchmarks, models trained on LRDD and LRDDv2 substantially outperform baselines trained on Drone-vs-Bird, with the largest gains in the small-bounding-box regime that defines long-range detection. Yet the long-range drone problem surfaced a deeper limit that better datasets alone cannot lift. When an object's visual signature shrinks to a few dozen pixels and shares texture, motion, or shape with its surroundings, the information needed to identify it is not fully present in the image. And drone detection turns out to be a relatively constrained instance of this bottleneck: at altitude the realistic candidate vocabulary is small, including drones, birds, and perhaps a handful of airborne categories, so even when pixels degrade, the space of plausible answers stays tightly bounded. General object detection enjoys no such luxury. A street scene at dusk admits dozens of plausible categories, an indoor scene hundreds, and an open-ended setting no a-priori bound at all. Under fog, motion blur, partial occlusion, or low light, the same pixel degradation that hurts drone detection compounds in the general case, weakening evidence for the right answer while multiplying the alternatives that remain consistent with it. Humans cope with these regimes by drawing on knowledge that lies outside the frame: the spatial layout of the scene, the categories that typically co-occur there, and the implicit physics of what can plausibly appear and where. The remainder of this dissertation pursues that observation, asking how vision systems can incorporate the kind of scene-level contextual knowledge that humans take for granted: first to rescue detections of familiar categories under adverse conditions, and ultimately to recognise objects from categories never encountered at training time. We first present LMOD, a transformer-based model that learns scene-wide contextual relationships exclusively from object labels, positions, and sizes, with no pixel access at all. Integrating LMOD with conventional detectors such as DETR and YOLOv8 yields consistent improvements on COCO, Visual Genome, and the challenging ExDark low-light dataset, with the largest gains under degraded imaging conditions. We then scale the contextual paradigm by coupling YOLO detectors with LLaMA 2, a Large Language Model, which validates low-confidence detections through scene-level contextual prompts. Experiments on COCO-2017 and the newly introduced COCO-2017-Blurred benchmark confirm significant gains, particularly for small and occluded objects. We then extend the work toward open-ended object detection, the ability to both detect and assign meaningful semantic labels to previously unseen objects without any predefined vocabulary. ADAM establishes the core architecture by constructing an Embedding-Label Repository (ELR) that pairs CLIP visual embeddings of unknown objects with candidate labels generated through context-aware LLM prompting; on COCO validation, ADAM reaches 61.3% Top-1 accuracy and transfers training-free to PASCAL VOC. COSRA extends ADAM with a complete detection front-end, using SAM and Faster R-CNN, attribute-based characterization across 11 categories and 155 descriptors, and an iterative semantic-grouping refinement mechanism. On COCO, it achieves 38.8% Novel AP, and on LVIS v1, it achieves 24.2% Rare AP, surpassing every open-vocabulary baseline despite requiring neither training nor a vocabulary at inference. Our most advanced framework upgrades the detection front-end to Mask R-CNN and introduces diffusion-generated confusion anchors: for each unknown object, Stable Diffusion XL synthesizes contrastive visual exemplars from second-ranked LLM predictions, localized by Florence-2 and validated by CLIP, enriching the ELR with the representational diversity needed to disambiguate visually similar categories. A cross-label iterative refinement mechanism then groups semantically similar labels and corrects errors across related categories through k-NN majority voting. On COCO and LVIS minival, this framework achieves 60.8% Novel AP50 and 52.4% APr respectively, surpassing all existing open-ended and open-vocabulary methods, including those with larger backbones and extensive supervised training, while requiring neither retraining nor vocabulary supervision. Collectively, this body of work establishes a unified framework for robust visual perception of small, rare, and novel objects in complex real-world environments, progressing from specialised long-range drone detection through context-aware closed-set detection to fully open-ended scene understanding.

Robust visual perception in challenging real-world scenarios

Files and links (1)

Abstract

Metrics

Details

Robust visual perception in challenging real-world scenarios

Files and links (1)

Abstract

Metrics

Details

Drexel University Social media