Optical Simulation & Image Quality for Automotive Semantic Segmentation

Cameras are often deployed in automated driving systems for multiple perception tasks, including object recognition. In previous work, a novel optical simulation framework, SOLAS, was proposed to simulate wide-angle and fisheye cameras on automotive data for 2D object detection. The main focus of this work is to assess semantic segmentation networks in relation to camera quality, mainly used for collision avoidance, lane marking detection, and enhanced road navigation. Characterising semantic segmentation performance across the full range of camera production tolerances is essential, given that image quality may vary across units. This study further develops the novel optical simulation framework, SOLAS, where a Pythonic ray-tracing library, KrakenOS, is adapted to simulate wide-angle surround-view optical systems across a range of optical defocus, directly in automotive scenes. Optical defocus is achieved through sensor displacement. In this paper, we propose extending SOLAS with ARDÁN, a pipeline for measuring image quality in automotive scenes. Furthermore, we demonstrate that our framework, utilising a diverse set of cameras and dense grids of Point Spread Functions (PSFs), is scalable in semantic segmentation, achieving consistent results across various datasets supported by statistical metrics. The primary metric used for performance evaluation is mean Intersection over Union (mIoU). Based on our results, the Segformer network with 24.73M parameters was the most robust across wide-angle automotive datasets, namely, nuScenes and Virtual KITTI 2.0. In nuScenes, image sharpness (cycles/pixel) dropped by 20.5% from a nominal focus, with a small Segformer performance drop from 55.21% mIoU by 0.37%. In Virtual KITTI 2.0, image sharpness dropped by 44.66% from a nominal focus, with a Segformer performance drop by 37.42%. However, for fisheye, the PIDNet model with 28.76M parameters was the most robust model across the defocus range. In Woodscape, image sharpness dropped by 53.45% from a nominal focus, with a PIDNet performance drop by 8.55%. In Parallel Domain Woodscape, image sharpness dropped by 34.46% from a nominal focus, with a PIDNet performance drop by 7.28%.