Abstract

Real-time object detection has seen tremendous advances in recent years, driven largely by the power of convolutional neural networks (CNNs) and transformer-based models. However, existing approaches still struggle to maintain detection accuracy under adverse weather conditions such as fog, rain, and nighttime low-light scenarios. These environments are critical for applications such as autonomous driving, aerial surveillance, and smart city infrastructure. This paper presents a robust transformer-based object detection framework designed to operate efficiently in challenging weather conditions without sacrificing real-time performance. The proposed system builds upon Vision Transformers (ViTs) and hybrid CNN-ViT architectures to capture both local texture and global context features. A novel weather-adaptive attention mechanism is introduced, enabling the model to dynamically reweight features based on visual degradation cues caused by environmental interference. We train and evaluate our framework using three leading weather-specific benchmark datasets: DAWN, Foggy Cityscapes, and NightOwls. These datasets encompass diverse visibility conditions, object categories, and urban scene complexities.

To ensure deployment feasibility in real-world systems, we incorporate lightweight architectural modifications, including quantization-aware training, positional encoding reduction, and pruning strategies. These optimizations significantly reduce model size and computational demand without compromising accuracy. Empirical results show that our model achieves real-time inference speeds of 25 to 30 FPS on edge-level NVIDIA Jetson devices, while improving mean Average Precision (mAP) by 10 to 14 percent under extreme weather conditions when compared to traditional CNN-based detectors such as YOLOv5 and Faster R-CNN. Additionally, ablation studies confirm the efficacy of hybrid backbones and weather-attentive feature fusion in handling occlusions, motion blur, and varying light intensities. This research offers a practical and scalable solution to a critical gap in robust computer vision, enabling safer and more reliable deployment in autonomous navigation and intelligent traffic systems that operate in non-ideal conditions.

Keywords

  • Real-time object detection
  • Vision Transformer (ViT)
  • hybrid CNN-ViT
  • fog
  • rain
  • night
  • edge AI
  • adverse weather
  • autonomous driving
  • mAP
  • inference time

References

  1. 1. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., ... & Tao, D. (2022). A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1), 87-110.
  2. 2. Khan, S., Naseer, M., Hayat, M., Zamir, S. W., Khan, F. S., & Shah, M. (2022). Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s), 1-41.
  3. 3. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., ... & Tao, D. (2022). A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1), 87-110.
  4. 4. Kenk, M. A., & Hassaballah, M. (2020). DAWN: vehicle detection in adverse weather nature dataset. arXiv preprint arXiv:2008.05402.
  5. 5. Li, R., Luo, Y., Park, I., & Xuan, Z. Improving the Robustness of Object Detection Under Hazardous Conditions.
  6. 6. Valanarasu, J. M. J., Yasarla, R., & Patel, V. M. (2022). Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2353-2363).
  7. 7. Jeon, M., Seo, J., & Min, J. (2024, May). Da-raw: Domain adaptive object detection for real-world adverse weather conditions. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2013-2020). IEEE.
  8. 8. Ding, Q., Li, P., Yan, X., Shi, D., Liang, L., Wang, W., ... & Wei, M. (2023). CF-YOLO: Cross fusion YOLO for object detection in adverse weather with a high-quality real snow dataset. IEEE Transactions on Intelligent Transportation Systems, 24(10), 10749-10759.
  9. 9. Appiah, E. O., & Mensah, S. (2024). Object detection in adverse weather condition for autonomous vehicles. Multimedia Tools and Applications, 83(9), 28235-28261.
  10. 10. Tiwari, A. K., Pattanaik, M., & Sharma, G. K. (2024). Low-light DEtection TRansformer (LDETR): object detection in low-light and adverse weather conditions. Multimedia Tools and Applications, 83(36), 84231-84248.
  11. 11. Petraq Kosho. (2025). Public-Private Collaboration in Michigan’s Post-COVID Economic Development. World Journal of Advanced Research and Reviews, 26(3), 629–638. https://doi.org/10.30574/wjarr.2025.26.3.2241
  12. 12. Hao, C. Y., Chen, Y. C., Chen, T. T., Lai, T. H., Chou, T. Y., Ning, F. S., & Chen, M. H. (2024). Synthetic Data-Driven Real-Time Detection Transformer Object Detection in Raining Weather Conditions. Applied Sciences, 14(11), 4910.
  13. 13. Gupta, H., Kotlyar, O., Andreasson, H., & Lilienthal, A. J. (2024). Robust object detection in challenging weather conditions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 7523-7532).
  14. 14. Aloufi, N., Alnori, A., & Basuhail, A. (2024). Enhancing Autonomous Vehicle Perception in Adverse Weather: A Multi Objectives Model for Integrated Weather Classification and Object Detection. Electronics, 13(15), 3063.
  15. 15. Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., & Girshick, R. (2021). Early convolutions help transformers see better. Advances in neural information processing systems, 34, 30392-30400.
  16. 16. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020, August). End-to-end object detection with transformers. In European conference on computer vision (pp. 213-229). Cham: Springer International Publishing.
  17. 17. Valanarasu, J. M. J., Yasarla, R., & Patel, V. M. (2022). Transweather: Transformer-based restoration of images degraded by adverse weather conditions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2353-2363).
  18. 18. Petraq Kosho. (2024). Ethical AI in Immigrant-Serving Workforce Development: A Global Perspective. World Journal of Advanced Research and Reviews, 24(1), 2775–2782. https://doi.org/10.30574/wjarr.2024.24.1.2953
  19. 19. Periyasamy, R., Sasi, S., Malagi, V. P., Shivaswamy, R., Chikkaiah, J., & Pathak, R. K. (2025). Artificial intelligence assisted photonic bio sensing for rapid bacterial diseases. Zeitschrift für Naturforschung A, (0).
  20. 20. Raj, L. V., Sasi, S., Rajeswari, P., Pushpa, B. R., Kulkarni, A. V., & Biradar, S. (2025). Design of FBG-based optical biosensor for the detection of malaria. Journal of Optics, 1-10.
  21. 21. Rajeswari, P., & Sasi, S. (2024). Efficient k-way partitioning of very-large-scale integration circuits with evolutionary computation algorithms. Bulletin of Electrical Engineering and Informatics, 13(6), 4002-4007.
  22. 22. Zhong, J., Wang, Y., Zhu, D., & Wang, Z. (2025). A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planning. arXiv preprint arXiv:2506.07236.
  23. 23. Wang, F., Bao, Q., Wang, Z., & Chen, Y. (2024, October). Optimizing Transformer based on high-performance optimizer for predicting employment sentiment in American social media content. In 2024 5th International Conference on Machine Learning and Computer Application (ICMLCA) (pp. 414-418). IEEE.
  24. 24. Gharatappeh, S., Sekeh, S., & Dhiman, V. (2025). Weather-Aware Object Detection Transformer for Domain Adaptation. arXiv preprint arXiv:2504.10877.
  25. 25. Tiwari, A. K., Pattanaik, M., & Sharma, G. K. (2024). Low-light DEtection TRansformer (LDETR): object detection in low-light and adverse weather conditions. Multimedia Tools and Applications, 83(36), 84231-84248.
  26. 26. Ikram, S., Sarwar, I., Ikram, A., & Abdullah-AI-Wahud, M. (2025). A Transformer-Based Multimodal Object Detection System for Real-World Applications. IEEE Access.
  27. 27. Kondapally, M., Kumar, K. N., & Mohan, C. K. (2024, June). Object Detection in Transitional Weather Conditions for Autonomous Vehicles. In 2024 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE.
  28. 28. Chen, S., Shu, T., Zhao, H., & Tang, Y. Y. (2023). MASK-CNN-Transformer for real-time multi-label weather recognition. Knowledge-Based Systems, 278, 110881.
  29. 29. Zhang, B. (2024). Enhanced Safety of Autonomous Driving in Real-World Adverse Weather conditions via Deep Learning-Based Object Detection (Doctoral dissertation, Université d'Ottawa| University of Ottawa).
  30. 30. Ye, T., Qin, W., Zhao, Z., Gao, X., Deng, X., & Ouyang, Y. (2023). Real-time object detection network in UAV-vision based on CNN and transformer. IEEE Transactions on Instrumentation and Measurement, 72, 1-13.
  31. 31. Shyam, P., & Yoo, H. (2024). Lightweight thermal super-resolution and object detection for robust perception in adverse weather conditions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 7471-7482).
  32. 32. Li, Y., & Shen, L. (2025). A Frequency Domain-Enhanced Transformer for Nighttime Object Detection. Sensors, 25(12), 3673.
  33. 33. Wan, Y., Wang, H., Lu, L., Lan, X., Xu, F., & Li, S. (2024). An Improved Real-Time Detection Transformer Model for the Intelligent Survey of Traffic Safety Facilities. Sustainability, 16(23), 10172.
  34. 34. Li, Y., & Liu, X. (2025, January). Transformer-based vehicle detection algorithm under foggy conditions. In Fifth International Conference on Signal Processing and Computer Science (SPCS 2024) (Vol. 13442, pp. 201-207). SPIE.
  35. 35. Zhang, G., Wang, L., & Chen, Z. (2024). A Step-Wise Domain Adaptation Detection Transformer for Object Detection under Poor Visibility Conditions. Remote Sensing, 16(15), 2722.
  36. 36. Johansen, A. S., Nasrollahi, K., Escalera, S., & Moeslund, T. B. (2023). Who cares about the weather? Inferring weather conditions for weather-aware object detection in thermal images. Applied Sciences, 13(18), 10295.
  37. 37. Jankovic, Branislava, Sabina Jangirova, Waseem Ullah, Latif U. Khan, and Mohsen Guizani. "UAV-Assisted Real-Time Disaster Detection Using Optimized Transformer Model." arXiv preprint arXiv:2501.12087 (2025).
  38. 38. Putatunda, R., Khan, M. A., Gangopadhyay, A., Wang, J., Busart, C., & Erbacher, R. F. (2023, June). Vision transformer-based real-time camouflaged object detection system at edge. In 2023 IEEE International Conference on Smart Computing (SMARTCOMP) (pp. 90-97). IEEE.
  39. 39. Xi, K., Bi, X., Xu, Z., Lei, F., & Yang, Z. (2024, November). Enhancing Problem-Solving Abilities with Reinforcement Learning-Augmented Large Language Models. In 2024 4th International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI) (pp. 130-133). IEEE.
  40. 40. Zhong, J., Wang, Y., Zhu, D., & Wang, Z. (2025). A Narrative Review on Large AI Models in Lung Cancer Screening, Diagnosis, and Treatment Planning. arXiv preprint arXiv:2506.07236.
  41. 41. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929