An Explainable Multimodal Hybrid Deep Learning Framework for Real-Time IoT-Based Fish Freshness Classification

Rian Ordila; Miratu Megasari; Yulanda Yulanda; Yuda Irawan; Refni Wahyuni

doi:10.29207/resti.v10i5.7844

Authors

Rian Ordila Universitas Hang Tuah Pekanbaru
Miratu Megasari Universitas Hang Tuah Pekanbaru
Yulanda Yulanda Universitas Hang Tuah Pekanbaru
Yuda Irawan Universitas Hang Tuah Pekanbaru
Refni Wahyuni Universitas Hang Tuah Pekanbaru

DOI:

https://doi.org/10.29207/resti.v10i5.7844

Keywords:

BiLSTM, EfficientNet B3, fish freshness classification, multimodal fusion, YOLO26

Abstract

Reliable assessment of fish freshness supports product quality, food safety, and consumer acceptance throughout fisheries supply chains. Sensory inspection remains common, but reliance on individual judgment can produce inconsistent assessments and limit monitoring efficiency. Here, a multimodal framework combines Computer Vision, IoT sensing, and Deep Learning to monitor Nile tilapia freshness in realtime. Its visual branch uses YOLO26 to locate fish eyes and EfficientNet B3 with CBAM to classify the extracted regions. A BiLSTM network learns temporal patterns from MQ136 and MQ137 measurements of H₂S and NH₃ associated with spoilage. An MLP then combines the class probabilities from both branches to assign the final freshness category. In the detection experiment, YOLO26 recorded 0.930 precision and 0.953 mAP@0.5. Visual classification with EfficientNet B3 plus CBAM reached 92% accuracy, whereas sensor classification with BiLSTM reached 93%, with AUC values as high as 0.99. Combining the modalities yielded 97% accuracy and a macro average F1 score of 0.97, exceeding the performance of either branch alone. Grad CAM maps located the main visual responses within eye regions relevant to freshness deterioration. These results support the framework's interpretability and its potential use for intelligent fish freshness assessment under near realtime monitoring conditions.

Downloads

Download data is not yet available.

References

[1] A. D. Cortés-Sánchez et al., “Processing, Quality and Elemental Safety of Fish,” 2024. doi: 10.3390/app14072903.

[2] D.-Y. Kim, S.-W. Park, and H.-S. Shin, “Fish Freshness Indicator for Sensing Fish Quality during Storage,” Foods, vol. 12, no. 9, 2023, doi: 10.3390/foods12091801.

[3] A.-F. M. El-Sayed and K. Fitzsimmons, “From Africa to the world—The journey of Nile tilapia,” Rev. Aquac., vol. 15, no. S1, pp. 6–21, Feb. 2023, doi: https://doi.org/10.1111/raq.12738.

[4] M. E. Abd El-Hack et al., “Effect of environmental factors on growth performance of Nile tilapia (Oreochromis niloticus),” Int. J. Biometeorol., vol. 66, no. 11, pp. 2183–2194, 2022, doi: 10.1007/s00484-022-02347-6.

[5] D. Shen, M. Zhang, A. S. Mujumdar, and Y. Ma, “Consumer-oriented smart dynamic detection of fresh food quality: recent advances and future prospects,” Crit. Rev. Food Sci. Nutr., vol. 64, no. 30, pp. 11281–11301, Nov. 2024, doi: 10.1080/10408398.2023.2235703.

[6] M. N. Madhubhashini, C. P. Liyanage, A. U. Alahakoon, and R. P. Liyanage, “Current applications and future trends of artificial senses in fish freshness determination: A review,” J. Food Sci., vol. 89, no. 1, pp. 33–50, Jan. 2024, doi: https://doi.org/10.1111/1750-3841.16865.

[7] M. Kundu, A. Vashisth, and P. Krishnan, “Instrumental detection of fish freshness in research and food industry: a review,” J. Food Sci. Technol., vol. 63, no. 1, pp. 22–50, 2026, doi: 10.1007/s13197-025-06497-4.

[8] D. Li and L. Du, “Recent advances of deep learning algorithms for aquacultural machine vision systems with emphasis on fish,” Artif. Intell. Rev., vol. 55, no. 5, pp. 4077–4116, 2022, doi: 10.1007/s10462-021-10102-3.

[9] R. Singh, C. Nickhil, R.Nisha, K. Upendar, B. Jithender, and S. C. Deka, “A Comprehensive Review of Advanced Deep Learning Approaches for Food Freshness Detection,” Food Eng. Rev., vol. 17, no. 1, pp. 127–160, 2025, doi: 10.1007/s12393-024-09385-3.

[10] S. U. Saba et al., “A Critical Review on the Application and Innovation in Smart Fisheries,” Eng. Reports, vol. 8, no. 1, p. e70608, Jan. 2026, doi: https://doi.org/10.1002/eng2.70608.

[11] S. M. Hamidy, Y. Kuvvetli, Y. Sakarya, S. T. Özkütük, and Y. Özoğul, “Deep Learning-Based Prediction of Fish Freshness and Purchasability Using Multi-Angle Image Data,” 2026. doi: 10.3390/foods15010068.

[12] S. Sattar, T. Abbas, M. Tabish, and G. Zhiqiang, “Computer vision in aquaculture: transforming fish freshness monitoring,” Crit. Rev. Food Sci. Nutr., pp. 1–29, Dec. 2025, doi: 10.1080/10408398.2025.2607533.

[13] M. Ashimgaliyev, E. R. Joseph, A. Zhumadillayeva, and A. Baimakhanova, “Development of Deep Learning Methods for Visual Document Classification Using Hybrid Vision Transformer–EfficientNet Architecture,” IEEE Access, vol. 14, pp. 28041–28053, 2026, doi: 10.1109/ACCESS.2025.3646481.

[14] M. Hayat, N. Ahmad, A. Nasir, and Z. A. Tariq, “Hybrid Deep Learning EfficientNetV2 and Vision Transformer (EffNetV2-ViT) Model for Breast Cancer Histopathological Image Classification,” IEEE Access, vol. 12, pp. 184119–184131, 2024, doi: 10.1109/ACCESS.2024.3503413.

[15] K. M. A. Ramadan, E. S. A. Bendary, H. B. Khalil, S. A. Ali, A. R. Ahmed, and M. A. A. Mahmoud, “Smart Detection of Food Spoilage Using Microbial Volatile Compounds: Technologies, Challenges, and Future Outlook,” J. Agric. Food Chem., vol. 73, no. 31, pp. 19222–19243, Aug. 2025, doi: 10.1021/acs.jafc.5c05927.

[16] M. Yumnam, K. Gopalakrishnan, S. Dhua, Y. Srivastava, and P. Mishra, “A Comprehensive Review on Smartphone-Based Sensor for Fish Spoilage Analysis: Applications and Limitations,” Food Bioprocess Technol., vol. 17, no. 12, pp. 4575–4597, 2024, doi: 10.1007/s11947-024-03391-3.

[17] H. G. Nanditha, S. Nishita, G. R. Kavana, G. Lakshmikanth, Y. Venkat, and K. Bandyopadhyay, “Innovative Wireless Technology for Monitoring Consumable Product Degradation,” in Emerging Electronics and Automation, M. Kankanhalli, S. Bhartiya, and P. S. Pravin, Eds., Singapore: Springer Nature Singapore, 2026, pp. 227–242, https://doi.org/10.1007/978-981-96-9554-6_19.

[18] L. Zhang, G. Ren, J. Du, S. Li, Y. Li, and D. Xu, “A novel spatio-temporal attention-based bidirectional LSTM model for moisture content prediction in drying process,” Dry. Technol., vol. 42, no. 14, pp. 2122–2136, Nov. 2024, doi: 10.1080/07373937.2024.2406441.

[19] V. Viancy and N. Gobalakrishnan, “Intelligent food quality monitoring: A hybrid dilated convolutional network and BiLSTM-MSVM model for robust beef quality assessment,” Food Control, vol. 176, p. 111396, 2025, doi: https://doi.org/10.1016/j.foodcont.2025.111396.

[20] P. Padmanabha et al., “Precision Without Complexity: A Comparative Study of YOLO26 Pose Variants for Distal Arm Landmark Detection,” 2026. doi: 10.3390/app16083968.

[21] B. Çarklı Yavuz, “Scale-Dependent Performance Analysis of YOLO26 and YOLOv11 for PPE Detection,” 2026. doi: 10.3390/electronics15061146.

[22] A. A. Abd El-Aziz, M. A. Mahmood, and S. A. El-Ghany, “EfficientNet-B3-Based Automated Deep Learning Framework for Multiclass Endoscopic Bladder Tissue Classification,” Diagnostics, vol. 15, no. 19, 2025, doi: 10.3390/diagnostics15192515.

[23] B. Chen and Z. Dang, “Fast PCB Defect Detection Method Based on FasterNet Backbone Network and CBAM Attention Mechanism Integrated With Feature Fusion Module in Improved YOLOv7,” IEEE Access, vol. 11, pp. 95092–95103, 2023, doi: 10.1109/ACCESS.2023.3311260.

[24] U. G. Redwan, T. Zaman, and H. Bin Mizan, “Spatio-temporal CNN-BiLSTM dynamic approach to emotion recognition based on EEG signal,” Comput. Biol. Med., vol. 192, p. 110277, 2025, doi: https://doi.org/10.1016/j.compbiomed.2025.110277.

[25] Muhardi, M. R. Amartha, R. Melyanti, and Y. Irawan, “Optimization of a New Adaptive Stacking Ensemble Model Integrated with IoT for Stress Level Detection Based on Physiological Signals,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 10, no. 1 SE-Computer Science Applications, Feb. 2026, doi: 10.29207/resti.v10i1.6770.

[26] H. Fonda, Yulanda, M. Ikhsanudin, Muhardi, and Y. Irawan, “Application of Certainty Factor Method to Identify Pests in Crystal Jamboo Plants Application of Certainty Factor Method to Identify Pests in,” J. Phys. Conf. Ser., vol. 1783, no. 1, p. 012053, 2021, doi: 10.1088/1742-6596/1783/1/012053.

[27] Y. Irawan, S. Defit, and R. Sovia, “HSEL: An adaptive rotational hybrid stacking ensemble with exponential accuracy weighting for real-time forest fire risk prediction,” Eng. Appl. Sci. Res., vol. 53, no. 5 SE-ORIGINAL RESEARCH, pp. 519–533, Sep. 2026, doi: 10.64960/easr.2026.266473.