Multimodal Contextual Learning for Maize Disease Diagnosis: Fusing Visual and Environmental Data
DOI:
https://doi.org/10.29207/resti.v10i4.7671Keywords:
convolutional neural network, deep learning, explainable AI, Grad-CAM, maize lethal necrosis, maize streak virus, multi-task learning, precision agricultureAbstract
Maize Lethal Necrosis (MLN) and Maize Streak Virus (MSV) threaten food security in Zimbabwe, yet traditional image-based deep learning models lack adaptability to varying agro-environmental conditions. This study presents a multimodal contextual learning framework that fuses visual symptoms with environmental data to improve diagnostic accuracy and interpretability. A multimodal Convolutional Neural Network integrates leaf imagery with six environmental parameters (temperature, humidity, rainfall, soil moisture, leafhopper count, days after planting) using a late-fusion architecture with multi-task learning for simultaneous disease classification and five-stage severity estimation, enhanced by Gradient-weighted Class Activation Mapping (Grad-CAM) for visual explanation. A synthesized dataset (n=9,356) representative of Zimbabwean conditions was used. The multimodal approach achieved 94.3% accuracy versus 91.5% for image-only baselines (p<0.001, 95% CI), a 33.2% relative error reduction, with MSV-MLN confusion decreasing by 40.7% (113 to 67 misclassifications). Multi-task performance reached 94.5% for classification and 81.5% for severity estimation (F1=0.805). Grad-CAM analysis revealed environmental integration enhanced attention by 38.1% and localization by 45.2%, with leafhopper count showing the strongest correlation (r=0.62). Ablation studies confirmed that environmental features provided the largest accuracy gain (+3.71%, p<0.001). This work demonstrates that integrating environmental context with visual symptoms enhances diagnostic accuracy and model interpretability, establishing a foundational proof-of-concept for context-aware agricultural AI.
Downloads
References
A. K. Biswal et al., “Maize Lethal Necrosis disease: review of molecular and genetic resistance mechanisms, socio-economic impacts, and mitigation strategies in sub-Saharan Africa,” BMC Plant Biol., vol. 22, no. 1, p. 542, Nov. 2022, doi.org/10.1186/s12870-022-03932-y.
S. Suara, A. Jha, P. Sinha, and A. A. Sekh, “Is Grad-CAM Explainable in Medical Images?,” in Computer Vision and Image Processing, vol. 2009, H. Kaur, V. Jakhetiya, P. Goyal, P. Khanna, B. Raman, and S. Kumar, Eds., in Communications in Computer and Information Science, vol. 2009. , Cham: Springer Nature Switzerland, 2024, pp. 124–135. doi: 10.1007/978-3-031-58181-6_11.
T. Ayoub Shaikh, T. Rasool, and F. Rasheed Lone, “Towards leveraging the role of machine learning and artificial intelligence in precision agriculture and smart farming,” Comput. Electron. Agric., vol. 198, p. 107119, Jul. 2022, doi: 10.1016/j.compag.2022.107119.
D. Wang, W. Cao, F. Zhang, Z. Li, S. Xu, and X. Wu, “A Review of Deep Learning in Multiscale Agricultural Sensing,” Remote Sens., vol. 14, no. 3, p. 559, Jan. 2022, doi: 10.3390/rs14030559.
M. Albahar, “A Survey on Deep Learning and Its Impact on Agriculture: Challenges and Opportunities,” Agriculture, vol. 13, no. 3, p. 540, Feb. 2023, doi: 10.3390/agriculture13030540.
S. R. Shah, S. Qadri, H. Bibi, S. M. W. Shah, M. I. Sharif, and F. Marinello, “Comparing Inception V3, VGG 16, VGG 19, CNN, and ResNet 50: A Case Study on Early Detection of a Rice Disease,” Agronomy, vol. 13, no. 6, p. 1633, Jun. 2023, doi: 10.3390/agronomy13061633.
Md. M. Islam et al., “DeepCrop: Deep learning-based crop disease prediction with web application,” J. Agric. Food Res., vol. 14, p. 100764, Dec. 2023, doi: 10.1016/j.jafr.2023.100764.
F. Shahoveisi, H. Taheri Gorji, S. Shahabi, S. Hosseinirad, S. Markell, and F. Vasefi, “Application of image processing and transfer learning for the detection of rust disease,” Sci. Rep., vol. 13, no. 1, p. 5133, Mar. 2023, doi: 10.1038/s41598-023-31942-9.
S. A. F. Harahap and I. Irmawan, “Performance comparison of MobileNet, EfficientNet, and Inception for predicting crop disease,” Sriwij. Electr. Comput. Eng. J., vol. 1, no. 1, pp. 30–36, Feb. 2024, doi: 10.62420/selco.v1i1.4.
R. R. Patil and S. Kumar, “Rice-Fusion: A Multimodality Data Fusion Framework for Rice Disease Diagnosis,” IEEE Access, vol. 10, pp. 5207–5222, 2022, doi: 10.1109/ACCESS.2022.3140815.
N. Zhang, H. Wu, H. Zhu, Y. Deng, and X. Han, “Tomato Disease Classification and Identification Method Based on Multimodal Fusion Deep Learning,” Agriculture, vol. 12, no. 12, p. 2014, Nov. 2022, doi: 10.3390/agriculture12122014.
J. Su, X. Zhu, S. Li, and W.-H. Chen, “AI meets UAVs: A survey on AI empowered UAV perception systems for precision agriculture,” Neurocomputing, vol. 518, pp. 242–270, Jan. 2023, doi: 10.1016/j.neucom.2022.11.020.
R. Rashid, W. Aslam, R. Aziz, and G. Aldehim, “An Early and Smart Detection of Corn Plant Leaf Diseases Using IoT and Deep Learning Multi-Models,” IEEE Access, vol. 12, pp. 23149–23162, 2024, doi: 10.1109/ACCESS.2024.3357099.
O. Elsherbiny, L. Zhou, Y. He, and Z. Qiu, “A novel hybrid deep network for diagnosing water status in wheat crop using IoT-based multimodal data,” Comput. Electron. Agric., vol. 203, p. 107453, Dec. 2022, doi: 10.1016/j.compag.2022.107453.
M. Bhandari, T. B. Shahi, A. Neupane, and K. B. Walsh, “BotanicX-AI: Identification of Tomato Leaf Diseases Using an Explanation-Driven Deep-Learning Model,” J. Imaging, vol. 9, no. 2, p. 53, Feb. 2023, doi: 10.3390/jimaging9020053.
N. Nigar, H. Muhammad Faisal, M. Umer, O. Oki, and J. Manappattukunnel Lukose, “Improving Plant Disease Classification With Deep-Learning-Based Prediction Model Using Explainable Artificial Intelligence,” IEEE Access, vol. 12, pp. 100005–100014, 2024, doi: 10.1109/ACCESS.2024.3428553.
G. Fenu and F. M. Malloci, “Using Multioutput Learning to Diagnose Plant Disease and Stress Severity,” Complexity, vol. 2021, no. 1, p. 6663442, Jan. 2021, doi: 10.1155/2021/6663442.
Z.-X. Yang, Y. Li, R.-F. Wang, P. Hu, and W.-H. Su, “Deep Learning in Multimodal Fusion for Sustainable Plant Care: A Comprehensive Review,” Sustainability, vol. 17, no. 12, p. 5255, Jun. 2025, doi: 10.3390/su17125255.
L. G. Divyanth, A. Ahmad, and D. Saraswat, “A two-stage deep-learning based segmentation model for crop disease quantification based on corn field imagery,” Smart Agric. Technol., vol. 3, p. 100108, Feb. 2023, doi: 10.1016/j.atech.2022.100108.
Y. Liu et al., “Exploring multi-features in UAV based optical and thermal infrared images to estimate disease severity of wheat powdery mildew,” Comput. Electron. Agric., vol. 225, p. 109285, Oct. 2024, doi: 10.1016/j.compag.2024.109285.
A. S. Keceli, A. Kaya, C. Catal, and B. Tekinerdogan, “Deep learning-based multi-task prediction system for plant disease and species detection,” Ecol. Inform., vol. 69, p. 101679, Jul. 2022, doi: 10.1016/j.ecoinf.2022.101679.
S. Mostafa, D. Mondal, K. Panjvani, L. Kochian, and I. Stavness, “Explainable deep learning in plant phenotyping,” Front. Artif. Intell., vol. 6, p. 1203546, Sep. 2023, doi: 10.3389/frai.2023.1203546.
A. Saporta et al., “Benchmarking saliency methods for chest X-ray interpretation,” Nat. Mach. Intell., vol. 4, no. 10, pp. 867–878, Oct. 2022, doi: 10.1038/s42256-022-00536-x.
D. S. Joseph, P. M. Pawar, and K. Chakradeo, “Real-Time Plant Disease Dataset Development and Detection of Plant Disease Using Deep Learning,” IEEE Access, vol. 12, pp. 16310–16333, 2024, doi: 10.1109/ACCESS.2024.3358333.
G. Dai, J. Fan, Z. Tian, and C. Wang, “PPLC-Net:Neural network-based plant disease identification model supported by weather data augmentation and multi-level attention mechanism,” J. King Saud Univ. - Comput. Inf. Sci., vol. 35, no. 5, p. 101555, May 2023, doi: 10.1016/j.jksuci.2023.101555.
Y. Lu, D. Chen, E. Olaniyi, and Y. Huang, “Generative adversarial networks (GANs) for image augmentation in agriculture: A systematic review,” Comput. Electron. Agric., vol. 200, p. 107208, Sep. 2022, doi: 10.1016/j.compag.2022.107208.
S. Lamba, A. Baliyan, and V. Kukreja, “GAN based image augmentation for increased CNN performance in Paddy leaf disease classification,” in 2022 2nd International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE), Greater Noida, India: IEEE, Apr. 2022, pp. 2054–2059. doi: 10.1109/ICACITE53722.2022.9823799.
Y. Haruna, S. Qin, and M. J. Mbyamm Kiki, “An Improved Approach to Detection of Rice Leaf Disease with GAN-Based Data Augmentation Pipeline,” Appl. Sci., vol. 13, no. 3, p. 1346, Jan. 2023, doi: 10.3390/app13031346.
J. Lu, L. Tan, and H. Jiang, “Review on Convolutional Neural Network (CNN) Applied to Plant Leaf Disease Classification,” Agriculture, vol. 11, no. 8, p. 707, Jul. 2021, doi: 10.3390/agriculture11080707.
W. Shafik, A. Tufail, C. De Silva Liyanage, and R. A. A. H. M. Apong, “Using transfer learning-based plant disease classification and detection for sustainable agriculture,” BMC Plant Biol., vol. 24, no. 1, p. 136, Feb. 2024, doi: 10.1186/s12870-024-04825-y.
M. Mushayi, H. Shimelis, J. Derera, and S. A. Tesfamariam, “Breeding for resistance to maize streak virus: challenges, progress and future directions: a review,” Front. Plant Sci., vol. 16, p. 1590870, Jun. 2025, doi: 10.3389/fpls.2025.1590870.
E. W. Ohlson and J. R. Wilson, “Maize Lethal Necrosis: Impact and Disease Management,” Outlooks Pest Manag., vol. 33, no. 2, pp. 45–51, Apr. 2022, doi: 10.1564/v33_apr_02.
Downloads
Published
Issue
Section
License
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
