Modified Q-Learning Algorithm for Mobile Robot Real-Time Path Planning using Reduced States
Abstract
Path planning is an essential algorithm in any autonomous mobile robot, including agricultural robots. One of the reinforcement learning methods that can be used for mobile robot path planning is the Q-Learning algorithm. However, the conventional Q-learning method explores all possible robot states in order to find the most optimum path. Thus, this method requires extensive computational cost especially when there are considerable grids to be computed. This study modified the original Q-Learning algorithm by removing the impassable area, so that these areas are not considered as grids to be computed. This modified Q-Learning method was simulated as path finding algorithm for autonomous mobile robot operated at the Agribusiness and Technology Park (ATP), IPB University. Two simulations were conducted to compare the original Q-Learning method and the modified Q-Learning method. The simulation results showed that the state reductions in the modified Q-Learning method can lower the computation cost to 50.71% from the computation cost of the original Q-Learning method, that is, an average computation time of 25.74s as compared to 50.75s, respectively. Both methods produce similar number of states as the robot’s optimal path, i.e. 56 states, based on the reward obtained by the robot while selecting the path. However, the modified Q-Learning algorithm is capable of finding the path to the destination point with a minimum learning rate parameter value of 0.2 when the discount factor value is 0.9.
Downloads
References
S. S. Valle and J. Kienzle, Agriculture 4.0 - Agricultural robotics and automated equipment for sustainable crop production, vol. 24. Rome: FAO, 2020.
M. Javaid, A. Haleem, R. P. Singh, and R. Suman, “Enhancing smart farming through the applications of Agriculture 4.0 technologies,” Int. J. Intell. Networks, vol. 3, no. July, pp. 150–164, 2022, doi: 10.1016/j.ijin.2022.09.004.
M. B. Ahsan, G. Leifeng, F. Mohammad, S. Azam, B. Xu, and S. J. Rayhan, “Barriers, Challenges, and Requirements for ICT Usage among Sub-Assistant Agricultural Officers in Bangladesh : Toward Sustainability in Agriculture,” Sustainability, vol. 15, no. 782, pp. 1–29, 2023.
S. Ruzzante, R. Labarta, and A. Bilton, “Adoption of agricultural technology in the developing world: A meta-analysis of the empirical literature,” World Dev., vol. 146, p. 105599, 2021, doi: 10.1016/j.worlddev.2021.105599.
L. F. P. Oliveira, A. P. Moreira, and M. F. Silva, “Advances in agriculture robotics: A state-of-the-art review and challenges ahead,” Robotics, vol. 10, no. 2, pp. 1–31, 2021, doi: 10.3390/robotics10020052.
L. C. Santos, F. N. Santos, E. J. Solteiro Pires, A. Valente, P. Costa, and S. Magalhaes, “Path planning for ground robots in agriculture: A short review,” in 2020 IEEE International Conference on Autonomous Robot Systems and Competitions, ICARSC 2020, 2020, pp. 61–66, doi: 10.1109/ICARSC49921.2020.9096177.
S. Chakraborty, D. Elangovan, P. L. Govindarajan, M. F. ELnaggar, M. M. Alrashed, and S. Kamel, “A Comprehensive Review of Path Planning for Agricultural Ground Robots,” Sustain., vol. 14, no. 15, pp. 1–19, 2022, doi: 10.3390/su14159156.
C. Cheng, J. Fu, H. Su, and L. Ren, “Recent Advancements in Agriculture Robots: Benefits and Challenge,” Machines, vol. 11, no. 48, pp. 1–24, 2023.
L. F. P. Oliveira, M. F. Silva, and A. P. Moreira, “Agricultural robotics: A state of the art survey,” Robot. Hum. Life- Proc. 23rd Int. Conf. Climbing Walk. Robot. Support Technol. Mob. Mach. CLAWAR 2020, no. August, pp. 279–286, 2020, doi: 10.13180/clawar.2020.24-26.08.44.
S. Fountas, N. Mylonas, I. Malounas, E. Rodias, C. H. Santos, and E. Pekkeriet, “Agricultural robotics for field operations,” Sensors (Switzerland), vol. 20, no. 9, pp. 1–27, 2020, doi: 10.3390/s20092672.
X. Yu et al., “A lab-customized autonomous humanoid apple harvesting robot,” Comput. Electr. Eng., vol. 96, p. 107459, Dec. 2021, doi: 10.1016/j.compeleceng.2021.107459.
K. Zhang, K. Lammers, P. Chu, Z. Li, and R. Lu, “System design and control of an apple harvesting robot,” Mechatronics, vol. 79, Nov. 2021, doi: 10.1016/j.mechatronics.2021.102644.
X. Gao et al., “Review of wheeled mobile robots’ navigation problems and application prospects in agriculture,” IEEE Access, vol. 6, pp. 49248–49268, 2018, doi: 10.1109/ACCESS.2018.2868848.
J. Chen, H. Qiang, J. Wu, G. Xu, and Z. Wang, “Navigation path extraction for greenhouse cucumber-picking robots using the prediction-point Hough transform,” Comput. Electron. Agric., vol. 180, Jan. 2021, doi: 10.1016/j.compag.2020.105911.
Y. Bai, B. Zhang, N. Xu, J. Zhou, J. Shi, and Z. Diao, “Vision-based navigation and guidance for agricultural autonomous vehicles and robots: A review,” Comput. Electron. Agric., vol. 205, Feb. 2023, doi: 10.1016/j.compag.2022.107584.
T. T. Mac, C. Copot, D. T. Tran, and R. De Keyser, “Heuristic approaches in robot path planning: A survey,” Rob. Auton. Syst., vol. 86, pp. 13–28, 2016, doi: 10.1016/j.robot.2016.08.001.
M. S. Abed, O. F. Lutfy, and Q. F. Al-Doori, “A Review on Path Planning Algorithms for Mobile Robots,” Eng. Technol. J., vol. 39, no. 05, pp. 804–820, 2021.
L. Wu, X. Huang, J. Cui, C. Liu, and W. Xiao, “Modified adaptive ant colony optimization algorithm and its application for solving path planning of mobile robot,” Expert Syst. Appl., vol. 215, p. 119410, Apr. 2023, doi: 10.1016/j.eswa.2022.119410.
H. Tian, “Research on Robot Path Planning Based on Improved Ant Colony Algorithm,” Int. J. Comput. Sci. Math., vol. 13, no. 1, pp. 80–92, 2021, doi: 10.1088/1742-6596/1992/3/032050.
R. Sarkar, D. Barman, and N. Chowdhury, “Domain knowledge based genetic algorithms for mobile robot path planning having single and multiple targets,” J. King Saud Univ. - Comput. Inf. Sci., vol. 34, no. 7, pp. 4269–4283, 2022, doi: 10.1016/j.jksuci.2020.10.010.
K. Hao, J. Zhao, Z. Li, Y. Liu, and L. Zhao, “Dynamic path planning of a three-dimensional underwater AUV based on an adaptive genetic algorithm,” Ocean Eng., vol. 263, p. 112421, Nov. 2022, doi: 10.1016/j.oceaneng.2022.112421.
H. S. Dewang, P. K. Mohanty, and S. Kundu, “A Robust Path Planning for Mobile Robot Using Smart Particle Swarm Optimization,” Procedia Comput. Sci., vol. 133, pp. 290–297, 2018, doi: 10.1016/j.procs.2018.07.036.
P. B. Fernandes, R. C. L. Oliveira, and J. V. Fonseca Neto, “Trajectory planning of autonomous mobile robots applying a particle swarm optimization algorithm with peaks of diversity,” Appl. Soft Comput., vol. 116, p. 108108, Feb. 2022, doi: 10.1016/j.asoc.2021.108108.
A. Al Hilli, M. Al-Ibadi, A. M. Alfadhel, S. H. Abdulshaheed, and A. H. Hadi, “Optimal path finding in stochastic quasi-dynamic environments using particle swarm optimization,” Expert Syst. Appl., vol. 186, p. 115706, Dec. 2021, doi: 10.1016/j.eswa.2021.115706.
M. Samadi Gharajeh and H. B. Jond, “An intelligent approach for autonomous mobile robots path planning based on adaptive neuro-fuzzy inference system,” Ain Shams Eng. J., vol. 13, no. 1, p. 101491, 2022, doi: 10.1016/j.asej.2021.05.005.
C. Ntakolia, S. Moustakidis, and A. Siouras, “Autonomous path planning with obstacle avoidance for smart assistive systems,” Expert Syst. Appl., vol. 213, p. 119049, Mar. 2023, doi: 10.1016/j.eswa.2022.119049.
R. Zhen, P. Lv, Z. Shi, and G. Chen, “A novel fuzzy multi-factor navigational risk assessment method for ship route optimization in costal offshore wind farm waters,” Ocean Coast. Manag., vol. 232, p. 106428, Feb. 2023, doi: 10.1016/j.ocecoaman.2022.106428.
O. O. Martins, A. A. Adekunle, O. M. Olaniyan, and B. O. Bolaji, “An Improved multi-objective a-star algorithm for path planning in a large workspace: Design, Implementation, and Evaluation,” Sci. African, vol. 15, p. e01068, 2022, doi: 10.1016/j.sciaf.2021.e01068.
L. Zhao, F. Wang, and Y. Bai, “Route planning for autonomous vessels based on improved artificial fish swarm algorithm,” Ships Offshore Struct., pp. 1–10, Jun. 2022, doi: 10.1080/17445302.2022.2081423.
S. Kumar and A. Sikander, “A modified probabilistic roadmap algorithm for efficient mobile robot path planning,” Eng. Optim., pp. 1–19, Aug. 2022, doi: 10.1080/0305215X.2022.2104840.
M. S. Das, S. Sanyal, and S. Mandal, “Navigation of Multiple Robots in Formative Manner in an Unknown Environment using Artificial Potential Field Based Path Planning Algorithm,” Ain Shams Eng. J., vol. 13, 2022, doi: 10.1016/j.asej.2021.101675.
F. Sui, X. Tang, Z. Dong, X. Gan, P. Luo, and J. Sun, “ACO+PSO+A*: A bi-layer hybrid algorithm for multi-task path planning of an AUV,” Comput. Ind. Eng., vol. 175, p. 108905, Jan. 2023, doi: 10.1016/j.cie.2022.108905.
B. Sahu, P. Kumar Das, and R. Kumar, “A Modified Cuckoo Search Algorithm implemented with SCA and PSO for Multi-robot Cooperation and Path Planning,” Cogn. Syst. Res., Jan. 2023, doi: 10.1016/j.cogsys.2023.01.005.
F. Gul, I. Mir, D. Alarabiat, H. M. Alabool, L. Abualigah, and S. Mir, “Implementation of bio-inspired hybrid algorithm with mutation operator for robotic path planning,” J. Parallel Distrib. Comput., vol. 169, pp. 171–184, Nov. 2022, doi: 10.1016/j.jpdc.2022.06.014.
G. Kulathunga, “A Reinforcement Learning based Path Planning Approach in 3D Environment,” Procedia Comput. Sci., vol. 212, pp. 152–160, 2021, doi: 10.1016/j.procs.2022.10.217.
F. Gismondi, C. Possieri, and A. Tornambe, “A solution to the path planning problem via algebraic geometry and reinforcement learning,” J. Franklin Inst., vol. 359, no. 2, pp. 1732–1754, Jan. 2022, doi: 10.1016/j.jfranklin.2021.12.003.
X. Zhang, S. Xia, X. Li, and T. Zhang, “Multi-objective particle swarm optimization with multi-mode collaboration based on reinforcement learning for path planning of unmanned air vehicles,” Knowledge-Based Syst., vol. 250, p. 109075, Aug. 2022, doi: 10.1016/j.knosys.2022.109075.
E. S. Low, P. Ong, C. Y. Low, and R. Omar, “Modified Q-learning with distance metric and virtual target on path planning of mobile robot,” Expert Syst. Appl., vol. 199, p. 117191, Aug. 2022, doi: 10.1016/j.eswa.2022.117191.
E. S. Low, P. Ong, and K. C. Cheah, “Solving the optimal path planning of a mobile robot using improved Q-Learning,” Rob. Auton. Syst., vol. 115, pp. 143–161, 2019, doi: 10.1016/j.robot.2019.02.013.
S. Gu, “An algorithm for path planning based on improved Q-Learning,” in The Genetic and Evolutionary Computing, 2019, pp. 20–29, doi: https://doi.org/10.1007/978-981-15-3308-2_3.
S. Gu and G. Mao, “An improved Q-Learning algorithm for path planning in maze environments,” in Intelligent Systems and Applications, 2020, vol. 1251, no. 2, pp. 545–557, doi: https://doi.org/10.1007/978-3-030-55187-2_40.
M. Zhao, H. Lu, S. Yang, and F. Guo, “The experience-memory Q-Learning algorithm for robot path planning in unknown environment,” IEEE Access, vol. 8, pp. 47824–47844, 2020, doi: 10.1109/ACCESS.2020.2978077.
C. Yan and X. Xiang, “A Path Planning Algorithm for UAV Based on Improved Q-Learning,” in 2nd International Conference on Robotics and Automation Sciences (ICRAS), 2018, pp. 46–50, doi: 10.1109/ICRAS.2018.8443226.
T. Zhang, X. Huo, S. Chen, B. Yang, and G. Zhang, “Hybrid path planning of a quadrotor UAV based on Q-Learning algorithm,” in Chinese Control Conference ( CCC), 2018, vol. 2018-July, pp. 5415–5419, doi: 10.23919/ChiCC.2018.8482604.
C. J. C. H. Watkins, “Technical Note Q-Learning,” Mach. Learn., vol. 8, pp. 279–292, 1992, doi: 10.1109/ICCC49849.2020.9238991.
R. S. Sutton and A. G. Barto, Reinforcement Learning: An introduction, Second. Cambridge, Massachusetts: MIT Press, 2018.
Copyright (c) 2023 Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi)
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright in each article belongs to the author
- The author acknowledges that the RESTI Journal (System Engineering and Information Technology) is the first publisher to publish with a license Creative Commons Attribution 4.0 International License.
- Authors can enter writing separately, arrange the non-exclusive distribution of manuscripts that have been published in this journal into other versions (eg sent to the author's institutional repository, publication in a book, etc.), by acknowledging that the manuscript has been published for the first time in the RESTI (Rekayasa Sistem dan Teknologi Informasi) journal ;
